[00:05:55] FIRING: MaxConntrack: Max conntrack at 90.6% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:06:03] 06cloud-services-team: MaxConntrack Max conntrack at 90.6% on cloudvirt1039:9100 - https://phabricator.wikimedia.org/T387179 (10phaultfinder) 03NEW [00:22:55] FIRING: MaxConntrack: Max conntrack at 89.98% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:30:55] RESOLVED: MaxConntrack: Max conntrack at 90.39% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:51:55] RESOLVED: MaxConntrack: Max conntrack at 82.86% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:54:55] (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804) [01:00:42] 10VPS-project-Wikistats: Add sylwiki to wikistats - https://phabricator.wikimedia.org/T386469#10577700 (10Pppery) 05Stalled→03Open [01:10:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [01:15:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [01:34:04] (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804) [03:19:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [03:24:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [04:57:55] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1122124 (owner: 10L10n-bot) [06:11:06] 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki sylwiki - https://phabricator.wikimedia.org/T386467#10577916 (10Marostegui) This can proceed, I have sanitized everything at T386463 [07:45:49] 10Tool-inteGraality: pywikibot.exceptions.ServerError: HTTPSConnectionPool(host='query.wikidata.org', port=443): Read timed out. (read timeout=45) - https://phabricator.wikimedia.org/T384882#10578044 (10JeanFred) @Palotabarat How did this happen − doing what, on which page etc? [09:06:02] 10Tool-inteGraality: Wikidata:WikiProject every politician/Statistics not updating - https://phabricator.wikimedia.org/T387070#10578127 (10JeanFred) Re-ran, it completed in ~400s. I guess the dataset is big and this is flirting with the timeouts. [09:25:05] 10Tool-refill: reFill fails to load, errors out with "Access to storage is not allowed from this context" - https://phabricator.wikimedia.org/T386446#10578168 (10Curb_Safe_Charmer) 05Open→03Declined [10:12:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [10:17:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [10:21:21] 06cloud-services-team, 10Toolforge (Toolforge iteration 17), 07Kubernetes: [jobs-api] when manually triggering a scheduled job, we strip off any unknown fields set in the k8s cronjob - https://phabricator.wikimedia.org/T387201 (10dcaro) 03NEW [10:21:58] 06cloud-services-team, 10Toolforge (Toolforge iteration 17): [jobs-api] when manually triggering a scheduled job, we strip off any unknown fields set in the k8s cronjob - https://phabricator.wikimedia.org/T387201#10578403 (10dcaro) [10:36:24] 10Tool-inteGraality: pywikibot.exceptions.ServerError: HTTPSConnectionPool(host='query.wikidata.org', port=443): Read timed out. (read timeout=45) - https://phabricator.wikimedia.org/T384882#10578441 (10Palotabarat) Hi @JeanFred ! This was the relevant page: https://www.wikidata.org/wiki/Wikidata:WikiProject_eve... [10:36:55] 06cloud-services-team, 10Toolforge (Toolforge iteration 17), 07Kubernetes, 13Patch-For-Review: [jobs-api] Allow Toolforge scheduled jobs to have a maximum runtime - https://phabricator.wikimedia.org/T306391#10578442 (10dcaro) This is now implemented :), you can pass `--timeout 123` 123 being the number of... [10:45:37] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki sylwiki - https://phabricator.wikimedia.org/T386467#10578456 (10fnegri) [11:04:21] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki sylwiki - https://phabricator.wikimedia.org/T386467#10578492 (10fnegri) 05Open→03In progress [11:04:36] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki satwiktionary - https://phabricator.wikimedia.org/T386634#10578495 (10fnegri) 05Open→03In progress [11:04:59] (03open) 10dcaro: scheduled_jobs: create job from k8s cronjob on manual trigger [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/144 [11:12:03] (03update) 10dcaro: scheduled_jobs: create job from k8s cronjob on manual trigger [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/144 [11:14:45] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10578519 (10Ladsgroup) The renames have finished and only these two left: ` ladsgroup@mwmaint2002:~$ sql centralauth -- -e "sele... [11:18:05] (03update) 10fnegri: Draft: Upgrade Kubernetes to 1.29 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/227 [11:20:05] (03update) 10dcaro: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [11:23:19] (03update) 10dcaro: scheduled_jobs: create job from k8s cronjob on manual trigger [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/144 [11:31:34] (03update) 10dcaro: volume-admission: add first test for disallowed path [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/675 [11:32:53] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki satwiktionary - https://phabricator.wikimedia.org/T386634#10578583 (10fnegri) 05In progress→03Resolved [11:33:02] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki sylwiki - https://phabricator.wikimedia.org/T386467#10578586 (10fnegri) 05In progress→03Resolved [11:33:13] (03update) 10dcaro: scheduled_jobs: create job from k8s cronjob on manual trigger [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/144 [11:37:21] (03update) 10dcaro: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [11:43:55] (03update) 10dcaro: volume-admission: add first test for disallowed path [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/675 [11:50:21] (03update) 10dcaro: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [11:50:31] (03update) 10dcaro: volume-admission: add first test for disallowed path [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/675 [12:30:26] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: wmfkeystonehooks: project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10578654 (10Andrew) 05In progress→03Resolved [12:31:07] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for o... - https://phabricator.wikimedia.org/T374830#10578659 [12:55:32] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2025.02.10 - 2025.02.28): Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10578718 (10Andrew) #3 feels like the right solution to... [12:56:51] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10578719 (10Ladsgroup) The first one gives exception: ` [8f1974a4-18bb-494c-b375-c2a189529772] 2025-02-25 12:55:29: Fatal except... [12:59:34] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10578734 (10Ladsgroup) Both are fixed now. There shouldn't be any user with conflicting username left. About to create global us... [13:02:39] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10578756 (10Ladsgroup) Technically I should just run migratePass1.php but let's take a look at the code first. See if you spot s... [13:03:50] 06cloud-services-team, 10Toolforge (Toolforge iteration 17): [jobs-api] when manually triggering a scheduled job, we strip off any unknown fields set in the k8s cronjob - https://phabricator.wikimedia.org/T387201#10578763 (10dcaro) p:05Triage→03Medium a:03dcaro [13:04:36] 06cloud-services-team, 10Toolforge (Toolforge iteration 17): [jobs-api] when manually triggering a scheduled job, we strip off any unknown fields set in the k8s cronjob - https://phabricator.wikimedia.org/T387201#10578768 (10dcaro) 05Open→03In progress [13:09:53] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudgw100[12] - https://phabricator.wikimedia.org/T386810#10578798 (10Andrew) >>! In T386810#10577268, @VRiley-WMF wrote: > Unracked and removed the following servers. However, the script has failed and returning... [13:10:30] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10578799 (10Andrew) Many thanks to everyone for working on this massive project! [13:13:37] (03open) 10marostegui: task_template_es_primary.txt: Remove line [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/8 (https://phabricator.wikimedia.org/T387211) [13:17:54] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10578814 (10Ladsgroup) I ran this instead: ` mwscript extensions/CentralAuth/maintenance/migrateAccount.php --wiki=labswiki --au... [13:20:27] (03update) 10dcaro: deploy-token: prevent accidental token overwrites [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/49 (owner: 10sstefanova) [13:26:54] (03PS1) 10David Caro: run_tests: consider the tests passed even if there's skipped ones [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1122573 [13:28:20] (03update) 10dcaro: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [13:35:23] (03approved) 10ladsgroup: task_template_es_primary.txt: Remove line [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/8 (https://phabricator.wikimedia.org/T387211) (owner: 10marostegui) [13:35:27] (03merge) 10ladsgroup: task_template_es_primary.txt: Remove line [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/8 (https://phabricator.wikimedia.org/T387211) (owner: 10marostegui) [13:40:21] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10578897 (10Ladsgroup) There are a couple of users with reversed hashes, I will see what I can do about them. [13:43:02] (03open) 10dcaro: bash-completion: add refresh [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/17 [13:44:59] (03update) 10dcaro: deploy-token: prevent accidental token overwrites [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/49 (owner: 10sstefanova) [13:45:29] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10578918 (10Ladsgroup) Progress: ` ladsgroup@mwmaint2002:~$ sql centralauth -- -e "select ln_name from localnames where ln_wiki... [13:53:12] (03update) 10dcaro: bash-completion: add refresh [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/17 [14:29:04] 06cloud-services-team, 10Cloud-VPS, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q3): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10579070 (10cmooney) @dcaro the interface status should now be there for you. In t... [14:35:35] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10579115 (10A_smart_kitten) `Delete page script` may be a wider issue (for want of a better word) than just Wikitech, cf. {T386790} [14:41:58] 06cloud-services-team, 10Toolforge, 07Epic: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools - https://phabricator.wikimedia.org/T127367#10579141 (10Andrew) We discussed loki storage a bit, it seems like we can get a decent prototype in lima-kilo with https://g... [14:51:28] 10Toolforge (Toolforge iteration 17), 13Patch-For-Review: [maintain-harbor] refactor config handling - https://phabricator.wikimedia.org/T386953#10579188 (10dcaro) p:05Triage→03High [14:55:53] (03update) 10dcaro: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) (owner: 10raymond-ndibe) [15:00:43] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS looku... - https://phabricator.wikimedia.org/T374830#10579209 [15:02:49] 10Toolforge (Toolforge iteration 17): [harbor] move harbor setup scripts from builds-builder to maintain-harbor - https://phabricator.wikimedia.org/T386964#10579230 (10dcaro) p:05Triage→03Low [15:03:04] 06cloud-services-team, 10Toolforge: [harbor] move harbor setup scripts from builds-builder to maintain-harbor - https://phabricator.wikimedia.org/T386964#10579232 (10dcaro) [15:03:52] 10Toolforge (Toolforge iteration 17): [components-cli] Add the `refresh` subcommand to the autocomplete file - https://phabricator.wikimedia.org/T384641#10579235 (10dcaro) a:03dcaro [15:03:56] 10Toolforge (Toolforge iteration 17): [components-cli] Add the `refresh` subcommand to the autocomplete file - https://phabricator.wikimedia.org/T384641#10579237 (10dcaro) 05Open→03In progress [15:04:27] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS looku... - https://phabricator.wikimedia.org/T374830#10579239 [15:04:29] 10Toolforge (Toolforge iteration 17): [components-api] Trigger builds on every deploy - https://phabricator.wikimedia.org/T384634#10579240 (10dcaro) a:05Raymond_Ndibe→03dcaro [15:07:02] 06cloud-services-team, 10Toolforge: [jobs-cli] If the pod exists and it has no logs, read the message status from it and output that - https://phabricator.wikimedia.org/T384251#10579249 (10dcaro) [15:08:25] 06cloud-services-team, 10Toolforge: [jobs-emailer] If the pod is in error status, try to get the status.message field in the email, otherwise just 'error' is not that useful - https://phabricator.wikimedia.org/T384252#10579256 (10dcaro) [15:18:35] 06cloud-services-team, 10Toolforge (Toolforge iteration 17), 10Sustainability (Incident Followup): [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components - https://phabricator.wikimedia.org/T380959#10579284 (10dcaro) [15:19:42] 06cloud-services-team, 10Toolforge (Toolforge iteration 17): [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#10579291 (10dcaro) [15:20:28] 06cloud-services-team, 10Toolforge (Toolforge iteration 17): Upgrade python buildpack to v0.17.0 or newer for Poetry support - https://phabricator.wikimedia.org/T374056#10579296 (10dcaro) [15:22:56] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 17), 07Epic: [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform - https://phabricator.wikimedia.org/T368600#10579330 (10dcaro) [15:23:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [15:24:26] 06cloud-services-team, 10Toolforge: [lima-kilo] toolforge_deploy_mr.py confusing message - https://phabricator.wikimedia.org/T362389#10579337 (10dcaro) 05Open→03Resolved a:03dcaro This is not there anymore, now it will do a rollout restart if the helmfile had no changes (to pull new images). [15:28:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [15:28:55] 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [maintain-harbor] refactor config handling - https://phabricator.wikimedia.org/T386953#10579354 (10dcaro) [15:28:55] 06cloud-services-team, 10Toolforge (Toolforge iteration 18): [jobs-api] when manually triggering a scheduled job, we strip off any unknown fields set in the k8s cronjob - https://phabricator.wikimedia.org/T387201#10579356 (10dcaro) [15:28:58] 06cloud-services-team, 10Toolforge (Toolforge iteration 18): [harbor, builds-builder] Audit robot account permissions - https://phabricator.wikimedia.org/T361708#10579364 (10dcaro) [15:28:59] 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [jobs-api] Save business models in a DB - https://phabricator.wikimedia.org/T359650#10579358 (10dcaro) [15:28:59] 10Toolforge (Toolforge iteration 18), 13Patch-For-Review, 07Upstream: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417#10579360 (10dcaro) [15:29:02] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [maintain-harbor] Have maintain-harbor use a robot account - https://phabricator.wikimedia.org/T361698#10579362 (10dcaro) [15:29:11] 10Toolforge (Toolforge iteration 18), 07Upstream: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#10579368 (10dcaro) [15:29:14] 10Toolforge (Toolforge iteration 18), 07Upstream: [builds-builder,jobs-api,upstream] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#10579371 (10dcaro) [15:29:18] 10Toolforge (Toolforge iteration 18): [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#10579366 (10dcaro) [15:29:22] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18): Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#10579375 (10dcaro) [15:29:26] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#10579373 (10dcaro) [15:29:34] 10Toolforge (Toolforge iteration 18): [components-cli] Add the `refresh` subcommand to the autocomplete file - https://phabricator.wikimedia.org/T384641#10579378 (10dcaro) [15:29:42] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [jobs-api,jobs-cli] Introduce a way to stop stuck cronjobs - https://phabricator.wikimedia.org/T377420#10579382 (10dcaro) [15:29:46] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18): [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868#10579386 (10dcaro) [15:29:50] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 07Epic, 13Patch-For-Review: [o11y,logging,infra] loki into lima-kilo - https://phabricator.wikimedia.org/T386480#10579380 (10dcaro) [15:29:54] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 07Kubernetes, 13Patch-For-Review: [jobs-api] Allow Toolforge scheduled jobs to have a maximum runtime - https://phabricator.wikimedia.org/T306391#10579384 (10dcaro) [15:29:58] 10Toolforge (Toolforge iteration 18), 07Documentation: [harbor,docs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092#10579388 (10dcaro) [15:30:02] 10Toolforge (Toolforge iteration 18): [components-api] add basic prometheus instrumentation - https://phabricator.wikimedia.org/T381249#10579394 (10dcaro) [15:30:06] 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: Persist important toolforge k8s components logs - https://phabricator.wikimedia.org/T383081#10579390 (10dcaro) [15:30:10] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: Toolforge: Replace all bastion with grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#10579392 (10dcaro) [15:30:14] 10Toolforge (Toolforge iteration 18): [components-api, components-cli] deploy-token: separate create from update - https://phabricator.wikimedia.org/T380706#10579396 (10dcaro) [15:30:18] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18), 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10579400 (10dcaro) [15:30:22] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [jobs-api] treat URLs with and without a trailing slash the same - https://phabricator.wikimedia.org/T383798#10579398 (10dcaro) [15:30:26] 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [jobs-api] Split the API, business, and k8s models - https://phabricator.wikimedia.org/T359808#10579402 (10dcaro) [15:30:31] 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [jobs-api,jobs-cli] restarting a continuous jobs causes for some seconds two jobs are running side by side - https://phabricator.wikimedia.org/T375366#10579404 (10dcaro) [15:30:39] 06cloud-services-team, 10Toolforge (Toolforge iteration 18): [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#10579406 (10dcaro) [15:30:43] 10Toolforge (Toolforge iteration 18): [components-api] Rename the CRDs groups to be `components-api.toolforge.org` - https://phabricator.wikimedia.org/T386829#10579409 (10dcaro) [15:30:47] 06cloud-services-team, 10Toolforge (Toolforge iteration 18): Upgrade python buildpack to v0.17.0 or newer for Poetry support - https://phabricator.wikimedia.org/T374056#10579407 (10dcaro) [15:30:51] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18), 07Epic: [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform - https://phabricator.wikimedia.org/T368600#10579408 (10dcaro) [15:30:55] 10Toolforge (Toolforge iteration 18): [components-api] Add support for port/helathcheck for continuous jobs in tool config/depolyment - https://phabricator.wikimedia.org/T362072#10579411 (10dcaro) [15:30:59] 10Toolforge (Toolforge iteration 18): [components-api] Trigger builds on every deploy - https://phabricator.wikimedia.org/T384634#10579410 (10dcaro) [15:31:03] 10Toolforge (Toolforge iteration 18): [builds-cli,builds-api] `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701#10579412 (10dcaro) [15:31:07] 10Toolforge (Toolforge iteration 18): [usage] Try to get an idea of the amount of tools that were created, but never started anything - https://phabricator.wikimedia.org/T379144#10579414 (10dcaro) [15:31:11] 10Toolforge (Toolforge iteration 18): [jobs-api] prepend date and pod name to filelog lines - https://phabricator.wikimedia.org/T372025#10579415 (10dcaro) [15:31:15] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 10Sustainability (Incident Followup): [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components - https://phabricator.wikimedia.org/T380959#10579413 (10dcaro) [15:31:19] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge (Toolforge iteration 18), 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#10579416 (10dcaro) [15:32:03] (03open) 10marostegui: task_template_es_primary.txt: Fix deployment CNAME [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/9 (https://phabricator.wikimedia.org/T387224) [15:32:16] (03approved) 10dcaro: deploy-token: prevent accidental token overwrites [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/49 (owner: 10sstefanova) [15:32:22] (03merge) 10dcaro: deploy-token: prevent accidental token overwrites [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/49 (owner: 10sstefanova) [15:35:45] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.79-20250225153240-8d8422b8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/676 [15:39:11] 06cloud-services-team, 10Cloud-VPS: Members of https://ldap.toolforge.org/group/project-recommendation-api not added to project-bastion - https://phabricator.wikimedia.org/T387226 (10XiaoXiao-WMF) 03NEW [15:44:53] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS looku... - https://phabricator.wikimedia.org/T374830#10579526 [15:47:15] 06cloud-services-team, 10Cloud-VPS: Members of https://ldap.toolforge.org/group/project-recommendation-api not added to project-bastion - https://phabricator.wikimedia.org/T387226#10579540 (10taavi) →14Duplicate dup:03T379550 [15:47:18] 06cloud-services-team, 10Cloud-VPS: openstack: keystone may be failing to add users to the bastion project - https://phabricator.wikimedia.org/T379550#10579542 (10taavi) [15:53:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [15:55:08] (03open) 10dcaro: toolconfig: added build from source (buildpack) support [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/52 (https://phabricator.wikimedia.org/T384634) [15:55:17] (03update) 10dcaro: toolconfig: added build from source (buildpack) support [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/52 (https://phabricator.wikimedia.org/T384634) [16:02:01] 06cloud-services-team, 06DC-Ops, 10Ganeti, 06Infrastructure-Foundations, and 2 others: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10579629 (10Andrew) @MoritzMuehlenhoff ping, is ganeti1044 ready to be moved? [16:13:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [16:19:08] (03merge) 10ladsgroup: task_template_es_primary.txt: Fix deployment CNAME [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/9 (https://phabricator.wikimedia.org/T387224) (owner: 10marostegui) [16:31:49] (03update) 10dcaro: toolconfig: added build from source (buildpack) support [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/52 (https://phabricator.wikimedia.org/T384634) [16:35:21] (03open) 10dcaro: deploy: support deploying source build components (buildpack) [repos/cloud/toolforge/components-api] (add_source_build_to_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T384634) [16:35:56] (03update) 10dcaro: deploy: support deploying source build components (buildpack) [repos/cloud/toolforge/components-api] (add_source_build_to_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T384634) [16:48:28] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10579799 (10Dillon) >>! In T376267#10577419, @Ladsgroup wrote: > I created `DHardy-WMF` for you: > ` > ladsgroup@mwmaint2002:~$ mwscript extensions/CentralAuth/maintenance/createLocalAccount.... [16:57:52] (03update) 10dcaro: toolconfig: added build from source (buildpack) support [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/52 (https://phabricator.wikimedia.org/T384634) [16:58:32] (03update) 10dcaro: deploy: support deploying source build components (buildpack) [repos/cloud/toolforge/components-api] (add_source_build_to_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T384634) [16:59:17] (03update) 10dcaro: toolconfig: added build from source (buildpack) support [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/52 (https://phabricator.wikimedia.org/T384634) [17:20:25] (03open) 10dcaro: volume-admission: use pull policy Always in local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/677 [17:23:13] (03update) 10dcaro: volume-admission: use pull policy Always in local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/677 [17:30:18] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS looku... - https://phabricator.wikimedia.org/T374830#10579908 [17:42:56] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10579966 (10Ladsgroup) Now only this left which is a different issue: ` ladsgroup@mwmaint2002:~$ sql centralauth -- -e "select l... [17:48:44] (03open) 10bd808: jobs: Add jobs.yaml tracking deployed jobs [toolforge-repos/gitlab-account-approval] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/21 [17:52:57] (03update) 10dcaro: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [17:55:19] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10580023 (10Aklapper) >>! In T161859#10579966, @Ladsgroup wrote: > Now only this left which is a different issue: > ` > | Delete... [17:59:33] (03open) 10dcaro: namespace: add comment about the tenancy label usage [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/66 [18:06:16] 10wikitech.wikimedia.org, 13Patch-For-Review: Re-enable account creation on Wikitech - https://phabricator.wikimedia.org/T377074#10580043 (10Ladsgroup) 05Stalled→03In progress [18:14:38] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10580065 (10bd808) [18:18:22] (03merge) 10bd808: jobs: Add jobs.yaml tracking deployed jobs [toolforge-repos/gitlab-account-approval] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/21 [18:20:06] (03update) 10dcaro: namespace: add comment about the tenancy label usage [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/66 [18:20:59] 10wikitech.wikimedia.org: Wikitech double redirect bot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376224#10580089 (10Ladsgroup) I added wikitech to the list of wikis I'm running the bot on (it runs as part of hourly jobs in toolforge in dexbot project). It w... [18:21:59] 10wikitech.wikimedia.org, 13Patch-For-Review: Re-enable account creation on Wikitech - https://phabricator.wikimedia.org/T377074#10580097 (10Ladsgroup) 05In progress→03Resolved a:03taavi [18:23:35] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188#10580107 (10Ladsgroup) I just dropped the whole table. [18:24:55] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188#10580112 (10bd808) 05Open→03Resolved a:03Ladsgroup [18:25:21] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10580115 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup wikitech is now a proper SUL wiki and all of authentication is the same as any other SUL wiki. [18:38:43] 10wikitech.wikimedia.org: Audit MediaWiki namespace message overrides for legacy Developer account related information - https://phabricator.wikimedia.org/T376231#10580179 (10Ladsgroup) Are these needed?: - https://wikitech.wikimedia.org/wiki/MediaWiki:Blog-text - https://wikitech.wikimedia.org/wiki/MediaWiki:... [18:44:01] 10wikitech.wikimedia.org: Audit MediaWiki namespace message overrides for legacy Developer account related information - https://phabricator.wikimedia.org/T376231#10580217 (10taavi) >>! In T376231#10580179, @Ladsgroup wrote: > Are these needed?: > - https://wikitech.wikimedia.org/wiki/MediaWiki:Blog-text > - h... [19:15:11] (03approved) 10raymond-ndibe: volume-admission: use pull policy Always in local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/677 (owner: 10dcaro) [19:15:12] (03update) 10raymond-ndibe: volume-admission: use pull policy Always in local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/677 (owner: 10dcaro) [19:15:15] (03merge) 10raymond-ndibe: volume-admission: use pull policy Always in local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/677 (owner: 10dcaro) [19:19:03] (03update) 10raymond-ndibe: bash-completion: add refresh [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/17 (owner: 10dcaro) [19:19:05] (03approved) 10raymond-ndibe: bash-completion: add refresh [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/17 (owner: 10dcaro) [19:19:09] (03merge) 10raymond-ndibe: bash-completion: add refresh [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/17 (owner: 10dcaro) [19:20:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [19:46:42] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 [19:50:57] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 [19:55:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [19:56:59] 10wikitech.wikimedia.org: Audit MediaWiki namespace message overrides for legacy Developer account related information - https://phabricator.wikimedia.org/T376231#10580420 (10Ladsgroup) > Probably could be removed, Wikitech hasn't had SMW in ages. just deleted them. [19:58:00] 10wikitech.wikimedia.org: Audit MediaWiki namespace message overrides for legacy Developer account related information - https://phabricator.wikimedia.org/T376231#10580422 (10Ladsgroup) > This looks like something that was at some point used in the sidebar? Since it's no longer there, we might be able to remove... [20:15:40] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10580463 (10Ladsgroup) [20:18:10] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10580466 (10Ladsgroup) I dropped these tables: ` drop table if exists echo_email_batch; drop table if exists echo_event; drop table if exists echo_not... [20:42:47] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10580552 (10Ladsgroup) I had to recreate the echo tables. It should read from the x1 tables, it doesn't exist in ruwiki (I checked) [21:23:45] 10VPS-project-Wikistats: Add sylwiki to wikistats - https://phabricator.wikimedia.org/T386469#10580756 (10Dzahn) 05Open→03Resolved ` MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ("syl","Sylheti","ᱥᱟᱱᱛᱟᱲᱤ",8); .. dzahn@... [21:25:56] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudgw100[12] - https://phabricator.wikimedia.org/T386810#10580766 (10VRiley-WMF) After trying to rerun it again, I keep getti{F58493864}ng this error (screenshot attached) @cmooney would you have an idea what m... [21:54:51] 10wikitech.wikimedia.org: Audit MediaWiki namespace message overrides for legacy Developer account related information - https://phabricator.wikimedia.org/T376231#10580875 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup [22:04:18] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10580886 (10Ladsgroup) 05In progress→03Resolved The last piece was https://gerrit.wikimedia.org/r/1122662 and now wikitech is fully integrated... [23:59:46] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10581146 (10Bugreporter) >>! In T161859#10580886, @Ladsgroup wrote: > The last piece was https://gerrit.wikimedia.org/r/1122662 and now wikitech i...