[02:28:35] 10Quarry, 06tools-platform-team: Quarry should support connecting to alternative extension databases - https://phabricator.wikimedia.org/T395800#12348175 (10Zache) p:05Low→03High Now when the Wikimedia Commons links tables are moved to separate server i will raise this as high. Relevant toolsreplica ticket... [05:28:59] 10Data-Services, 06tools-infrastructure-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12348378 (10Marostegui) >>! In T438200#12346627, @fnegri wrote: > clouddb1022 is now alerting :/ > >> WARN Memory 95% used. Largest process: mysqld (1497) = 72.7% I don't think this is unexpected a... [05:57:03] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-74 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:07:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-74 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [06:17:03] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-74 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:27:03] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-74 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:09:42] (03update) 10tlepage: global: add deploy events [repos/cloud/toolforge/logs-api] (generalize_logs_write_auth) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) (owner: 10dcaro) [07:09:45] (03update) 10tlepage: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) (owner: 10dcaro) [07:10:51] (03update) 10tlepage: refactor: Parse Loki result streams into logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [07:10:51] 10Data-Services, 06tools-infrastructure-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12348548 (10Marostegui) For what is worth, this is a random host in production: ` PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4487 mysql 20 0 8221.1g 393.0g... [07:11:39] (03merge) 10tlepage: refactor: Parse Loki result streams into logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [07:11:56] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [07:14:19] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.50-20260922071153-5b5450bc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1427 [07:15:32] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [07:16:15] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Documentation: [phase 1]Write tutorial for migrating build service jobs to push-to-deploy - https://phabricator.wikimedia.org/T437477#12348561 (10komla) I have addressed the above. It was a conceptual misunderstanding on my part. The draft is here: http... [07:17:39] 10PAWS, 10Toolforge, 06tools-infrastructure-team, 06Data-Platform-SRE: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062#12348568 (10fgiunchedi) Update: unit status is reported for automount too, meaning SystemdUnitFailed alert in production will be picking it up automatically. I also have u... [07:22:03] !log tlepage@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [07:24:01] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [07:26:47] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [07:32:35] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [07:35:12] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [07:35:57] (03update) 10tlepage: global: add deploy events [repos/cloud/toolforge/logs-api] (generalize_logs_write_auth) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) (owner: 10dcaro) [07:35:59] (03update) 10tlepage: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) (owner: 10dcaro) [07:36:02] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [07:36:09] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [07:38:25] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [07:38:28] (03PS6) 10David Caro: wmcs.ceph.osd.show_info: adapt for long output [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/904770 [07:38:49] (03CR) 10David Caro: wmcs.ceph.osd.show_info: adapt for long output [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/904770 (owner: 10David Caro) [07:38:53] (03CR) 10David Caro: [C:03+2] wmcs.ceph.osd.show_info: adapt for long output [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/904770 (owner: 10David Caro) [07:42:09] (03Merged) 10jenkins-bot: wmcs.ceph.osd.show_info: adapt for long output [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/904770 (owner: 10David Caro) [07:47:28] (03Abandoned) 10David Caro: depool_and_destroy: only zap devices if there were any [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1171540 (owner: 10David Caro) [07:47:58] (03PS3) 10David Caro: wmsc.k8s.reboot: add --skip-drain [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1195629 [07:48:19] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [07:48:52] (03merge) 10tlepage: logs-api: bump to 0.0.50-20260922071153-5b5450bc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1427 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [07:50:57] 10Tool-wikitermbase, 07patch-welcome: Add a button to group citations using {{ترقيم استشهادات}} - https://phabricator.wikimedia.org/T437813#12348675 (10Osps7) Implemented grouped citation support in `Gadget-WikiTerm.js`. Added a "تجميع الاستشهادات" button for each result group, generating a `{{ترقيم استشهادات... [07:57:03] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-74 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:57:23] RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-74 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [08:17:13] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge, 06tools-platform-team: [DRAFT] Decision Request - Initial product approach to integrating Toolforge UI functionality with Toolsadmin - https://phabricator.wikimedia.org/T393010#12348732 (10dcaro) cc. @TLepage-WMF @CCiufo-WMF @aputhin [09:15:13] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-09-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T438504#12348981 (10fnegri) 05In progress→03Resolved @Arcstur thanks for the additional context. Unfortunately bots and scrapers are currently an issue for many Toolf... [09:17:41] (03update) 10waldir: Apply blurred backdrop only to opaque images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/19 [09:41:15] 10Tool-inteGraality: Dashboard-level page with health/trend information - https://phabricator.wikimedia.org/T438830 (10JeanFred) 03NEW [10:02:42] 10Data-Services, 06tools-infrastructure-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12349329 (10fnegri) The same alert was triggering in 2024: {T365164} and at the time I added a note about it in https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting#Wikireplicas_memory_issues_(... [10:39:21] (03CR) 10David Caro: "@andrew please give it a look, otherwise I'll just abandon the patch as I'm not going to work on it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1195629 (owner: 10David Caro) [10:41:00] (03merge) 10taavi: tools: Add k8s backup host [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/128 (https://phabricator.wikimedia.org/T339934) [10:48:42] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools--k8s-backup-1.tools.eqiad1.wikimedia.cloud [10:48:43] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools--k8s-backup-1.tools.eqiad1.wikimedia.cloud [10:48:46] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-backup-1.tools.eqiad1.wikimedia.cloud [10:50:07] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-backup-1.tools.eqiad1.wikimedia.cloud [12:59:36] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [12:59:44] (03update) 10tlepage: global: add deploy events [repos/cloud/toolforge/logs-api] (generalize_logs_write_auth) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) (owner: 10dcaro) [12:59:46] (03update) 10tlepage: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) (owner: 10dcaro) [13:02:58] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] Ldap gets reset when restarting the VM, so existing user SSL certificates are not valid anymore - https://phabricator.wikimedia.org/T427801#12349963 (10fnegri) > toolforge_run_ansible.sh -D --tags ldap Yes, that worked! [13:07:44] 10Cloud-VPS (Project-requests): Request creation of wikivisage VPS project - https://phabricator.wikimedia.org/T438842 (10Premeditated) 03NEW [13:18:35] (03approved) 10fnegri: config: add defaults.alerts.enable setting [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/191 (https://phabricator.wikimedia.org/T432864) (owner: 10dcaro) [13:43:48] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12350162 (10Redmin) [14:03:26] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12350254 (10Andrew) I'm going to try Ben's test, rebuilding osd.334, osd.335, osd.336 and osd.337. For starters I need to... [14:04:37] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [14:04:42] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [14:07:47] (03update) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [14:09:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [14:17:02] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Consider deprecating the 'suspend' VM feature - https://phabricator.wikimedia.org/T431307#12350308 (10Andrew) 05Open→03Resolved suspend, pause and lock are now disallowed for all users. I will be surprised if anyone misses them! [14:20:19] (03update) 10dcaro: config: add defaults.alerts.enable setting [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/191 (https://phabricator.wikimedia.org/T432864) [14:22:18] (03update) 10dcaro: config: add defaults.alerts.enabled setting [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/191 (https://phabricator.wikimedia.org/T432864) [14:22:20] (03update) 10dcaro: config: add defaults.alerts.enabled setting [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/191 (https://phabricator.wikimedia.org/T432864) [14:22:26] (03update) 10dcaro: config: add defaults.alerts.enabled setting [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/191 (https://phabricator.wikimedia.org/T432864) [14:25:30] (03update) 10dcaro: config: add defaults.alerts.enabled setting [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/191 (https://phabricator.wikimedia.org/T432864) [14:27:56] (03merge) 10dcaro: config: add defaults.alerts.enabled setting [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/191 (https://phabricator.wikimedia.org/T432864) [14:31:10] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.217-20260922142808-269f594b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1428 (https://phabricator.wikimedia.org/T432864) [14:32:13] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [14:33:08] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12350449 (10Redmin) [14:34:36] 10VPS-project-Codesearch, 06Infrastructure-Foundations, 13Patch-For-Review: Codesearch can't build: ERROR: golang1.15 not found - https://phabricator.wikimedia.org/T438777#12350478 (10Krinkle) @MoritzMuehlenhoff Ack. In the future I suggest checking Codesearch before deleting images from the registry, so tha... [14:44:15] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12350566 (10Redmin) [14:44:43] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12350574 (10Redmin) p:05Medium→03High [14:45:21] (03approved) 10fnegri: components-api: bump to 0.0.217-20260922142808-269f594b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1428 (https://phabricator.wikimedia.org/T432864) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:47:40] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12350628 (10Redmin) [15:08:59] (03CR) 10Andrew Bogott: "This looks just fine to me! A sentence or two about when you'd use this might be nice, either in the commit comment or the help string." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1195629 (owner: 10David Caro) [15:16:53] 10Tool-delintbot: Avoid logical edit conflicts - https://phabricator.wikimedia.org/T438870 (10Redmin) 03NEW [15:17:10] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [15:17:21] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [15:20:54] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [15:39:05] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [15:46:31] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wmcs-uptime away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435759#12350972 (10komla) @TheresNoTime any luck on progress with this? [15:48:00] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wikidocumentaries away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435789#12350973 (10komla) @Zache noted. [15:51:47] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate etytree away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435770#12351001 (10komla) @epanteleo any update? I've also followed up on our email thread. Please let me know if you are having any challenges. [16:03:30] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [16:03:31] (03update) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [16:03:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:03:48] (03open) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [16:03:49] (03open) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [16:04:20] 10PAWS, 06tools-platform-team: [paws] Do a 1h check with @fnegri on bad behavior detection - https://phabricator.wikimedia.org/T438364#12351071 (10aputhin) 05In progress→03Resolved [16:07:30] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [16:08:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:12:10] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [16:13:08] (03approved) 10dcaro: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [16:17:59] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [16:19:26] (03PS4) 10David Caro: wmsc.k8s.reboot: add --skip-drain [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1195629 [16:19:41] (03CR) 10David Caro: [C:03+2] "Done" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1195629 (owner: 10David Caro) [16:20:20] 10Tool-wikiloves: Wikiloves statistics are not being updated - https://phabricator.wikimedia.org/T437822#12351160 (10Nemoralis) Tested the new database layout locally, works fine. {F104323951} [16:20:48] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [16:21:10] (03merge) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [16:21:14] (03update) 10tlepage: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) (owner: 10dcaro) [16:21:45] (03update) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [16:22:58] (03Merged) 10jenkins-bot: wmsc.k8s.reboot: add --skip-drain [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1195629 (owner: 10David Caro) [16:23:30] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [16:24:03] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.51-20260922162121-2e392b69 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1429 [16:24:03] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.51-20260922162121-2e392b69 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1429 [16:24:31] 06tools-infrastructure-team, 13Patch-For-Review: Track toolforge virtualization footprint: ram/cores/instance &c - https://phabricator.wikimedia.org/T436275#12351213 (10Andrew) Here's a summary dashboard that should tell us most of what we need for hardware purchase planning. Not a lot of growth lately! https:... [16:25:47] (03approved) 10dcaro: components-api: bump to 0.0.217-20260922142808-269f594b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1428 (https://phabricator.wikimedia.org/T432864) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:25:56] (03merge) 10dcaro: components-api: bump to 0.0.217-20260922142808-269f594b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1428 (https://phabricator.wikimedia.org/T432864) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:26:09] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [16:26:23] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12351219 (10dcaro) 05In progress→03Resolved [16:26:52] (03update) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [16:29:11] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: CloudVPSDesignateLeaks + managed_resource_id - https://phabricator.wikimedia.org/T402457#12351258 (10Andrew) 05Open→03Resolved The bullseye deprecation has purged almost all affected VMs. There are probably a few left that were upgraded... [16:31:18] !log tlepage@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [16:33:40] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12351292 (10dcaro) Looks better, a few other notes: {F104324012} I would not say 'components service configuration', just tool configuration. The command is missin... [16:33:43] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12351294 (10dcaro) 05Open→03In progress [16:34:07] (03update) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [16:39:28] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [16:41:31] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [16:41:40] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [16:44:38] !log tlepage@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [16:47:39] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [16:48:21] (03update) 10dcaro: logs-api: bump to 0.0.51-20260922162121-2e392b69 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1429 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:50:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [16:52:48] !log tlepage@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [16:56:33] (03open) 10dcaro: poetry: add uvicorn to the main dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/44 [16:57:57] (03approved) 10tlepage: poetry: add uvicorn to the main dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/44 (owner: 10dcaro) [17:00:39] (03approved) 10dcaro: poetry: add uvicorn to the main dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/44 [17:00:47] (03merge) 10dcaro: poetry: add uvicorn to the main dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/44 [17:03:36] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.52-20260922170107-5d68238b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1429 [17:03:55] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.52-20260922170107-5d68238b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1429 [17:04:12] (03update) 10tlepage: logs-api: bump to 0.0.52-20260922170107-5d68238b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1429 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:04:14] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [17:09:24] !log tlepage@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [17:12:30] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [17:17:40] !log tlepage@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [17:23:17] (03open) 10tlepage: explicitly add httpx back too [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/45 [17:23:42] (03update) 10tlepage: explicitly add httpx back too [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/45 [17:24:42] (03update) 10tlepage: explicitly add httpx back too [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/45 [17:25:48] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12351621 (10fnegri) [17:26:25] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12351624 (10fnegri) 05Open→03In progress [17:51:28] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [17:51:31] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [17:51:37] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [17:53:04] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [18:09:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:19:24] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [19:19:32] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [19:19:53] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [19:30:26] (03PS1) 10MusikAnimal: Repository: Make executeQueryBuilder apply the timeout and the breaker [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344066 (https://phabricator.wikimedia.org/T437830) [19:30:29] (03PS1) 10MusikAnimal: Repository: Route links-table queries to their own replica section [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344067 (https://phabricator.wikimedia.org/T437830) [19:30:31] (03PS1) 10MusikAnimal: PageInfo, Page: Read links tables from the links connection [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344068 (https://phabricator.wikimedia.org/T437830) [19:30:33] (03PS1) 10MusikAnimal: CategoryEdits: Intersect categories and revisions in code [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344069 (https://phabricator.wikimedia.org/T437830) [19:33:29] (03PS2) 10MusikAnimal: Repository: Make executeQueryBuilder apply the timeout and the breaker [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344066 (https://phabricator.wikimedia.org/T437830) [19:33:29] (03PS2) 10MusikAnimal: Repository: Route links-table queries to their own replica section [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344067 (https://phabricator.wikimedia.org/T437830) [19:33:29] (03PS2) 10MusikAnimal: PageInfo, Page: Read links tables from the links connection [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344068 (https://phabricator.wikimedia.org/T437830) [19:33:30] (03PS2) 10MusikAnimal: CategoryEdits: Intersect categories and revisions in code [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344069 (https://phabricator.wikimedia.org/T437830) [19:35:28] (03PS3) 10MusikAnimal: Repository: Make executeQueryBuilder apply the timeout and the breaker [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344066 (https://phabricator.wikimedia.org/T437830) [19:35:28] (03PS3) 10MusikAnimal: Repository: Route links-table queries to their own replica section [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344067 (https://phabricator.wikimedia.org/T437830) [19:35:28] (03PS3) 10MusikAnimal: PageInfo, Page: Read links tables from the links connection [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344068 (https://phabricator.wikimedia.org/T437830) [19:35:29] (03PS3) 10MusikAnimal: CategoryEdits: Intersect categories and revisions in code [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344069 (https://phabricator.wikimedia.org/T437830) [19:44:50] (03open) 10lucaswerkmeister: toolforge-cd: fix deployment description on force-push [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/100 [19:47:26] (03merge) 10taavi: toolforge-cd: fix deployment description on force-push [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/100 (owner: 10lucaswerkmeister) [20:31:47] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 06Product Safety and Integrity, 10Temporary accounts: Check impact of Temporary Accounts on Toolforge tools - https://phabricator.wikimedia.org/T378516#12352484 (10Dreamy_Jazz) 05Open→03Resolved Given temporary accounts are everywhere, I... [20:37:24] 10VPS-Projects, 06Product Safety and Integrity, 10Temporary accounts: Prepare UTRS for temporary accounts - https://phabricator.wikimedia.org/T351028#12352515 (10Dreamy_Jazz) 05Open→03Resolved a:03Dreamy_Jazz @AmandaNP do you still need this task? Given temporary accounts are now on all wikis, I wo... [20:37:30] 10VPS-Projects, 06Product Safety and Integrity, 10Temporary accounts: Prepare UTRS for temporary accounts - https://phabricator.wikimedia.org/T351028#12352521 (10Dreamy_Jazz) a:05Dreamy_Jazz→03None [20:42:55] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-rdb01 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436480#12352573 (10Southparkfan) 05Open→03In progress a:03Southparkfan [20:49:34] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [20:49:44] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [21:24:03] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-cumin-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436470#12352721 (10Southparkfan) 05In progress→03Resolved ✅ [21:49:49] 10Tool-wsindex, 06MediaWiki-Media-Platform-Team, 10Wikipedia-iOS-App, 06WikipediaApp, and 2 others: Update apps to support for the move from upload.wikimedia.org to thumb.wikimedia.org - https://phabricator.wikimedia.org/T434821#12352826 (10Ladsgroup) >>! In T434821#12346350, @Seddon wrote: > @Ladsgroup ar... [22:16:04] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12352928 (10Andrew) ` # ceph device ls | grep cloudcephosd1046 MTFDDAK3T8TGA-1BC1ZABDA_242349549459 clou... [22:50:55] 10Toolforge, 06tools-platform-team: [jobs-api] Allow storing one-off jobs - https://phabricator.wikimedia.org/T433868#12352983 (10connorshea) > Thanks for the input! > If you don't mind, I have a couple questions, do you store the one-off definitions anywhere? > Would `toolforge webservice shell` command do? (... [23:13:26] 10Tool-Gerrit-Patch-Uploader: "Project" dropdown is empty, preventing patch submission - https://phabricator.wikimedia.org/T436321#12353031 (10bd808) This is functionally the same issue as {T434075}. The python code tries to use `ssh gerrit gerrit ls-projects` to get the list of known projects from gerrit. This... [23:25:19] 10Tool-Gerrit-Patch-Uploader: "Project" dropdown is empty, preventing patch submission - https://phabricator.wikimedia.org/T436321#12353068 (10bd808) `lang=shell-session tools.gerrit-patch-uploader@tools-bastion-14:~$ ssh-keygen -f '/data/project/gerrit-patch-uploader/.ssh/known_hosts' -R '[gerrit.wikimedia.org]... [23:26:49] 10Tool-Gerrit-Patch-Uploader: "Project" dropdown is empty, preventing patch submission - https://phabricator.wikimedia.org/T436321#12353069 (10bd808) 05Open→03Resolved a:03bd808 {F104374077,size=full}