[00:07:39] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12372176 (10Doc_James) Sounds good J [00:13:07] RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [00:15:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [00:25:07] RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [00:36:52] 10Tools: eatchabot using a lot of NFS storage - https://phabricator.wikimedia.org/T284968#12372213 (10Andrew) 05Open→03Resolved a:03Andrew This is down to 8.5G -- not bad! [00:36:54] 06cloud-services-team, 10Toolforge, 06tools-platform-team: tools-nfs-2 almost out of disk space (October 2023 edition) - https://phabricator.wikimedia.org/T349895#12372219 (10Andrew) [00:36:55] 06cloud-services-team, 10Toolforge, 06tools-platform-team: 2024-02-27: toolforge NFS cleanup - https://phabricator.wikimedia.org/T358554#12372217 (10Andrew) [00:36:58] 14cloud-services-team (Kanban), 10Data-Services, 06tools-platform-team: 2021-06-15: Tools NFS share cleanup - https://phabricator.wikimedia.org/T284964#12372222 (10Andrew) [00:43:44] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Improving toolforge NFS patrol - https://phabricator.wikimedia.org/T439494 (10Andrew) 03NEW [00:47:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [00:48:18] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [00:48:26] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [00:50:32] (03PS1) 10Andrea Denisse: kubernetes: Add dummy KLAXON_CORTO_TOKEN for oncall [labs/private] - 10https://gerrit.wikimedia.org/r/1345674 (https://phabricator.wikimedia.org/T439495) [00:56:33] 06tools-platform-team: components-api: pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496 (10Raymond_Ndibe) 03NEW [00:56:49] 06tools-platform-team: components-api: pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12372281 (10Raymond_Ndibe) a:03Raymond_Ndibe [01:35:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [01:35:19] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [01:36:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [01:36:05] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [01:39:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [01:39:19] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [01:42:00] PROBLEM - Host cloudcephosd1045 is DOWN: PING CRITICAL - Packet loss = 100% [01:42:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [01:44:23] ACKNOWLEDGEMENT - SSH on cloudcephosd1045 is CRITICAL: CRITICAL - Socket timeout after 10 seconds Andrew Bogott rebooting and re-partitioning https://wikitech.wikimedia.org/wiki/SSH/monitoring [01:44:23] ACKNOWLEDGEMENT - Host cloudcephosd1045 is DOWN: PING CRITICAL - Packet loss = 100% Andrew Bogott rebooting and re-partitioning [01:44:30] RECOVERY - Host cloudcephosd1045 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [01:46:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [01:47:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [01:50:02] PROBLEM - Host cloudcephosd1045 is DOWN: PING CRITICAL - Packet loss = 100% [01:52:30] RECOVERY - Host cloudcephosd1045 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [01:53:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [01:56:45] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [02:00:22] 06tools-platform-team: components-api: pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12372314 (10Raymond_Ndibe) 05Open→03In progress [02:01:26] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [toollabs-images] stop hardcoding port 8000 in the webservice-runner scripts - https://phabricator.wikimedia.org/T432078#12372318 (10Raymond_Ndibe) 05In progress→03Resolved [02:10:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [05:43:23] 10Tool-delintbot: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12372497 (10Redmin) p:05High→03Medium [06:21:24] 10Cloud-VPS, 06tools-infrastructure-team, 10Wikispeech-Text-to-Speech, 13Patch-For-Review: Error when creating snapshot for instance - https://phabricator.wikimedia.org/T439363#12372552 (10Sebastian_Berlin-WMSE) >>! In T439363#12371321, @Andrew wrote: > That said... once policy enforcement is sorted out we... [07:09:56] (03open) 10arthurtaylor: Add ticket loop tracking, jasmine tests [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/4 [07:10:39] (03update) 10arthurtaylor: Add ticket loop tracking, jasmine tests [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/4 [07:11:29] (03merge) 10arthurtaylor: Add ticket loop tracking, jasmine tests [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/4 [07:15:47] (03open) 10arthurtaylor: Fixup wording [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/5 [07:15:52] (03approved) 10arthurtaylor: Fixup wording [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/5 [07:15:55] (03merge) 10arthurtaylor: Fixup wording [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/5 [07:18:06] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12372676 (10fgiunchedi) I set jbod for cloudcephosd1055 via the mgmt web interface, and tried cephosd-efi recipe, which prompted me for a partitioning choi... [08:00:23] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create continuous job deployment tutorial - https://phabricator.wikimedia.org/T437903#12372771 (10komla) Combined this with the scheduled job tutorial into one tutorial: T437901 [08:01:24] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create scheduled job deployment tutorial - https://phabricator.wikimedia.org/T437901#12372780 (10komla) Link here: https://wikitech.wikimedia.org/wiki/User:SSapaty_(WMF)/PTD/scheduled_and_continuous_jobs_with_toolforge_components [08:06:12] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [o11y,k8s,infra] Current stats used for capacity are not reliable enough - https://phabricator.wikimedia.org/T439512 (10dcaro) 03NEW p:05Triage→03High [08:06:34] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [o11y,k8s,infra] Current stats used for capacity are not reliable enough - https://phabricator.wikimedia.org/T439512#12372813 (10dcaro) Also related {T390228} [08:08:10] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [o11y,k8s,infra] Current stats used for capacity are not reliable enough - https://phabricator.wikimedia.org/T439512#12372815 (10dcaro) The output of kubectl describe nodes: ` dcaro@acme$ cat resources.txt | grep cpu cpu 1050m... [08:09:52] 10Toolforge, 06tools-platform-team: components-api: pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12372819 (10taavi) [08:23:32] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [08:24:19] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [08:30:55] 10Cloud-VPS (Project-requests), 06tools-platform-team: Request creation of wikivisage VPS project - https://phabricator.wikimedia.org/T438842#12372881 (10dcaro) 05Open→03Resolved a:03dcaro >>! In T438842#12365039, @Premeditated wrote: > I set up `wikivisage-face` as a dedicated tool on Toolforge. The... [08:34:27] (03approved) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [08:34:41] (03merge) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [08:37:12] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.55-20260929083455-54dd3b40 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1435 (https://phabricator.wikimedia.org/T432571) [08:37:24] (03PS1) 10Slyngshede: P:idp add missing dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1345925 [08:38:08] (03approved) 10dcaro: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) (owner: 10fnegri) [08:38:28] (03CR) 10Slyngshede: [V:03+2 C:03+2] P:idp add missing dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1345925 (owner: 10Slyngshede) [08:41:35] 10Toolforge, 06tools-platform-team: [components-api] return `event_type` for run-logs - https://phabricator.wikimedia.org/T439517 (10dcaro) 03NEW [08:41:38] 10Toolforge, 06tools-platform-team: [components-api] return `event_type` for run-logs - https://phabricator.wikimedia.org/T439517#12372942 (10dcaro) p:05Triage→03High a:03dcaro [08:46:26] 10Cloud-VPS (Project-requests), 06tools-platform-team: Request creation of wikivisage VPS project - https://phabricator.wikimedia.org/T438842#12372947 (10JJMC89) 05Resolved→03Declined [08:46:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [08:48:56] 10Toolforge, 06tools-platform-team: [components-api] return `source` for all logs - https://phabricator.wikimedia.org/T439517#12372950 (10dcaro) [08:59:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [08:59:52] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:12:26] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [09:15:40] (03approved) 10dcaro: logs-api: bump to 0.0.55-20260929083455-54dd3b40 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1435 (https://phabricator.wikimedia.org/T432571) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:15:46] (03merge) 10dcaro: logs-api: bump to 0.0.55-20260929083455-54dd3b40 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1435 (https://phabricator.wikimedia.org/T432571) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:21:54] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Improving toolforge NFS patrol - https://phabricator.wikimedia.org/T439494#12373108 (10Volans) For adding quotas (reporting only first) basically there are two options: 1. legacy external quota files with `quotacheck` and `quotaon`, it should... [09:28:54] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12373128 (10dcaro) 05In progress→03Resolved Deployed [09:30:17] (03open) 10dcaro: global: add source to all logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T439517) [09:43:47] 10Toolforge, 06tools-platform-team: [logs-cli] support run-logs and deploy events - https://phabricator.wikimedia.org/T439528 (10dcaro) 03NEW [09:43:57] (03open) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [09:45:04] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, and 2 others: [toolforge,storage,infra,k8s] Investigate persistent volume support - https://phabricator.wikimedia.org/T384596#12373259 (10dcaro) [09:45:17] 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [toolforge,storage] Add storage capabilities for tools - https://phabricator.wikimedia.org/T293670#12373260 (10dcaro) [09:53:00] 10Toolforge, 06tools-platform-team: [logs-api] Add message to deployment failed event - https://phabricator.wikimedia.org/T439530 (10dcaro) 03NEW [10:11:32] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12373389 (10fgiunchedi) >>! In T419892#12372676, @fgiunchedi wrote: > I then looked at cloudcephosd and I don't see any reason why th... [10:12:41] (03open) 10dcaro: api_models: add message to the deployment failed event [repos/cloud/toolforge/logs-api] (add_source_to_all) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/47 (https://phabricator.wikimedia.org/T439530) [10:17:11] (03update) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [10:30:08] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Improving toolforge NFS patrol - https://phabricator.wikimedia.org/T439494#12373539 (10fgiunchedi) I'm +1 on reporting space usage like @Volans mentioned. And if we report those via metrics then we can get per-tool alerts, and notify maintainer... [10:46:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [11:18:01] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36560726793 (https://github.com/cluebotng/component-configs/commits/8638ec82624573a20aa5fbc7d9b44374a34b35e4) [11:18:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [11:38:18] (03update) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [11:42:40] (03update) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [11:54:23] (03close) 10dcaro: api_models: add message to the deployment failed event [repos/cloud/toolforge/logs-api] (add_source_to_all) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/47 (https://phabricator.wikimedia.org/T439530) [11:54:45] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [logs-api] Add message to deployment failed event - https://phabricator.wikimedia.org/T439530#12373899 (10dcaro) 05Open→03Invalid [11:55:07] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [logs-cli] support run-logs and deploy events - https://phabricator.wikimedia.org/T439528#12373900 (10dcaro) p:05Triage→03High [12:05:41] (03update) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [12:06:11] (03update) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [12:08:19] 10Tool-delintbot: Replace tt tags with span tag that uses the monospace font - https://phabricator.wikimedia.org/T439546 (10Redmin) 03NEW [12:08:38] 10Tool-delintbot: Replace tt tags with span tag that uses the monospace font - https://phabricator.wikimedia.org/T439546#12373959 (10Redmin) p:05Triage→03Medium [12:09:32] 10Tool-delintbot: Replace tt tags with span tags that uses the monospace font - https://phabricator.wikimedia.org/T439546#12373960 (10Redmin) [12:16:55] (03update) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [12:39:58] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12374073 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1004 for host cloudcephosd1055.eqiad.wmnet with OS bookworm [12:42:15] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [12:42:22] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [12:53:40] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12374148 (10fnegri) [13:00:46] 10Toolforge (Push-to-Deploy), 06tools-platform-team: components-api: pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12374180 (10Raymond_Ndibe) [13:06:04] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12374201 (10Andrew) [] cloudcephosd1016 [] cloudcephosd1017 [] cloudcephosd1018 [] cloudcephosd1019 [] cloudcephosd1020 []... [13:10:22] 10Tool-delintbot: Preserve pre, samp, kbd and var tag content - https://phabricator.wikimedia.org/T439553 (10Redmin) 03NEW [13:10:41] 10Tool-delintbot: Preserve pre, samp, kbd and var tag content - https://phabricator.wikimedia.org/T439553#12374249 (10Redmin) p:05Triage→03Low [13:12:45] 10Tool-delintbot: Replace tt tags with span tags that uses the monospace font - https://phabricator.wikimedia.org/T439546#12374265 (10Redmin) 05Open→03Resolved a:03Redmin Done in https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/commit/ee6a5f38b185794aeb089b0aa1781e979afe4906. [13:15:54] 10Toolforge, 06tools-platform-team, 07OKR-Work: Read the opted-in tools from components-api - https://phabricator.wikimedia.org/T439554 (10fnegri) 03NEW [13:16:15] 10Toolforge, 06tools-platform-team, 07OKR-Work: Read the opted-in tools from components-api - https://phabricator.wikimedia.org/T439554#12374288 (10fnegri) p:05Triage→03High [13:16:24] 10Toolforge, 06tools-platform-team, 07OKR-Work: Read the opted-in tools from components-api - https://phabricator.wikimedia.org/T439554#12374289 (10fnegri) [13:16:28] 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12374290 (10fnegri) [13:16:40] 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12374297 (10fnegri) [13:16:41] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12374298 (10fnegri) [13:18:11] 10Toolforge, 06tools-platform-team, 07OKR-Work: Read the opted-in tools from components-api - https://phabricator.wikimedia.org/T439554#12374303 (10fnegri) 05Open→03In progress [13:20:49] 10Tool-Global-user-contributions, 07PHP 8.5 support: labs/tools/guc fails its CI tests in PHP 8.5 due to old Phan install - https://phabricator.wikimedia.org/T419080#12374331 (10Jdforrester-WMF) Thank you! [13:20:57] (03approved) 10dcaro: maintainer-alerts: remove tool- prefix [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/72 (https://phabricator.wikimedia.org/T437081) (owner: 10fnegri) [13:34:34] 06tools-infrastructure-team: Track toolforge virtualization footprint: ram/cores/instance &c - https://phabricator.wikimedia.org/T436275#12374390 (10Andrew) [13:34:37] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [o11y,k8s,infra] Current stats used for capacity are not reliable enough - https://phabricator.wikimedia.org/T439512#12374391 (10Andrew) [13:36:00] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:37:32] (03open) 10r4356th: Avoid logical edit conflicts [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/23 (https://phabricator.wikimedia.org/T438870) [13:37:40] (03update) 10r4356th: Avoid logical edit conflicts [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/23 (https://phabricator.wikimedia.org/T438870) [13:37:46] (03merge) 10r4356th: Avoid logical edit conflicts [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/23 (https://phabricator.wikimedia.org/T438870) [13:38:29] 10Tool-delintbot, 13Patch-For-Review: Avoid logical edit conflicts - https://phabricator.wikimedia.org/T438870#12374414 (10Redmin) 05Open→03Resolved a:03Redmin [13:53:37] (03open) 10dcaro: deploy: add deploy.sh script [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/9 [13:54:23] (03update) 10dcaro: deploy: add deploy.sh script [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/9 [13:55:32] (03open) 10dcaro: deploy.sh: remove misleading comment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/379 [13:59:22] 10Cloud-VPS, 06tools-infrastructure-team, 10Wikispeech-Text-to-Speech, 13Patch-For-Review: Error when creating snapshot for instance - https://phabricator.wikimedia.org/T439363#12374530 (10Andrew) > Is there a timeline for when instance snapshots could be enabled? Not at the moment -- this isn't on our r... [14:02:30] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] CreateSnapshot: Fix policy check [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1345644 (https://phabricator.wikimedia.org/T439363) (owner: 10Andrew Bogott) [14:04:48] (03open) 10r4356th: Preserve pre, samp, kbd and var tag content [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/24 (https://phabricator.wikimedia.org/T439553) [14:04:50] (03update) 10r4356th: Preserve pre, samp, kbd and var tag content [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/24 (https://phabricator.wikimedia.org/T439553) [14:05:07] (03merge) 10r4356th: Preserve pre, samp, kbd and var tag content [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/24 (https://phabricator.wikimedia.org/T439553) [14:05:22] (03update) 10r4356th: Preserve pre, samp, kbd and var tag content [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/24 (https://phabricator.wikimedia.org/T439553) [14:05:28] (03open) 10andrew: Update Horizon submodule version [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/11 [14:05:34] (03merge) 10andrew: Update Horizon submodule version [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/11 [14:06:03] 10Tool-delintbot: Preserve pre, samp, kbd and var tag content - https://phabricator.wikimedia.org/T439553#12374573 (10Redmin) 05Open→03Resolved [14:06:48] 10Tool-delintbot: Avoid logical edit conflicts - https://phabricator.wikimedia.org/T438870#12374586 (10Redmin) [14:07:53] (03approved) 10fnegri: deploy.sh: remove misleading comment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/379 (owner: 10dcaro) [14:08:12] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Improving toolforge NFS patrol - https://phabricator.wikimedia.org/T439494#12374595 (10taavi) For the first two: Nuking big log files gives you that space back temporarily, but it doesn't do anything about the process that's generating the log... [14:08:16] (03approved) 10fnegri: deploy: add deploy.sh script [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/9 (owner: 10dcaro) [14:24:08] 10Cloud-Services: Horizon Trove Database database Configurations Parameter field is empty - https://phabricator.wikimedia.org/T439562 (10WikiBayer) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replac... [14:42:07] RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [14:42:37] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [14:50:45] 10Cloud-VPS, 06tools-infrastructure-team: Horizon Trove Database database Configurations Parameter field is empty - https://phabricator.wikimedia.org/T439562#12374916 (10WikiBayer) [14:51:32] 10Horizon, 06tools-infrastructure-team: Horizon Trove Database database Configurations Parameter field is empty - https://phabricator.wikimedia.org/T439562#12374919 (10JJMC89) [15:14:51] (03merge) 10dcaro: deploy.sh: remove misleading comment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/379 [15:15:05] (03merge) 10dcaro: deploy: add deploy.sh script [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/9 [15:17:20] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.7-20260929151521-c2f499b8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1436 [15:17:20] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.7-20260929151521-c2f499b8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1436 [15:19:25] 10Toolforge, 06tools-platform-team: [jobs-api] Make pre-built image webservices move all logs to logs-api based - https://phabricator.wikimedia.org/T439574 (10dcaro) 03NEW [15:19:39] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.572-20260929151505-9f6db6d8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1437 [15:19:39] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.572-20260929151505-9f6db6d8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1437 [15:20:08] 10Toolforge, 06tools-platform-team: [jobs-api] Make pre-built image webservices move all logs to logs-api based - https://phabricator.wikimedia.org/T439574#12375090 (10taavi) →14Duplicate dup:03T401102 [15:20:15] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Prebuilt webservice containers should not log to NFS by default - https://phabricator.wikimedia.org/T401102#12375092 (10taavi) [15:20:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:25:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:33:42] (03PS2) 10Majavah: python: Log to stdout/stderr only [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1226300 (https://phabricator.wikimedia.org/T401102) [15:33:51] (03PS2) 10Majavah: lighttpd: Log to stdout/stderr only [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1226301 (https://phabricator.wikimedia.org/T401102) [15:40:11] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Prebuilt webservice containers should not log to NFS by default - https://phabricator.wikimedia.org/T401102#12375213 (10dcaro) This depends on us moving all the webservices to jobs-api though, linking it [15:40:43] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Prebuilt webservice containers should not log to NFS by default - https://phabricator.wikimedia.org/T401102#12375216 (10dcaro) >>! In T401102#12375213, @dcaro wrote: > This depends on us moving all the webservices to jobs-api th... [15:55:43] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [15:55:44] (03update) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [15:55:52] (03update) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [15:55:58] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [16:03:48] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:06:30] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [components-api] return `source` for all logs - https://phabricator.wikimedia.org/T439517#12375438 (10dcaro) [16:06:42] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli] support run-logs and deploy events - https://phabricator.wikimedia.org/T439528#12375440 (10dcaro) [16:07:31] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-cli, jobs-api] support --webservice option - https://phabricator.wikimedia.org/T428898#12375442 (10aputhin) 05In progress→03Resolved [16:08:04] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] show a warning message stating it's in beta/unstable - https://phabricator.wikimedia.org/T439581 (10dcaro) 03NEW [16:13:06] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [16:14:08] (03approved) 10dcaro: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) (owner: 10fnegri) [16:14:54] (03update) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/21 [16:14:57] (03open) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/21 [16:15:14] (03update) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/33 [16:15:20] (03open) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/33 [16:15:33] (03update) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/28 [16:15:37] (03open) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/28 [16:16:35] (03update) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/60 [16:16:41] (03open) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/60 [16:17:35] (03update) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/45 [16:17:42] (03open) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/45 [16:27:16] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [16:27:30] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [16:27:47] (03update) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [16:27:56] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [16:28:11] (03update) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [16:29:18] (03update) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [16:29:39] (03merge) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [16:29:41] (03update) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [16:31:18] (03update) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [16:31:58] (03merge) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [16:32:01] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [16:32:01] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.8-20260929162949-8bda6ff7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1436 (https://phabricator.wikimedia.org/T432865) [16:32:09] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.8-20260929162949-8bda6ff7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1436 (https://phabricator.wikimedia.org/T432865) [16:33:23] (03approved) 10dcaro: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) (owner: 10fnegri) [16:34:06] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.9-20260929163209-a3bb1242 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1436 (https://phabricator.wikimedia.org/T432865) [16:34:17] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.9-20260929163209-a3bb1242 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1436 (https://phabricator.wikimedia.org/T432865) [16:34:17] (03CR) 10FNegri: [C:03+1] deploy,gitlab: add logs-cli [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341237 (https://phabricator.wikimedia.org/T432572) (owner: 10David Caro) [16:35:01] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12375644 (10fnegri)... [16:54:27] (03update) 10r4356th: Add ty for checking types [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/18 [16:54:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:56:52] (03update) 10r4356th: Add ty for checking types [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/18 [16:57:30] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12375745 (10dcaro) >... [16:58:26] (03CR) 10David Caro: [C:03+2] deploy,gitlab: add logs-cli [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341237 (https://phabricator.wikimedia.org/T432572) (owner: 10David Caro) [17:01:45] (03Merged) 10jenkins-bot: deploy,gitlab: add logs-cli [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341237 (https://phabricator.wikimedia.org/T432572) (owner: 10David Caro) [17:04:04] 10Cloud-VPS, 06tools-infrastructure-team: Trove UI doesn't work very well for deprecated datastore versions - https://phabricator.wikimedia.org/T439593 (10Andrew) 03NEW [17:04:22] (03update) 10r4356th: Add ty for checking types [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/18 [17:06:41] (03update) 10r4356th: Add ty for checking types [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/18 [17:08:58] (03update) 10r4356th: Add ty for checking types [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/18 [17:10:37] (03update) 10r4356th: Add ty for checking types [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/18 [17:10:40] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [17:11:39] (03merge) 10r4356th: Add ty for checking types [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/18 [17:14:06] (03open) 10fnegri: alerts-api: Use the root URL for prometheus hosts [repos/cloud/toolforge/toolforge-deploy] (bump_alerts-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1438 (https://phabricator.wikimedia.org/T432865) [17:14:49] (03close) 10fnegri: alerts-api: Use the root URL of each prometheus host [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1434 (https://phabricator.wikimedia.org/T432865) [17:14:59] 10Tool-wikinewsie: Wikinewsie link to main updated section of Wikipedia article - https://phabricator.wikimedia.org/T431962#12375844 (10Chimaifeanyi29) a:05Chimaifeanyi29→03None [17:23:15] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [17:23:15] (03update) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [17:23:27] (03update) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [17:23:34] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [17:24:22] 10Toolforge, 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12375917 (10BLiviero-WMF) I would still encourage examining what was running on the worker(s); and figuring out how often this happens, and if it's the same culprits. This would better inf... [17:45:53] 10Toolforge, 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12376044 (10Andrew) I don't think we have a good way of knowing what was scheduled on the node at the time of failure. And, anyway -- is identifying the specific culprit useful? Our kubernet... [17:58:13] 10Tool-delintbot: Handle cases of extension tags with template parameters - https://phabricator.wikimedia.org/T416008#12376101 (10Redmin) p:05Low→03Medium [17:58:14] 10Toolforge, 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12376103 (10Andrew) "Memory cgroup out of memory" is firing all over the dang place -- I just grepped syslogs throughout toolforge for "Memory cgroup out of memory: Killed" and found it on 2... [18:07:46] 10Toolforge, 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12376245 (10Andrew) In the attached log from -74, behavior is similar. A container running as k8s-status is getting killed and re-run in the same place over and over. Eventually the noise sh... [18:14:17] (03update) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/21 [18:14:59] (03update) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/33 [18:16:26] (03update) 10renovatebot: Update pnpm to v12.8.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/60 [18:18:24] 10Tool-Pageviews, 06Moderator-Tools-Team, 10XTools: Multiple wmcloud.org tools returning 503 / API errors (XTools & Pageviews Analysis) - https://phabricator.wikimedia.org/T439614 (10Eduardo7z) 03NEW [18:24:46] 10Toolforge, 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12376313 (10Andrew) OK, I'm learning as a go here :) That "Memory cgroup out of memory: Killed" message is the cgroups working properly: a process in a pod has run over its limits and is bei... [18:36:36] 10Tool-Pageviews, 06Moderator-Tools-Team, 10XTools: Multiple wmcloud.org tools returning 503 / API errors (XTools & Pageviews Analysis) - https://phabricator.wikimedia.org/T439614#12376348 (10MusikAnimal) 05Open→03In progress a:03MusikAnimal Pageviews was apparently failing because it was out of disk s... [18:52:21] (03merge) 10ahecht: Query globalimagelinks on the new x4 links cluster [toolforge-repos/globalusagecount] - 10https://gitlab.wikimedia.org/toolforge-repos/globalusagecount/-/merge_requests/1 (owner: 10pacmand) [19:23:44] 10Toolforge, 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12376550 (10Andrew) Loki logs started failing on 2026-09-12; do we think this node was working at all after that day? [19:25:30] 10Tool-Pageviews, 06Moderator-Tools-Team, 10XTools: Multiple wmcloud.org tools returning 503 / API errors (XTools & Pageviews Analysis) - https://phabricator.wikimedia.org/T439614#12376553 (10MusikAnimal) 05In progress→03Resolved I didn't add the new X4 connection after the last deployment :/ Sorry!... [19:57:05] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:02:05] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:41:16] 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T435704#12376921 (10LibUp-bot) A new upstream version of Pywikibot is now available: 11.8.0. * https://wikitech.wikimedia.org/wiki/PAWS/Admin#Update_the_pywikibot_version * https://gerrit.wikimedia.org/g/pywik... [20:41:17] 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T439635 (10LibUp-bot) 03NEW [21:58:35] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-rdb01 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436480#12377213 (10Southparkfan) 05In progress→03Resolved Turns out removing the hack was easier than anticipated. [22:04:40] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [22:04:47] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [22:28:30] 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12377278 (10tchin) `http://stream.wikimedia.beta.wmcloud.org` and `http://str... [22:55:20] 10Tool-wikinewsie, 06WikiNYC Tech Committee: Set up Wikinewsie tool as alpha release on Toolforge - https://phabricator.wikimedia.org/T427575#12377353 (10Pharos) This is now complete, with the ceremonial alpha release on September 26, 2026, as part of my WikiConference North America presentation! The latest im... [22:57:09] 10Tool-wikinewsie, 06WikiNYC Tech Committee: Set up Wikinewsie tool as alpha release on Toolforge - https://phabricator.wikimedia.org/T427575#12377361 (10Pharos) 05In progress→03Resolved [22:57:15] 10Tool-wikinewsie: Wikinewsie suggested headline from AI recap - https://phabricator.wikimedia.org/T436419#12377362 (10Pharos) 05In progress→03Resolved