[00:11:00] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [00:11:05] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [00:16:00] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [00:16:05] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [00:16:31] (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [00:16:31] (03update) 10renovatebot: Update python Docker tag to v3.14.8 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [01:16:20] (03update) 10renovatebot: Update python Docker tag to v3.14.8 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [02:21:52] (03open) 10ladsgroup: serve the site with the Toolforge static webservice instead of Node [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/7 [03:38:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [03:38:42] 10Tool-wikinewsie: Wikinewsie language selector fails to show full language names on mobile - https://phabricator.wikimedia.org/T439939 (10Pharos) 03NEW [03:38:46] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [03:39:58] 10Tool-wikinewsie: Wikinewsie language selector fails to show full language names on mobile - https://phabricator.wikimedia.org/T439939#12388066 (10Pharos) [03:39:59] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [03:50:17] 10Tool-wikinewsie: Wikinewsie elements fail to translate upon reload from language selector - https://phabricator.wikimedia.org/T439940 (10Pharos) 03NEW [03:51:19] 10Tool-wikinewsie: Wikinewsie elements fail to translate upon reload from language selector - https://phabricator.wikimedia.org/T439940#12388082 (10Pharos) [03:56:58] 10Tool-wikinewsie: Wikinewsie elements fail to translate upon reload from language selector - https://phabricator.wikimedia.org/T439940#12388084 (10Pharos) [03:57:36] 10Tool-wikinewsie: Wikinewsie elements fail to translate upon reload from language selector - https://phabricator.wikimedia.org/T439940#12388086 (10Pharos) [03:58:28] PROBLEM - Host cloudcephosd1017 is DOWN: PING CRITICAL - Packet loss = 100% [04:00:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [04:00:42] RECOVERY - Host cloudcephosd1017 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [04:03:40] PROBLEM - Host cloudcephosd1017 is DOWN: PING CRITICAL - Packet loss = 100% [04:04:42] RECOVERY - Host cloudcephosd1017 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms [04:07:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [04:07:57] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [04:10:06] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [04:10:12] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [04:12:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [05:25:43] (03update) 10waldir: Various styling fixes [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/25 [05:26:26] (03merge) 10waldir: Various styling fixes [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/25 [05:41:55] (03merge) 10waldir: Update information about contributors and history of the project [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/28 [05:45:57] (03open) 10r4356th: Make bot operate concurrently and restructure delinter code [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/25 (https://phabricator.wikimedia.org/T421480) [06:56:30] (03open) 10countcount: Allow cross-origin reads of the plain-text bot output [toolforge-repos/stimmberechtigung] - 10https://gitlab.wikimedia.org/toolforge-repos/stimmberechtigung/-/merge_requests/8 [06:56:42] (03merge) 10countcount: Allow cross-origin reads of the plain-text bot output [toolforge-repos/stimmberechtigung] - 10https://gitlab.wikimedia.org/toolforge-repos/stimmberechtigung/-/merge_requests/8 [06:56:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [06:57:03] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:01:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [07:02:03] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:59:42] 10Tool-Pageviews: NaN error on the Page Views page - https://phabricator.wikimedia.org/T439452#12388286 (10MusikAnimal) It's because the logarithmic scale is automatically being shown due to the spike on September 13. With a logarithmic scale, it is impossible to have a value of zero. This should still be fixed,... [08:13:43] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [08:13:50] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [10:20:02] FIRING: CloudVirtDisabled: cloudvirt1071 out of service for long - TODO - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1071 - https://alerts.wikimedia.org/?q=alertname%3DCloudVirtDisabled [10:22:36] (03merge) 10fnegri: maintainer-alerts: remove tool- prefix [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/72 (https://phabricator.wikimedia.org/T437081) [10:28:18] (03merge) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [10:28:19] (03update) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [10:30:28] (03update) 10fnegri: build: Upgrade Poetry dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/11 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:31:23] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.10-20261002102828-29c7e276 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1442 (https://phabricator.wikimedia.org/T437081) [10:31:36] (03merge) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [10:33:49] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.11-20261002103146-d9ef1155 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1442 (https://phabricator.wikimedia.org/T432865 https://phabricator.wikimedia.org/T437081) [10:33:58] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.11-20261002103146-d9ef1155 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1442 (https://phabricator.wikimedia.org/T432865 https://phabricator.wikimedia.org/T437081) [10:45:13] 10Tool-wikiloves: Wikiloves statistics are not being updated - https://phabricator.wikimedia.org/T437822#12388768 (10Nemoralis) 05Stalled→03Resolved a:03JeanFred [10:52:39] (03open) 10fnegri: alerts-api: Turn on the emails in tools and toolsbeta [repos/cloud/toolforge/toolforge-deploy] (bump_alerts-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1443 (https://phabricator.wikimedia.org/T432865) [11:01:30] (03update) 10fnegri: alerts-api: Turn on the emails in tools and toolsbeta [repos/cloud/toolforge/toolforge-deploy] (bump_alerts-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1443 (https://phabricator.wikimedia.org/T432865) [11:05:59] !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component alerts-api (T432865) [11:06:04] T432865: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865 [11:07:06] !log fnegri@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api (T432865) [11:08:27] 10Tool-tourism-mapper: Implement Wikimedia OAuth authentication - https://phabricator.wikimedia.org/T439959 (10JosefAnthony) 03NEW [11:09:01] 10Tool-tourism-mapper: Implement Wikimedia OAuth authentication - https://phabricator.wikimedia.org/T439959#12388832 (10JosefAnthony) 05Open→03Resolved a:03JosefAnthony [11:22:12] FIRING: ToolFrequentRestarts: Containers of test-maintainer-alerts are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [12:12:12] RESOLVED: ToolFrequentRestarts: Containers of test-maintainer-alerts are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [12:37:28] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [12:37:35] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [12:40:12] 10Toolforge, 06tools-platform-team: [jobs-api] Make pre-built image webservices move all logs to logs-api based - https://phabricator.wikimedia.org/T439574#12389050 (10aputhin) [12:47:18] 10Tool-itwiki, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): If the section "Voci correlate" becomes empty, remove that section - https://phabricator.wikimedia.org/T338084#12389085 (10valerio.bozzolan) a:03valerio.bozzolan [12:47:31] 06cloud-services-team, 10Horizon, 10Striker, 06tools-infrastructure-team, and 2 others: Move cloudweb hosts to cloud racks? - https://phabricator.wikimedia.org/T411783#12389087 (10aputhin) [12:48:01] 10Tool-itwiki, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Improve OrphanizerBot to handle disambiguation pages differently - https://phabricator.wikimedia.org/T320693#12389092 (10valerio.bozzolan) p:05Triage→03Medium a:03valerio.bozzolan [12:53:13] PROBLEM - Host cloudcephosd1018 is DOWN: PING CRITICAL - Packet loss = 100% [12:54:41] RECOVERY - Host cloudcephosd1018 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [12:59:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [13:02:03] PROBLEM - Host cloudcephosd1018 is DOWN: PING CRITICAL - Packet loss = 100% [13:03:41] RECOVERY - Host cloudcephosd1018 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [13:06:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:06:45] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [13:11:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:44:36] 10Toolforge (Quota-requests): Request increased quota for mna Toolforge tool - https://phabricator.wikimedia.org/T439975 (10connorshea) 03NEW [13:45:39] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component alerts-api (T432865) [13:45:43] T432865: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865 [13:46:39] (03update) 10fnegri: build: Upgrade Poetry dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/11 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:46:47] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api (T432865) [13:51:05] FIRING: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:01:05] RESOLVED: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:42:37] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [15:01:52] (03update) 10fnegri: alerts-api: bump to 0.0.11-20261002103146-d9ef1155 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1442 (https://phabricator.wikimedia.org/T432865 https://phabricator.wikimedia.org/T437081) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:02:02] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Create template for maintainer alerts emails - https://phabricator.wikimedia.org/T437081#12389792 (10fnegri) 05In progress→03Resolved The HTML version of the templates, along with a txt version containing the same text, is now merg... [15:02:09] (03merge) 10fnegri: alerts-api: bump to 0.0.11-20261002103146-d9ef1155 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1442 (https://phabricator.wikimedia.org/T432865 https://phabricator.wikimedia.org/T437081) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:02:15] (03update) 10fnegri: alerts-api: Turn on the emails in tools and toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1443 (https://phabricator.wikimedia.org/T432865) [15:02:40] (03update) 10fnegri: alerts-api: Turn on the emails in tools and toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1443 (https://phabricator.wikimedia.org/T432865) [15:02:57] (03approved) 10fnegri: alerts-api: Turn on the emails in tools and toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1443 (https://phabricator.wikimedia.org/T432865) [15:03:14] (03update) 10fnegri: alerts-api: Turn on the emails in tools and toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1443 (https://phabricator.wikimedia.org/T432865) [15:04:39] (03update) 10fnegri: alerts-api: Turn on the emails in tools and toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1443 (https://phabricator.wikimedia.org/T432865) [15:04:51] (03merge) 10fnegri: alerts-api: Turn on the emails in tools and toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1443 (https://phabricator.wikimedia.org/T432865) [15:10:18] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12389899 (10fnegri) 05In progress→03Resolved This is live and working both in toolsbeta and tools! 🎉 **Note:** emails are curr... [15:16:46] (03update) 10fnegri: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/10 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:16:47] (03approved) 10fnegri: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/10 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:16:59] (03update) 10fnegri: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/10 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:22:55] (03merge) 10fnegri: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/10 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:23:03] (03update) 10fnegri: build: Upgrade Poetry dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/11 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:23:06] (03approved) 10fnegri: build: Upgrade Poetry dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/11 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:23:21] (03update) 10fnegri: build: Upgrade Poetry dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/11 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:25:30] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.12-20261002152314-b1fa7b42 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1444 [15:32:37] (03merge) 10fnegri: build: Upgrade Poetry dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/11 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:35:22] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.13-20261002153248-60981375 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1444 [15:35:32] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.13-20261002153248-60981375 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1444 [15:38:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:55:31] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12390191 (10fnegri) //Note: we are experimenting with cross-posting to Phabricator the weekly report that previously was only posted to Asana, where only few people have acc... [16:15:18] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,trove [16:15:31] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,trove [16:31:11] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:43:09] (03approved) 10fnegri: cli: add warning message for beta status [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/14 (https://phabricator.wikimedia.org/T439581) (owner: 10dcaro) [16:46:03] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12390548 (10fnegri) [17:02:56] 10Cloud-VPS, 06tools-infrastructure-team, 07Grafana: CloudVPS project grafana dashboards are empty - https://phabricator.wikimedia.org/T440021 (10Don-vip) 03NEW [17:04:45] (03approved) 10fnegri: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [17:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:09:51] (03update) 10fnegri: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] (add_source_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [17:10:50] (03update) 10r4356th: Make bot operate concurrently and restructure delinter code [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/25 (https://phabricator.wikimedia.org/T421480) [17:16:29] (03update) 10fnegri: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [17:17:32] (03update) 10fnegri: cli: add the --job-names flag [repos/cloud/toolforge/logs-cli] (add_lines_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/10 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [17:18:04] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [17:18:07] (03approved) 10fnegri: cli: add the --job-names flag [repos/cloud/toolforge/logs-cli] (add_lines_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/10 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [17:18:10] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [17:20:00] (03update) 10r4356th: Make bot operate concurrently and restructure delinter code [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/25 (https://phabricator.wikimedia.org/T421480) [17:21:02] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [17:28:22] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:31:24] FIRING: [2x] SystemdUnitFailed: keystone.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:33:22] FIRING: [4x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:33:29] FIRING: SystemdUnitCrashLoop: keystone.service crashloop on cloudcontrol1011:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [17:36:24] FIRING: [4x] SystemdUnitFailed: keystone.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:37:44] PROBLEM - Host cloudcephosd1019 is DOWN: PING CRITICAL - Packet loss = 100% [17:38:22] RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1011.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:39:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:39:46] RECOVERY - Host cloudcephosd1019 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [17:40:07] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [17:41:24] FIRING: [2x] SystemdUnitFailed: keystone.service on cloudcontrol1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:42:56] PROBLEM - Host cloudcephosd1019 is DOWN: PING CRITICAL - Packet loss = 100% [17:44:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:44:46] RECOVERY - Host cloudcephosd1019 is UP: PING OK - Packet loss = 0%, RTA = 0.38 ms [17:44:47] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:46:24] RESOLVED: [2x] SystemdUnitFailed: keystone.service on cloudcontrol1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:47:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [17:47:39] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [17:49:47] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:52:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [17:56:01] RESOLVED: SystemdUnitCrashLoop: keystone.service crashloop on cloudcontrol1011:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [18:01:25] (03update) 10r4356th: Make bot operate concurrently and restructure delinter code [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/25 (https://phabricator.wikimedia.org/T421480) [18:10:23] (03update) 10r4356th: Make bot operate concurrently and restructure delinter code [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/25 (https://phabricator.wikimedia.org/T421480) [18:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:20:10] (03merge) 10r4356th: Make bot operate concurrently and restructure delinter code [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/25 (https://phabricator.wikimedia.org/T421480) [18:32:21] 10Tool-delintbot: Make bot operate concurrently - https://phabricator.wikimedia.org/T421480#12390837 (10Redmin) 05Open→03Resolved a:03Redmin Done along with these other notable changes: - Got rid of 'mode' argument - Removed the 'cat' global variable - Introduced a new custom lint category '0' which te... [18:54:08] (03open) 10r4356th: Report the number of lint errors fixed in one run [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/26 [19:11:23] (03update) 10r4356th: Report the number of lint errors fixed in one run [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/26 [19:11:51] (03update) 10r4356th: Report the number of lint errors fixed in one run [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/26 [19:12:01] (03merge) 10r4356th: Report the number of lint errors fixed in one run [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/26 [20:18:18] 10Tool-wikinewsie: Wikinewsie image captions - https://phabricator.wikimedia.org/T440058 (10Pharos) 03NEW [20:29:03] 10Tool-wikinewsie: Portal:Current events mode for Wikinewsie - https://phabricator.wikimedia.org/T432971#12391503 (10Pharos) It's interesting that ITN and Current events seem to differ wildly in implementation across Wikipedia language editions. For ITN, the main task will be figuring out which single article i... [20:34:35] 10Tool-wikinewsie: Wikinewsie doesn't seem to update after Wikidata item is created - https://phabricator.wikimedia.org/T431957#12391513 (10Pharos) Maybe this could be done as a more generalized "Refresh article card" button? Similar issues occur when a Wikipedia article has been moved/renamed recently, and Wiki... [23:15:40] (03update) 10renovatebot: Update cimg/rust Docker tag to v1.99 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/47 [23:15:42] (03open) 10renovatebot: Update cimg/rust Docker tag to v1.99 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/47 [23:45:15] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [23:45:21] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [23:51:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [23:52:03] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [23:56:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [23:57:03] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown