[01:16:44] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [01:20:06] PROBLEM - Host cloudcephosd1044 is DOWN: PING CRITICAL - Packet loss = 100% [01:22:36] RECOVERY - Host cloudcephosd1044 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms [01:23:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [01:25:31] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [01:28:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [01:28:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [01:28:33] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [03:58:56] (03open) 10raymond-ndibe: config: change config schema to v1beta2 while maintaining v1beta1 support [repos/cloud/toolforge/components-api] (fix_component_type_discriminator) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/194 (https://phabricator.wikimedia.org/T439496) [03:59:00] (03update) 10raymond-ndibe: config: change config schema to v1beta2 while maintaining v1beta1 support [repos/cloud/toolforge/components-api] (fix_component_type_discriminator) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/194 (https://phabricator.wikimedia.org/T439496) [04:00:06] (03open) 10raymond-ndibe: models: make component_type the component union discriminator [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/195 (https://phabricator.wikimedia.org/T439496) [04:07:59] (03update) 10raymond-ndibe: models: make component_type the component union discriminator [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/195 (https://phabricator.wikimedia.org/T439496) [04:10:12] (03update) 10raymond-ndibe: config: change config schema to v1beta2 while maintaining v1beta1 support [repos/cloud/toolforge/components-api] (fix_component_type_discriminator) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/194 (https://phabricator.wikimedia.org/T439496) [05:38:59] (03open) 10countcount: Update brace-expansion and fast-uri to patched versions [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/22 [05:39:01] (03open) 10countcount: Update oauthlib to 4.0.0 for security advisories [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/44 [05:39:32] (03open) 10countcount: Update pyjwt and oauthlib to patched releases [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/34 [05:39:32] (03open) 10countcount: Update brace-expansion and fast-uri to patched versions [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/61 [05:39:34] (03open) 10countcount: Update pyjwt to 2.15.1 and oauthlib to 4.0.0 for security advisories [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/31 [05:39:40] (03open) 10countcount: Update brace-expansion and undici lockfile entries for security advisories [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/46 [05:39:53] (03open) 10countcount: Update brace-expansion and fast-uri to patched versions [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/29 [05:41:59] (03merge) 10countcount: Update brace-expansion and fast-uri to patched versions [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/29 [05:42:08] (03update) 10countcount: Update brace-expansion and undici lockfile entries for security advisories [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/46 [05:42:32] (03merge) 10countcount: Update pyjwt to 2.15.1 and oauthlib to 4.0.0 for security advisories [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/31 [05:42:40] (03open) 10countcount: Update rustls to 0.23.45 (RUSTSEC-2026-0285) [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/81 [05:43:02] (03merge) 10countcount: Update brace-expansion and fast-uri to patched versions [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/22 [05:43:15] (03merge) 10countcount: Update oauthlib to 4.0.0 for security advisories [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/44 [05:43:17] (03open) 10countcount: Update rustls to 0.23.45 to fix RUSTSEC-2026-0285 [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/29 [05:43:25] (03merge) 10countcount: Update brace-expansion and fast-uri to patched versions [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/61 [05:43:51] (03update) 10countcount: Update pyjwt and oauthlib to patched releases [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/34 [05:44:21] (03merge) 10countcount: Update pyjwt and oauthlib to patched releases [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/34 [05:44:52] (03merge) 10countcount: Update brace-expansion and undici lockfile entries for security advisories [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/46 [05:48:53] (03open) 10countcount: Trim implementation details from the README [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/82 [05:51:59] (03merge) 10countcount: Update rustls to 0.23.45 to fix RUSTSEC-2026-0285 [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/29 [05:55:20] (03update) 10countcount: Trim implementation details from the README [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/82 [05:57:18] (03update) 10countcount: Trim implementation details from the README [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/82 [06:02:41] (03update) 10countcount: Update rustls to 0.23.45 (RUSTSEC-2026-0285) and fix new Clippy lints [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/81 [06:02:44] (03update) 10countcount: Update rustls to 0.23.45 (RUSTSEC-2026-0285) and fix new Clippy lints [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/81 [06:03:09] (03update) 10countcount: Update rustls to 0.23.45 (RUSTSEC-2026-0285) and fix new Clippy lints [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/81 [06:05:48] (03merge) 10countcount: Update rustls to 0.23.45 (RUSTSEC-2026-0285) and fix new Clippy lints [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/81 [06:08:58] (03update) 10countcount: Trim implementation details from the README [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/82 [06:11:27] (03update) 10countcount: Trim implementation details from the README [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/82 [06:12:06] (03merge) 10countcount: Trim implementation details from the README [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/82 [06:14:29] (03update) 10renovatebot: Update pnpm to v12.8.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/21 [06:15:05] (03merge) 10countcount: Update pnpm to v12.8.1 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/33 (owner: 10renovatebot) [06:15:05] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [06:15:23] (03update) 10renovatebot: Update pnpm to v12.8.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/28 [06:15:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [06:18:37] (03update) 10renovatebot: Update pnpm to v12.8.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/60 [06:18:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [06:19:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [06:19:52] (03update) 10renovatebot: Update pnpm to v12.8.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/45 [06:20:07] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [06:20:48] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [06:21:14] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [06:21:22] (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [06:21:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [06:21:40] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [06:22:19] (03update) 10countcount: Update pnpm to v12.8.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/45 (owner: 10renovatebot) [06:22:46] (03merge) 10countcount: Update pnpm to v12.8.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/60 (owner: 10renovatebot) [06:22:52] (03merge) 10countcount: Update pnpm to v12.8.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/28 (owner: 10renovatebot) [06:23:02] (03merge) 10countcount: Update pnpm to v12.8.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/21 (owner: 10renovatebot) [06:23:58] (03merge) 10countcount: Update pnpm to v12.8.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/45 (owner: 10renovatebot) [06:31:51] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [06:32:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [06:32:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [06:32:31] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12377823 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1004 for host cloudcephosd1055.eqiad.wmn... [06:33:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [06:35:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [07:17:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:22:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:58:02] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [07:59:13] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12377978 (10fgiunchedi) I was mistaken again, even with drives on two different buses the ordering is not predictable. New idea/plan to get hwraid + jbod w... [08:07:17] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [08:20:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:25:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:26:31] 10Cloud-VPS, 06tools-infrastructure-team, 10Wikispeech-Text-to-Speech: Error when creating snapshot for instance - https://phabricator.wikimedia.org/T439363#12378116 (10Sebastian_Berlin-WMSE) Thanks for the explanation. I understand at least part of it πŸ™‚ >>! In T439363#12374530, @Andrew wrote: > [...] used... [08:27:23] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:27:47] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [08:29:10] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:29:30] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [08:30:54] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:36:00] (03merge) 10countcount: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 (owner: 10renovatebot) [08:36:31] (03merge) 10countcount: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 (owner: 10renovatebot) [08:36:38] (03merge) 10countcount: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 (owner: 10renovatebot) [08:37:13] (03merge) 10countcount: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 (owner: 10renovatebot) [08:37:19] (03merge) 10countcount: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 (owner: 10renovatebot) [08:39:15] 10Toolforge, 06tools-platform-team: [jobs-api] Collect and send the user to logs-api when sending events - https://phabricator.wikimedia.org/T439679 (10dcaro) 03NEW [08:44:16] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [08:51:17] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:55:04] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [09:00:12] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:02:49] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [09:11:00] !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component alerts-api (T432865) [09:11:05] T432865: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865 [09:12:05] !log fnegri@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api (T432865) [09:16:39] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12378352 (10fgiunchedi) >>! In T437407#12355411, @fgiunchedi wrote: > I got a chance to play with the testing... [09:27:12] FIRING: ToolFrequentRestarts: Containers in tool-test-maintainer-alerts are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [09:31:14] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component alerts-api (T432865) [09:31:19] T432865: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865 [09:32:12] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api (T432865) [09:39:15] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:42:01] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [09:46:29] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:01:18] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:02:47] (03merge) 10fnegri: alerts-api: Use the root URL for prometheus hosts [repos/cloud/toolforge/toolforge-deploy] (bump_alerts-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1438 (https://phabricator.wikimedia.org/T432865) [10:02:49] (03update) 10fnegri: alerts-api: bump to 0.0.9-20260929163209-a3bb1242 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1436 (https://phabricator.wikimedia.org/T432865) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:03:28] (03update) 10fnegri: alerts-api: bump to 0.0.9-20260929163209-a3bb1242 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1436 (https://phabricator.wikimedia.org/T432865) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:03:57] (03merge) 10fnegri: alerts-api: bump to 0.0.9-20260929163209-a3bb1242 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1436 (https://phabricator.wikimedia.org/T432865) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:05:05] (03approved) 10dcaro: jobs-api: bump to 0.0.572-20260929151505-9f6db6d8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1437 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:05:12] (03update) 10dcaro: jobs-api: bump to 0.0.572-20260929151505-9f6db6d8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1437 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:05:15] (03update) 10dcaro: jobs-api: bump to 0.0.572-20260929151505-9f6db6d8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1437 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:06:29] (03merge) 10dcaro: jobs-api: bump to 0.0.572-20260929151505-9f6db6d8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1437 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:16:25] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [10:17:12] RESOLVED: ToolFrequentRestarts: Containers in tool-test-maintainer-alerts are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [10:37:50] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [10:38:00] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: How long should archived toolforge tools last? - https://phabricator.wikimedia.org/T439486#12378720 (10aputhin) So long as this is well documented as part of the process of archiving, I think it's very reasonable. This retention period not only... [10:38:03] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [10:42:50] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [10:43:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:43:03] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [10:57:50] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [10:58:03] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [10:59:48] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [10:59:59] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [11:02:50] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [11:03:03] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [11:15:27] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Deploy hiddenparma application - https://phabricator.wikimedia.org/T439418#12378843 (10taavi) My current preference is the container on a standalone VM option given it seems to have the smalles required additional complexity. [11:15:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [11:48:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [11:53:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [11:55:51] 10Tool-wikicordo: Filenames with β€œ?” are not URL‑encoded, causing incorrect deletion request URLs - https://phabricator.wikimedia.org/T418633#12379014 (10kolbert) I had the same issue with https://commons.wikimedia.org/wiki/Commons:Deletion%20requests/File:The%20past.%20The%20present.%20The%20future?%20(23941501... [11:59:25] 10Toolforge, 06tools-platform-team: [jobs-api] Remove the logs endpoint - https://phabricator.wikimedia.org/T439706 (10dcaro) 03NEW [12:24:46] (03update) 10dcaro: global: add source to all logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T439517) [12:25:04] (03update) 10dcaro: global: add source to all logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T439517) [12:28:13] (03update) 10fnegri: global: add source to all logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T439517) (owner: 10dcaro) [12:28:54] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli] support run-logs and deploy events - https://phabricator.wikimedia.org/T439528#12379184 (10fnegri) [12:28:56] (03approved) 10fnegri: global: add source to all logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T439517) (owner: 10dcaro) [12:29:24] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [components-api] return `source` for all logs - https://phabricator.wikimedia.org/T439517#12379186 (10fnegri) 05Openβ†’03In progress [12:31:24] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [12:31:27] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [12:34:07] PROBLEM - Host cloudcephosd1043 is DOWN: PING CRITICAL - Packet loss = 100% [12:36:37] RECOVERY - Host cloudcephosd1043 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [12:38:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [12:41:24] FIRING: [8x] SystemdUnitFailed: ceph-osd@186.service on cloudcephosd1043:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:42:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [12:43:00] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [12:43:04] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [12:43:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [12:45:31] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [12:45:32] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [12:46:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [12:46:15] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [12:48:21] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [12:48:22] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [12:48:28] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [12:52:53] 10Toolforge, 06tools-platform-team: [lima-kilo,logging,logs-api] find a replacement for minio (s3 for loki) - https://phabricator.wikimedia.org/T439720 (10dcaro) 03NEW [12:52:56] 10Toolforge, 06tools-platform-team: [lima-kilo,logging,logs-api] find a replacement for minio (s3 for loki) - https://phabricator.wikimedia.org/T439720#12379371 (10dcaro) p:05Triageβ†’03High [12:53:56] 10Toolforge, 06tools-platform-team: [lima-kilo,logging,logs-api] find a replacement for minio (s3 for loki) - https://phabricator.wikimedia.org/T439720#12379378 (10dcaro) [13:02:54] RESOLVED: [8x] SystemdUnitFailed: ceph-osd@186.service on cloudcephosd1043:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:09:10] 06tools-platform-team, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 06Traffic: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12379479 (10fnegri) [13:10:04] 06tools-platform-team, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 06Traffic: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12379487 (10MoritzMuehlenhoff) [13:16:13] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli] support run-logs and deploy events - https://phabricator.wikimedia.org/T439528#12379528 (10fnegri) 05Openβ†’03In progress [13:24:47] 06tools-platform-team, 06Collaboration-Services, 06Infrastructure-Foundations, 06ServiceOps, and 2 others: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12379559 (10Jelto) Adding #collaboration-services tag for wmf-navigator [13:26:33] (03approved) 10dcaro: global: add source to all logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T439517) [13:26:37] (03update) 10dcaro: global: add source to all logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T439517) [13:26:43] (03merge) 10dcaro: global: add source to all logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T439517) [13:28:43] 06cloud-services-team, 10Horizon, 10Striker, 06tools-infrastructure-team, and 3 others: Move cloudweb hosts to cloud racks? - https://phabricator.wikimedia.org/T411783#12379579 (10taavi) 05Openβ†’03Declined Per the network sync meeting, we will be doing {T392478} instead. [13:29:51] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.56-20260930132654-1c1f2893 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1439 (https://phabricator.wikimedia.org/T439517) [13:30:30] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:39:35] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12379660 (10aputhin) [13:39:46] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12379663 (10aputhin) p:05Triageβ†’03High [13:43:15] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] support job stop/pause - https://phabricator.wikimedia.org/T436638#12379686 (10aputhin) [13:51:30] (03update) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [13:52:42] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [13:55:49] (03update) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [14:04:02] 10Cloud-VPS, 06tools-platform-team: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729 (10fgiunchedi) 03NEW [14:04:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [14:05:02] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] show a warning message stating it's in beta/unstable - https://phabricator.wikimedia.org/T439581#12379869 (10aputhin) p:05Triageβ†’03Medium [14:05:05] (03open) 10dcaro: global: send job events to logs-api [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380 (https://phabricator.wikimedia.org/T432570) [14:05:51] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [jobs-api] Send webservice start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12379895 (10dcaro) a:05TLepage-WMFβ†’03dcaro [14:06:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [14:06:41] 10Cloud-VPS, 06tools-platform-team: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12379902 (10fgiunchedi) [14:08:53] 10Cloud-VPS, 06tools-infrastructure-team: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12379938 (10fgiunchedi) [14:12:19] 10Cloud-VPS, 06tools-infrastructure-team: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12379943 (10fgiunchedi) [14:16:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [14:16:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [14:17:04] 06tools-platform-team, 10Toolhub: [Toolhub] Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439732 (10aputhin) 03NEW [14:17:27] 06tools-platform-team, 10Toolhub: [Toolhub] Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439732#12379984 (10aputhin) p:05Triageβ†’03Medium [14:17:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [14:17:48] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [14:18:10] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [14:18:11] 06tools-platform-team, 06Collaboration-Services, 06Infrastructure-Foundations, 06ServiceOps, and 2 others: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12379989 (10aputhin) Separating the Toolhub work onto its own subtask: T439732 [14:18:27] (03approved) 10dcaro: logs-api: bump to 0.0.56-20260930132654-1c1f2893 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1439 (https://phabricator.wikimedia.org/T439517) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:18:32] (03merge) 10dcaro: logs-api: bump to 0.0.56-20260930132654-1c1f2893 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1439 (https://phabricator.wikimedia.org/T439517) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:18:49] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [components-api] return `source` for all logs - https://phabricator.wikimedia.org/T439517#12379996 (10dcaro) 05In progressβ†’03Resolved [14:21:28] 10Toolforge, 06tools-platform-team: [jobs-api] Remove the logs endpoint - https://phabricator.wikimedia.org/T439706#12380027 (10aputhin) p:05Triageβ†’03Low [14:29:31] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [jobs-api] Send start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12380078 (10dcaro) [14:31:27] 10Toolforge, 06tools-platform-team: [lima-kilo,toolforge_deploy,functional-tests] fail if any of the components last deployment failed - https://phabricator.wikimedia.org/T438937#12380086 (10aputhin) [14:34:37] 10Toolforge, 06tools-platform-team: [lima-kilo,toolforge_deploy,functional-tests] fail if any of the components last deployment failed - https://phabricator.wikimedia.org/T438937#12380108 (10aputhin) p:05Triageβ†’03Low Setting priority to low as this is not a usual occurrence overall. It happened on a refact... [14:42:21] 10Toolforge, 06tools-platform-team: [jobs-api] Error with mounts on one-off jobs - https://phabricator.wikimedia.org/T438730#12380146 (10dcaro) This is failing: ` local.tf-test2@toolslocal:~$ toolforge jobs run --command 'test' --image python3.13 --mount=none oneoff2 ERROR: 1 validation error for CommonOptions... [14:42:37] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [14:44:12] 10Toolforge, 06tools-platform-team: [jobs-api] Error with mounts on one-off jobs - https://phabricator.wikimedia.org/T438730#12380151 (10aputhin) p:05Triageβ†’03Low Might affect other types of jobs besides one-of tho. [14:46:24] 10Toolforge, 06tools-platform-team: [jobs-api] Do not require NFS mounts for one-off jobs (and potentially pre-built image jobs) - https://phabricator.wikimedia.org/T438730#12380169 (10aputhin) [14:46:55] 10Toolforge, 06tools-platform-team: [jobs-api] Do not require NFS mounts for one-off jobs using pre-built images (and potentially other job types) - https://phabricator.wikimedia.org/T438730#12380171 (10dcaro) [14:49:15] 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T439635#12380183 (10aputhin) p:05Triageβ†’03High [15:06:27] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12380284 (10fnegri) If we can avoid migrating all existing configs, I think it would be nice. This is a new... [15:15:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [15:16:19] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [15:16:49] (03open) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] (support_run_logs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [15:22:29] (03update) 10dcaro: global: send job events to logs-api [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380 (https://phabricator.wikimedia.org/T432570) [15:23:32] (03update) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] (support_run_logs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [15:29:37] (03update) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [15:30:39] (03update) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [15:38:25] (03update) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [15:38:35] 10Cloud-VPS (Project-requests): Resize legacy.wikiapiary.eqiad1.wikimedia.cloud to g4.core16.ram32.disk20 - https://phabricator.wikimedia.org/T439317#12380524 (10Raymond_Ndibe) Hello @MarkAHershberger it's a bit unclear what your request is. Your request doesn't follow the request format outlined here https://p... [15:41:48] (03update) 10fnegri: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) (owner: 10dcaro) [15:41:49] (03approved) 10fnegri: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) (owner: 10dcaro) [15:41:51] (03update) 10fnegri: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) (owner: 10dcaro) [15:43:18] (03approved) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [15:43:24] (03merge) 10dcaro: global: support run-logs and deploy events [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T439528) [15:43:26] (03update) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [15:43:57] (03update) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [15:46:07] (03CR) 10Hnowlan: [C:03+1] alert: add a dummy Klaxon managers incident URL [labs/private] - 10https://gerrit.wikimedia.org/r/1345218 (owner: 10Andrea Denisse) [15:46:39] 10Tool-spacemedia, 10Toolforge (Quota-requests): Request increased quota for spacemedia Toolforge tool - https://phabricator.wikimedia.org/T439237#12380560 (10dcaro) +1 I think that you are the first hitting the quota! πŸŽ‰ [15:49:53] 10Cloud-VPS (Project-requests): Resize legacy.wikiapiary.eqiad1.wikimedia.cloud to g4.core16.ram32.disk20 - https://phabricator.wikimedia.org/T439317#12380592 (10dcaro) For reference, you can resize the instance yourself by using the button on the right menu when the instance is selected: {F105589945} [15:56:48] (03update) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [15:57:39] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create continuous job deployment tutorial - https://phabricator.wikimedia.org/T437903#12380636 (10komla) [15:58:09] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Documentation: Write tutorial for migrating build service jobs to push-to-deploy - https://phabricator.wikimedia.org/T437477#12380637 (10komla) [15:58:36] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [jobs-api] Send start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12380639 (10fnegri) [15:58:37] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Identify candidate tools for Push-to-Deploy - https://phabricator.wikimedia.org/T437897#12380640 (10komla) [15:58:49] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Identify candidate maintainers - https://phabricator.wikimedia.org/T437898#12380643 (10komla) [15:59:06] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12380644 (10komla) [15:59:22] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create code change and redeploy tutorial - https://phabricator.wikimedia.org/T437900#12380646 (10komla) [15:59:38] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create scheduled job deployment tutorial - https://phabricator.wikimedia.org/T437901#12380647 (10komla) [16:01:45] (03update) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [16:08:23] 10Tool-spacemedia, 10Toolforge (Quota-requests): Request increased quota for spacemedia Toolforge tool - https://phabricator.wikimedia.org/T439237#12380693 (10Don-vip) Can I have a prize or a medal for that? :D [16:09:00] (03open) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] (add_source_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [16:19:34] 10Tools, 13Patch-For-Review, 07PHP 8.5 support: labs/tools/toolbase fails its CI tests in PHP 8.5 due to old Phan version - https://phabricator.wikimedia.org/T419082#12380815 (10Krinkle) p:05Triageβ†’03Low a:03Krinkle [16:19:35] (03PS1) 10Krinkle: build: Update Phan to fix PHP 8.5 build [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346673 (https://phabricator.wikimedia.org/T419082) [16:19:41] (03CR) 10CI reject: [V:04-1] build: Update Phan to fix PHP 8.5 build [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346673 (https://phabricator.wikimedia.org/T419082) (owner: 10Krinkle) [16:21:58] (03PS2) 10Krinkle: build: Update Phan to fix PHP 8.5 build [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346673 (https://phabricator.wikimedia.org/T419082) [16:22:47] (03CR) 10Krinkle: "check experimental" [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346673 (https://phabricator.wikimedia.org/T419082) (owner: 10Krinkle) [16:27:52] (03CR) 10Krinkle: "recheck" [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346673 (https://phabricator.wikimedia.org/T419082) (owner: 10Krinkle) [16:28:09] (03CR) 10Krinkle: [C:03+2] build: Update Phan to fix PHP 8.5 build [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346673 (https://phabricator.wikimedia.org/T419082) (owner: 10Krinkle) [16:28:44] (03Merged) 10jenkins-bot: build: Update Phan to fix PHP 8.5 build [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346673 (https://phabricator.wikimedia.org/T419082) (owner: 10Krinkle) [16:44:31] (03PS1) 10Krinkle: build: Update PHPUnit from 9.x to 10.x and further to 12.x [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346694 (https://phabricator.wikimedia.org/T328919) [16:45:53] (03CR) 10Krinkle: [C:03+2] build: Update PHPUnit from 9.x to 10.x and further to 12.x [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346694 (https://phabricator.wikimedia.org/T328919) (owner: 10Krinkle) [16:46:40] (03Merged) 10jenkins-bot: build: Update PHPUnit from 9.x to 10.x and further to 12.x [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346694 (https://phabricator.wikimedia.org/T328919) (owner: 10Krinkle) [16:51:35] (03PS1) 10Krinkle: Tag v2.3.0 [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346695 [16:53:33] (03CR) 10Krinkle: [C:03+2] Tag v2.3.0 [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346695 (owner: 10Krinkle) [16:53:50] 10Tools, 07PHP 8.5 support: labs/tools/toolbase fails its CI tests in PHP 8.5 due to old Phan version - https://phabricator.wikimedia.org/T419082#12381135 (10Krinkle) 05Openβ†’03Resolved [16:54:14] (03Merged) 10jenkins-bot: Tag v2.3.0 [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346695 (owner: 10Krinkle) [16:56:55] (03PS1) 10Krinkle: build: Update phan to fix PHP 8.5 build [labs/tools/usage] - 10https://gerrit.wikimedia.org/r/1346700 (https://phabricator.wikimedia.org/T419078) [16:59:19] (03open) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [17:00:20] (03PS1) 10Krinkle: docs: Update toolforge deployment from php7.4 to php8.4 [labs/tools/usage] - 10https://gerrit.wikimedia.org/r/1346702 (https://phabricator.wikimedia.org/T419078) [17:00:34] (03CR) 10Krinkle: "check experimental" [labs/tools/usage] - 10https://gerrit.wikimedia.org/r/1346700 (https://phabricator.wikimedia.org/T419078) (owner: 10Krinkle) [17:00:45] (03open) 10fnegri: Swap minio with silo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1440 (https://phabricator.wikimedia.org/T439720) [17:02:14] (03CR) 10Krinkle: [C:03+2] build: Update phan to fix PHP 8.5 build [labs/tools/usage] - 10https://gerrit.wikimedia.org/r/1346700 (https://phabricator.wikimedia.org/T419078) (owner: 10Krinkle) [17:03:06] (03CR) 10Krinkle: [C:03+2] docs: Update toolforge deployment from php7.4 to php8.4 [labs/tools/usage] - 10https://gerrit.wikimedia.org/r/1346702 (https://phabricator.wikimedia.org/T419078) (owner: 10Krinkle) [17:03:23] (03Merged) 10jenkins-bot: build: Update phan to fix PHP 8.5 build [labs/tools/usage] - 10https://gerrit.wikimedia.org/r/1346700 (https://phabricator.wikimedia.org/T419078) (owner: 10Krinkle) [17:03:43] (03Merged) 10jenkins-bot: docs: Update toolforge deployment from php7.4 to php8.4 [labs/tools/usage] - 10https://gerrit.wikimedia.org/r/1346702 (https://phabricator.wikimedia.org/T419078) (owner: 10Krinkle) [17:10:40] (03update) 10fnegri: Swap minio with silo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1440 (https://phabricator.wikimedia.org/T439720) [17:11:16] (03approved) 10dcaro: Swap minio with silo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1440 (https://phabricator.wikimedia.org/T439720) (owner: 10fnegri) [17:11:51] (03update) 10fnegri: Swap minio with silo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1440 (https://phabricator.wikimedia.org/T439720) [17:12:44] (03merge) 10fnegri: Swap minio with silo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1440 (https://phabricator.wikimedia.org/T439720) [17:15:08] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [lima-kilo,logging,logs-api] find a replacement for minio (s3 for loki) - https://phabricator.wikimedia.org/T439720#12381257 (10fnegri) p:05Highβ†’03Medium @dcaro and myself found a quick fix, forcing the `minio` subchart in Loki to pull the images fr... [17:18:38] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [lima-kilo,logging,logs-api] find a replacement for minio (s3 for loki) - https://phabricator.wikimedia.org/T439720#12381296 (10fnegri) Also see the upstream discussion at https://github.com/grafana-community/helm-charts/issues/830 [17:25:48] (03PS1) 10Krinkle: build: Fix PHP 8.5 build [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/1346718 (https://phabricator.wikimedia.org/T419076) [17:32:20] (03CR) 10Krinkle: "check experimental" [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/1346718 (https://phabricator.wikimedia.org/T419076) (owner: 10Krinkle) [17:37:47] (03CR) 10Jforrester: [C:03+1] "<3" [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/1346718 (https://phabricator.wikimedia.org/T419076) (owner: 10Krinkle) [17:37:48] (03open) 10jonkolbert: Fix URL encoding of special characters in DR links [toolforge-repos/wikicordo] - 10https://gitlab.wikimedia.org/toolforge-repos/wikicordo/-/merge_requests/2 [17:38:38] (03PS2) 10Krinkle: build: Fix PHP 8.5 build and update toolforge deployment to php8.4 [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/1346718 (https://phabricator.wikimedia.org/T419076) [17:38:59] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install (3) new cloud dump hosts - https://phabricator.wikimedia.org/T439754 (10RobH) 03NEW [17:40:38] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install (3) new cloud dump hosts - https://phabricator.wikimedia.org/T439754#12381419 (10RobH) a:03Andrew @Andrew, In the racking details for these 3 new hosts you listed hostnames dumps100[123], but we already have existing '... [17:40:59] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install (3) new cloud dump hosts - https://phabricator.wikimedia.org/T439754#12381428 (10RobH) [17:41:20] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install (3) dumps100[123] - https://phabricator.wikimedia.org/T439754#12381431 (10RobH) [17:46:44] (03CR) 10Krinkle: [C:03+2] build: Fix PHP 8.5 build and update toolforge deployment to php8.4 [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/1346718 (https://phabricator.wikimedia.org/T419076) (owner: 10Krinkle) [17:47:24] 10Tools, 13Patch-For-Review, 07PHP 8.4 support, 07PHP 8.5 support: labs/tools/blankpages fails its CI tests in PHP 8.4 and 8.5 - https://phabricator.wikimedia.org/T419076#12381459 (10Krinkle) 05Openβ†’03Resolved a:03Krinkle [17:52:44] (03update) 10jonkolbert: Fix URL encoding of special characters in DR links [toolforge-repos/wikicordo] - 10https://gitlab.wikimedia.org/toolforge-repos/wikicordo/-/merge_requests/2 [18:06:49] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [18:06:50] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [18:06:56] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [18:07:05] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [18:07:06] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [18:08:45] (03CR) 10Krinkle: build: Fix PHP 8.5 build and update toolforge deployment to php8.4 [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/1346718 (https://phabricator.wikimedia.org/T419076) (owner: 10Krinkle) [18:08:52] (03CR) 10Krinkle: [C:03+2] build: Fix PHP 8.5 build and update toolforge deployment to php8.4 [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/1346718 (https://phabricator.wikimedia.org/T419076) (owner: 10Krinkle) [18:09:15] (03PS1) 10Krinkle: docs: Update toolforge deployment from php7.4 to php8.4 [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346737 (https://phabricator.wikimedia.org/T419077) [18:09:36] (03PS1) 10Krinkle: build: Update phan to fix PHP 8.5 build [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346738 (https://phabricator.wikimedia.org/T419077) [18:11:22] (03CR) 10Krinkle: [C:03+2] docs: Update toolforge deployment from php7.4 to php8.4 [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346737 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:12:06] (03CR) 10Krinkle: "check experimental" [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346738 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:12:43] (03CR) 10Krinkle: [C:03+2] build: Update phan to fix PHP 8.5 build [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346738 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:17:19] (03PS2) 10Krinkle: build: Update phan to fix PHP 8.5 build [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346738 (https://phabricator.wikimedia.org/T419077) [18:17:19] (03PS2) 10Krinkle: docs: Update toolforge deployment from php7.4 to php8.4 [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346737 (https://phabricator.wikimedia.org/T419077) [18:17:28] (03CR) 10Krinkle: build: Update phan to fix PHP 8.5 build [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346738 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:17:34] (03CR) 10Krinkle: [C:03+2] build: Update phan to fix PHP 8.5 build [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346738 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:17:38] (03CR) 10Krinkle: docs: Update toolforge deployment from php7.4 to php8.4 [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346737 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:17:40] (03CR) 10Krinkle: [C:03+2] docs: Update toolforge deployment from php7.4 to php8.4 [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346737 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:21:12] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [18:21:14] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [18:21:19] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [18:21:31] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [18:21:32] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [18:22:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [18:25:32] PROBLEM - Host cloudcephosd1043 is DOWN: PING CRITICAL - Packet loss = 100% [18:27:58] (03Merged) 10jenkins-bot: build: Fix PHP 8.5 build and update toolforge deployment to php8.4 [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/1346718 (https://phabricator.wikimedia.org/T419076) (owner: 10Krinkle) [18:28:00] (03CR) 10CI reject: [V:04-1] docs: Update toolforge deployment from php7.4 to php8.4 [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346737 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:28:00] (03Merged) 10jenkins-bot: build: Update phan to fix PHP 8.5 build [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346738 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:28:02] (03Merged) 10jenkins-bot: docs: Update toolforge deployment from php7.4 to php8.4 [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1346737 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:28:28] RECOVERY - Host cloudcephosd1043 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [18:29:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [18:30:35] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [18:34:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [18:47:02] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [18:47:07] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [19:02:20] (03CR) 10Umherirrender: build: Update PHPUnit from 9.x to 10.x and further to 12.x (031 comment) [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1346694 (https://phabricator.wikimedia.org/T328919) (owner: 10Krinkle) [19:32:20] 10Tool-wikicordo: Filenames with β€œ?” are not URL‑encoded, causing incorrect deletion request URLs - https://phabricator.wikimedia.org/T418633#12381856 (10kolbert) a:03kolbert [22:47:03] (03open) 10dzahn: remove the Google Fonts import for the Blogly preview logo [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/6 (https://phabricator.wikimedia.org/T439779)