[02:28:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [02:29:03] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [02:33:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [02:34:03] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [03:53:03] (03merge) 10ladsgroup: Fix URL encoding of special characters in DR links [toolforge-repos/wikicordo] - 10https://gitlab.wikimedia.org/toolforge-repos/wikicordo/-/merge_requests/2 (owner: 10jonkolbert) [03:59:53] 10Tool-wikicordo: Filenames with “?” are not URL‑encoded, causing incorrect deletion request URLs - https://phabricator.wikimedia.org/T418633#12383402 (10Ladsgroup) 05Open→03Resolved Fixed now. Thanks! [04:25:12] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [04:25:18] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [05:11:25] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 (owner: 10renovatebot) [05:12:01] (03merge) 10countcount: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 (owner: 10renovatebot) [05:15:11] PROBLEM - Host cloudcephosd1047 is DOWN: PING CRITICAL - Packet loss = 100% [05:17:28] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 (owner: 10renovatebot) [05:17:41] RECOVERY - Host cloudcephosd1047 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [05:18:18] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [05:22:37] PROBLEM - Host cloudcephosd1047 is DOWN: PING CRITICAL - Packet loss = 100% [05:24:41] RECOVERY - Host cloudcephosd1047 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [05:25:08] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [05:25:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [05:25:30] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [05:27:55] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [05:30:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [05:30:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [05:31:07] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [05:41:02] 10Tool-coverage: Create a proper frontend for coverage.toolforge.org - https://phabricator.wikimedia.org/T438645#12383486 (10Legoktm) a:03Legoktm https://coverage.toolforge.org/ has a bunch of improvements, I still have a few more ideas on what to add. [06:13:01] (03open) 10countcount: Update yoke-derive to 0.8.4, the previous release was yanked [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/83 [06:13:45] (03update) 10countcount: Update yoke-derive to 0.8.4, the previous release was yanked [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/83 [06:20:19] (03merge) 10countcount: Update yoke-derive to 0.8.4, the previous release was yanked [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/83 [06:23:08] (03merge) 10countcount: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 (owner: 10renovatebot) [06:33:42] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/10 [06:33:48] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/10 [06:33:54] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/8 [06:34:02] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/48 [06:34:13] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/25 [06:34:13] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/8 [06:34:35] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/100 [06:34:49] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/11 [06:34:50] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/196 [06:35:02] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/48 [06:35:11] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/183 [06:35:21] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/9 [06:35:23] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/94 [06:35:45] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/44 [06:35:50] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/25 [06:35:52] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/49 [06:36:10] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/44 [06:36:35] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/101 [06:36:42] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/61 [06:36:49] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/153 [06:36:57] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/197 [06:37:05] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/100 [06:37:06] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/122 [06:37:15] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/381 [06:37:20] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/62 [06:37:24] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/120 [06:37:29] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/154 [06:37:31] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/11 [06:37:33] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/36 [06:37:34] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/196 [06:37:42] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/53 [06:37:47] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/382 [06:37:53] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/79 [06:38:00] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/121 [06:38:08] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/168 [06:38:10] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/107 [06:38:11] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/108 [06:38:17] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/95 [06:38:28] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/343 [06:38:29] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/77 [06:38:31] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/183 [06:38:32] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/70 [06:38:39] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/78 [06:38:40] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/9 [06:38:44] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/94 [06:38:51] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/44 [06:38:53] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/49 [06:39:02] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/44 [06:39:16] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/101 [06:39:18] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/61 [06:39:25] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/153 [06:39:28] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/197 [06:39:34] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/122 [06:39:37] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/381 [06:39:40] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/62 [06:39:41] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/120 [06:39:43] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/154 [06:39:45] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/36 [06:39:47] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/53 [06:39:51] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/382 [06:39:55] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/79 [06:39:59] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/121 [06:40:03] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/168 [06:40:07] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/107 [06:40:11] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/108 [06:40:15] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/95 [06:40:19] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/343 [06:40:23] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/77 [06:40:27] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/70 [06:40:31] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/78 [06:42:07] 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Add more CPUs to quibble jobs - https://phabricator.wikimedia.org/T439808 (10fgiunchedi) 03NEW [06:53:09] (03open) 10countcount: Pass query params to mysql_query as a tuple [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/32 [07:02:51] (03merge) 10countcount: Pass query params to mysql_query as a tuple [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/32 [07:16:29] (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [08:27:54] (03open) 10countcount: Fix crash comparing naive and aware datetimes in criteria checks [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/33 [08:28:09] (03update) 10countcount: Fix crash comparing naive and aware datetimes in criteria checks [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/33 [08:28:24] (03merge) 10countcount: Fix crash comparing naive and aware datetimes in criteria checks [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/33 [08:29:08] (03update) 10dcaro: global: send job events to logs-api [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380 (https://phabricator.wikimedia.org/T432570) [08:37:10] (03open) 10waldir: T432938: Match browser locales to available translations [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/27 [08:38:08] (03approved) 10waldir: Use consistent ellipsis in messages [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/26 (owner: 10amire80) [08:38:44] (03update) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [08:39:57] (03merge) 10waldir: Use consistent ellipsis in messages [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/26 (owner: 10amire80) [08:49:45] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 (owner: 10renovatebot) [08:56:26] (03update) 10dcaro: global: send job events to logs-api [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380 (https://phabricator.wikimedia.org/T432570) [08:58:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:07:29] (03update) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [09:09:18] (03update) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [09:10:35] (03update) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [09:13:20] (03update) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] (add_source_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [09:13:29] (03update) 10fnegri: global: send job events to logs-api [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380 (https://phabricator.wikimedia.org/T432570) (owner: 10dcaro) [09:14:39] (03update) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] (add_source_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [09:16:07] (03update) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] (add_source_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [09:17:09] (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [09:17:24] (03update) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] (add_source_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [09:19:59] (03update) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [09:23:10] (03update) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [09:33:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:55:02] 06tools-platform-team, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 06Traffic: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12384184 (10Jelto) The legacy CNAME `url-downloader` URL was removed from wmf-navigator with the patch above. [09:55:56] (03update) 10chimaifeanyi29: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 [09:55:59] (03update) 10chimaifeanyi29: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 [09:59:23] 06tools-infrastructure-team: Considering having cloudvirt hardware tiers - https://phabricator.wikimedia.org/T439833 (10fgiunchedi) 03NEW [10:02:16] 10Cloud-VPS, 06tools-infrastructure-team: Considering having cloudvirt hardware tiers - https://phabricator.wikimedia.org/T439833#12384212 (10taavi) [10:13:40] (03update) 10dcaro: global: send job events to logs-api [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380 (https://phabricator.wikimedia.org/T432570) [10:13:58] (03update) 10dcaro: global: send job events to logs-api [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380 (https://phabricator.wikimedia.org/T432570) [10:15:25] (03update) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [10:16:17] (03update) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] (add_source_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [10:16:48] (03update) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [10:31:27] 10Cloud-VPS, 06tools-infrastructure-team: Considering having cloudvirt hardware tiers - https://phabricator.wikimedia.org/T439833#12384340 (10fgiunchedi) [10:37:23] (03merge) 10sarai-wmf: remove the Google Fonts import for the Blogly preview logo [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/6 (https://phabricator.wikimedia.org/T439779) (owner: 10dzahn) [10:41:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [10:46:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [11:55:40] (03update) 10dcaro: global: send job events to logs-api [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380 (https://phabricator.wikimedia.org/T432570) [11:55:52] (03update) 10dcaro: global: send job events to logs-api [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380 (https://phabricator.wikimedia.org/T432570) [12:04:45] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/50 [12:51:46] (03approved) 10dcaro: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) (owner: 10fnegri) [12:57:21] (03update) 10taavi: eqiad1: cloudinfra: Add security group for WAF (requestctl) nodes [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/365 (https://phabricator.wikimedia.org/T439418) [12:57:21] (03open) 10taavi: eqiad1: cloudinfra: Add security group for WAF (requestctl) nodes [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/365 (https://phabricator.wikimedia.org/T439418) [12:57:41] (03update) 10taavi: eqiad1: cloudinfra: Add security group for WAF (requestctl) nodes [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/365 (https://phabricator.wikimedia.org/T439418) [13:08:56] (03PS1) 10Majavah: Add dummy hiddenparma password [labs/private] - 10https://gerrit.wikimedia.org/r/1347598 (https://phabricator.wikimedia.org/T439418) [13:14:17] (03update) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [13:16:16] (03update) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] (add_source_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [13:18:39] (03open) 10waldir: Update mailmap file [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/28 [13:18:40] (03update) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [13:18:56] (03update) 10waldir: Update mailmap file [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/28 [13:21:40] (03PS2) 10Majavah: Add dummy hiddenparma database password [labs/private] - 10https://gerrit.wikimedia.org/r/1347598 (https://phabricator.wikimedia.org/T439418) [13:25:18] (03approved) 10fnegri: global: send job events to logs-api [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380 (https://phabricator.wikimedia.org/T432570) (owner: 10dcaro) [13:26:16] (03open) 10dcaro: cli: add the --job-names flag [repos/cloud/toolforge/logs-cli] (add_lines_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/10 (https://phabricator.wikimedia.org/T432572) [13:31:57] 10Cloud-VPS (Project-requests): Resize legacy.wikiapiary.eqiad1.wikimedia.cloud to g4.core16.ram32.disk20 - https://phabricator.wikimedia.org/T439317#12385040 (10MarkAHershberger) Thanks, @dcaro. I tried resizing multiple times but it just returned "Error" (this can be seen in Horizon now) and no indication of w... [13:38:48] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [13:38:55] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [13:44:35] 10Cloud-VPS, 06tools-infrastructure-team: Resize legacy.wikiapiary.eqiad1.wikimedia.cloud to g4.core16.ram32.disk20 - https://phabricator.wikimedia.org/T439317#12385149 (10taavi) The current instance uses the custom `g4.cores8.ram24.disk20.ephemeral90.4xiops` flavor which includes an extra 90G extra disk parti... [13:46:00] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wmcs-uptime away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435759#12385167 (10komla) Will you be able to meet a deadline of October 31st? [13:46:10] 10Cloud-VPS (Debian Bullseye Deprecation), 10Continuous-Integration-Infrastructure: Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12385168 (10komla) Will you be able to meet a deadline of October 31st? [13:50:26] 10Cloud-VPS (Debian Bullseye Deprecation), 10SRE-swift-storage: Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12385202 (10komla) Will you be able to meet a deadline of October 31st? [13:51:44] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12385216 (10fgiunchedi) Ok `cephosd.cfg` is not going to work, and that's not surprising since I used the wrong recipe as these systems are efi, not bios,... [13:53:52] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12385231 (10komla) Will you be able to meet a deadline of October 31st? [13:57:14] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wikidocumentaries away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435789#12385254 (10komla) Will you be able to meet a deadline of October 31st? [14:03:39] (03merge) 10dcaro: global: send job events to logs-api [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380 (https://phabricator.wikimedia.org/T432570) [14:06:17] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:06:21] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [14:07:25] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.573-20261001140352-7830a972 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1441 (https://phabricator.wikimedia.org/T432570) [14:07:33] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.573-20261001140352-7830a972 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1441 (https://phabricator.wikimedia.org/T432570) [14:11:40] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12385345 (10fgiunchedi) cc @Jclark-ctr @VRiley-WMF @wiki_willy JFYI [14:12:39] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:19:47] (03update) 10waldir: Update mailmap file [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/28 [14:21:24] (03PS1) 10Sohom Datta: Revert "EditCounter: fix local-time breaking timecard" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1347632 [14:21:35] (03update) 10waldir: Update information about contributors and history of the project [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/28 [14:25:44] (03PS2) 10Sohom Datta: Revert "EditCounter: fix local-time breaking timecard" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1347632 (https://phabricator.wikimedia.org/T439249) [14:27:16] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:28:14] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:42:37] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [14:43:11] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:51:22] (03approved) 10dcaro: jobs-api: bump to 0.0.573-20261001140352-7830a972 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1441 (https://phabricator.wikimedia.org/T432570) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:51:27] (03merge) 10dcaro: jobs-api: bump to 0.0.573-20261001140352-7830a972 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1441 (https://phabricator.wikimedia.org/T432570) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:51:51] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [jobs-api] Send start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12385696 (10dcaro) 05In progress→03Resolved [15:07:00] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881408868 (https://github.com/cluebotng/component-configs/commits/2eac12b733c0a202e1363dde61dd15496bc3d385) [15:07:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [15:09:26] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881820713 (https://github.com/cluebotng/component-configs/commits/860f13c0de005840e3e832d5dc6d17d9a4863409) [15:09:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [15:25:29] (03open) 10dcaro: cli: add --deployment-ids flag [repos/cloud/toolforge/logs-cli] (add_jobs_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/11 (https://phabricator.wikimedia.org/T432572) [15:30:57] (03open) 10dcaro: cli: add --build-ids flag [repos/cloud/toolforge/logs-cli] (add_deployments_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/12 (https://phabricator.wikimedia.org/T432572) [15:33:12] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/8 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:33:18] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/8 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:36:30] 10Toolforge, 06tools-platform-team: [logs-cli,logs-api] add functional tests for all the new "finished" features - https://phabricator.wikimedia.org/T439884 (10dcaro) 03NEW [15:39:35] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/9 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:42:12] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/9 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:43:11] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/9 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:43:34] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/9 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:48:32] (03open) 10dcaro: d/changelog: bump to 0.0.3 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/13 (https://phabricator.wikimedia.org/T439528) [15:51:22] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-cli [15:51:41] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12386078 (10fnegri) One alternative that we discussed today with @dcaro and @Raymond_Ndibe could be to use t... [15:51:54] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli [15:52:23] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-cli [15:52:50] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli [15:53:23] (03approved) 10dcaro: d/changelog: bump to 0.0.3 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/13 (https://phabricator.wikimedia.org/T439528) [15:53:28] (03merge) 10dcaro: d/changelog: bump to 0.0.3 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/13 (https://phabricator.wikimedia.org/T439528) [15:54:00] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli] support run-logs and deploy events - https://phabricator.wikimedia.org/T439528#12386088 (10dcaro) 05In progress→03Resolved [15:56:21] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Create template for maintainer alerts emails - https://phabricator.wikimedia.org/T437081#12386117 (10dcaro) (I think I did not miss any MR xd) [16:06:33] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12386226 (10dcaro) [16:10:15] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] show a warning message stating it's in beta/unstable - https://phabricator.wikimedia.org/T439581#12386235 (10dcaro) After giving it some though, I think this should be added on the cli side (not on the API), as the API might be used by the jobs cli t... [16:15:25] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [16:15:31] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [16:20:18] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] show a warning message stating it's in beta/unstable - https://phabricator.wikimedia.org/T439581#12386269 (10dcaro) a:03dcaro [16:20:23] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] show a warning message stating it's in beta/unstable - https://phabricator.wikimedia.org/T439581#12386271 (10dcaro) 05Open→03In progress [16:20:49] (03open) 10dcaro: cli: add warning message for beta status [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/14 (https://phabricator.wikimedia.org/T439581) [16:22:56] (03update) 10dcaro: cli: add warning message for beta status [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/14 (https://phabricator.wikimedia.org/T439581) [16:23:29] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] allow specifying `source_repo`+`ref` for the config - https://phabricator.wikimedia.org/T402764#12386282 (10dcaro) a:05dcaro→03None [16:25:50] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [16:25:51] (03update) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [16:25:54] (03update) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [16:25:56] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [16:28:27] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [16:33:20] (03update) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [16:34:28] (03update) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [16:57:25] PROBLEM - Host cloudcephosd1042 is DOWN: PING CRITICAL - Packet loss = 100% [17:00:41] RECOVERY - Host cloudcephosd1042 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [17:01:10] (03CR) 10Herron: [C:03+1] alert: add a dummy Klaxon managers incident URL [labs/private] - 10https://gerrit.wikimedia.org/r/1345218 (owner: 10Andrea Denisse) [17:10:52] (03update) 10chimaifeanyi29: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 [18:17:08] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [18:17:14] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [18:21:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [18:22:40] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [18:22:47] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [18:22:48] 10Cloud-VPS, 06tools-infrastructure-team: Resize legacy.wikiapiary.eqiad1.wikimedia.cloud to g4.core16.ram32.disk20 - https://phabricator.wikimedia.org/T439317#12386895 (10MarkAHershberger) I see one unused 90G disk. Is this it? ` $ sudo lsblk /dev/sdd -O --json { "blockdevices": [ { "align... [18:26:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:08:33] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [19:08:38] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [19:15:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:19:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [19:20:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:23:13] PROBLEM - Host cloudcephosd1042 is DOWN: PING CRITICAL - Packet loss = 100% [19:25:41] RECOVERY - Host cloudcephosd1042 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [19:26:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:27:43] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [19:31:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:47:13] PROBLEM - Host cloudcephosd1016 is DOWN: PING CRITICAL - Packet loss = 100% [19:48:34] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [19:48:41] RECOVERY - Host cloudcephosd1016 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [19:52:13] PROBLEM - Host cloudcephosd1016 is DOWN: PING CRITICAL - Packet loss = 100% [19:52:31] (03update) 10chimaifeanyi29: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 [19:53:00] (03update) 10chimaifeanyi29: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 [19:53:41] RECOVERY - Host cloudcephosd1016 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [19:54:12] (03open) 10rluts: Add database integration and project management features [toolforge-repos/findsmith] - 10https://gitlab.wikimedia.org/toolforge-repos/findsmith/-/merge_requests/1 [19:54:33] (03merge) 10rluts: Add database integration and project management features [toolforge-repos/findsmith] - 10https://gitlab.wikimedia.org/toolforge-repos/findsmith/-/merge_requests/1 [19:55:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:56:03] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [20:00:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [20:07:59] (03open) 10rluts: Update database configuration to use ToolsDB env vars [toolforge-repos/findsmith] - 10https://gitlab.wikimedia.org/toolforge-repos/findsmith/-/merge_requests/2 [20:08:18] (03merge) 10rluts: Update database configuration to use ToolsDB env vars [toolforge-repos/findsmith] - 10https://gitlab.wikimedia.org/toolforge-repos/findsmith/-/merge_requests/2 [20:15:38] (03update) 10chimaifeanyi29: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 [20:19:42] (03update) 10chimaifeanyi29: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 [20:57:04] (03update) 10waldir: Update information about contributors and history of the project [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/28 [21:35:14] 10Toolforge, 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12387514 (10BLiviero-WMF) Had Claude munge wikimedia-cloud-feed and SAL for the past 3 months to profile now many instances of nfs worker down it saw. capturing for posterity: **Alerts o... [22:40:57] 10VPS-project-Wikistats: upgrade wikistats cloud VPS project to trixie - https://phabricator.wikimedia.org/T401859#12387670 (10Dzahn) Migrated wikistats from wikistats-bookworm to wikistats-trixie on 2026-10-01. * Database copied with the new migrate-db.sh script (MR pending), which streams a fresh mysql... [22:41:46] 10VPS-project-Wikistats: upgrade wikistats cloud VPS project to trixie - https://phabricator.wikimedia.org/T401859#12387671 (10Dzahn) a:03Dzahn [23:20:55] 10Tool-wikinewsie, 07good first task, 13Patch-For-Review: Wikinewsie recent changes should link to article card - https://phabricator.wikimedia.org/T438640#12387704 (10ZhaoFJx) 05Open→03In progress [23:28:12] (03CR) 10MusikAnimal: [C:03+2] Revert "EditCounter: fix local-time breaking timecard" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1347632 (https://phabricator.wikimedia.org/T439249) (owner: 10Sohom Datta) [23:29:13] (03Merged) 10jenkins-bot: Revert "EditCounter: fix local-time breaking timecard" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1347632 (https://phabricator.wikimedia.org/T439249) (owner: 10Sohom Datta)