[00:27:42] 10wikitech.wikimedia.org: Wikitech static is down with a 502 HTTP error (Bad Gateway) - https://phabricator.wikimedia.org/T427081#12506721 (10bd808) a:05Wargo→03Andrew [00:57:20] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12506817 (10Amdrel) > Would it be possible to have "media actions" higher up within the tools? Right now it is near the bottom I've confirmed this is possible... [01:05:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:46:21] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434982 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06) [01:46:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [01:46:49] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434850 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06) [01:46:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [01:47:21] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434964 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06) [01:47:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [01:47:38] 10Tool-masto-collab: masto-collab: Don't hardcode Meta-Wiki for OAuth - https://phabricator.wikimedia.org/T336221#12506893 (10Legoktm) 05Open→03Resolved a:03Legoktm This was done back in https://gitlab.wikimedia.org/toolforge-repos/masto-collab/-/commit/5c169f46b5b15e2724bc94baaff0166f42a68560 [02:00:00] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:06:08] 10Tool-Pageviews: Massviews fails with "Error querying the replica database" when the source is a Commons category - https://phabricator.wikimedia.org/T443242#12506907 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal Thanks for reporting! [02:07:09] (03open) 10raymond-ndibe: toolforge-cd,utils: auto-create phab tasks for the monthly upgrade MRs [repos/cloud/cicd/gitlab-ci] (remove_needs_review_label) - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/102 (https://phabricator.wikimedia.org/T443234) [02:24:06] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [02:25:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:29:06] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [02:40:28] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567113718 (https://github.com/cluebotng/component-configs/commits/10880256092163e5d6d4d97b1d2a0453e51c1f2c) [02:40:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [02:40:51] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216275 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d) [02:40:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [02:41:37] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216287 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d) [02:41:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [02:41:47] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216266 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d) [02:41:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [03:06:27] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/209 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:08:01] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/209 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:09:02] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/209 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:12:21] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.230-20261103030914-7d05f699 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1486 [03:12:30] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.230-20261103030914-7d05f699 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1486 [03:13:21] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41570372141 (https://github.com/cluebotng/component-configs/commits/9e06eef94c362f2813b0a34a82356b7d503c3f4f) [03:13:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [03:20:36] 06tools-platform-team: [all] first of the month dependency upgrades for 03-11-2026 - https://phabricator.wikimedia.org/T443392 (10Raymond_Ndibe) 03NEW [03:20:48] 06tools-platform-team: [all] first of the month dependency upgrades for 03-11-2026 - https://phabricator.wikimedia.org/T443392#12506984 (10Raymond_Ndibe) 05Open→03In progress [03:41:37] (03open) 10raymond-ndibe: build: pin kubernetes to the v33 client [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/65 (https://phabricator.wikimedia.org/T443392) [03:41:49] (03update) 10raymond-ndibe: build: pin kubernetes to the v33 client [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/65 (https://phabricator.wikimedia.org/T443392) [03:42:10] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/63 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:42:22] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/64 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:42:40] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] (pin_kubernetes_33) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/64 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:43:06] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] (pin_kubernetes_33) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/63 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:50:12] 06tools-platform-team, 13Patch-For-Review: [all] first of the month dependency upgrades for 03-11-2026 - https://phabricator.wikimedia.org/T443392#12506989 (10Raymond_Ndibe) [03:51:00] (03approved) 10raymond-ndibe: build: pin kubernetes to the v33 client [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/65 (https://phabricator.wikimedia.org/T443392) [03:51:06] (03merge) 10raymond-ndibe: build: pin kubernetes to the v33 client [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/65 (https://phabricator.wikimedia.org/T443392) [03:51:12] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/63 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:51:13] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/64 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:51:14] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/63 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:51:38] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/64 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:51:48] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/63 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:53:14] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/63 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:53:44] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.98-20261103035121-dac69b7e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1495 (https://phabricator.wikimedia.org/T443392) [03:55:46] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.99-20261103035333-965379ef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1495 (https://phabricator.wikimedia.org/T443392) [03:55:54] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.99-20261103035333-965379ef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1495 (https://phabricator.wikimedia.org/T443392) [03:56:13] 06tools-platform-team, 13Patch-For-Review: [all] first of the month dependency upgrades for 03-11-2026 - https://phabricator.wikimedia.org/T443392#12506992 (10Raymond_Ndibe) **List of the mrs (since this is not yet automated):** - https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/1... [03:56:31] 06tools-platform-team, 13Patch-For-Review: [all] first of the month dependency upgrades for 03-11-2026 - https://phabricator.wikimedia.org/T443392#12506993 (10Raymond_Ndibe) [03:59:03] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/64 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:59:11] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/64 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [04:01:18] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/64 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [04:03:33] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.100-20261103040136-f3db824c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1495 (https://phabricator.wikimedia.org/T443392) [04:03:36] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.100-20261103040136-f3db824c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1495 (https://phabricator.wikimedia.org/T443392) [04:11:12] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988#12507000 (10Raymond_Ndibe) a:03Raymond_Ndibe [04:17:54] 10Toolforge, 06tools-platform-team: [*-api] Add a check for consistency between the tool in the path and X-Toolforge-Tool - https://phabricator.wikimedia.org/T441196#12507019 (10Raymond_Ndibe) a:03Raymond_Ndibe [04:18:17] 10Toolforge, 06tools-platform-team: [*-api] Add a check for consistency between the tool in the path and X-Toolforge-Tool - https://phabricator.wikimedia.org/T441196#12507021 (10Raymond_Ndibe) 05Open→03In progress [04:19:05] 10Toolforge, 06tools-platform-team: [*-api] don't allow non-canonical urls anywhere - https://phabricator.wikimedia.org/T441195#12507022 (10Raymond_Ndibe) a:03Raymond_Ndibe [04:19:19] 10Toolforge, 06tools-platform-team: [*-api] don't allow non-canonical urls anywhere - https://phabricator.wikimedia.org/T441195#12507024 (10Raymond_Ndibe) 05Open→03In progress [04:42:46] (03open) 10ibrahemqasim: Add missing Commons image cleanup [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/31 [04:43:38] (03update) 10ibrahemqasim: Add missing Commons image cleanup [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/31 [04:44:22] (03merge) 10ibrahemqasim: Add missing Commons image cleanup [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/31 [04:55:04] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12507053 (10Redmin) [05:07:46] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12507064 (10Redmin) [05:09:21] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41581755584 (https://github.com/cluebotng/component-configs/commits/ad0a4ebfac738d2c3b772e79b296d5c00c79c38e) [05:09:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [05:13:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [05:13:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:13:22] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582112461 (https://github.com/cluebotng/component-configs/commits/753eb7b9e738029a5443693413aff58e1d3fa349) [05:13:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [05:18:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [05:18:06] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:18:12] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582696321 (https://github.com/cluebotng/component-configs/commits/085afd71093b712517702f3975ce1a3bea759e1e) [05:18:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [05:18:41] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582696228 (https://github.com/cluebotng/component-configs/commits/085afd71093b712517702f3975ce1a3bea759e1e) [05:18:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [05:26:06] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:27:51] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41583533049 (https://github.com/cluebotng/component-configs/commits/417cde41c250a5f0b6f21e72ce00a343ad270295) [05:27:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [05:31:06] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:35:51] (03update) 10ibrahemqasim: Fix some bugs [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/28 [05:37:06] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:42:06] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:51:40] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41586264061 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a) [05:51:41] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41586263970 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a) [05:51:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [05:51:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [05:54:01] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264019 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a) [05:54:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [05:54:04] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264139 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a) [05:54:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [05:54:13] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264050 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a) [05:54:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [05:54:15] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264114 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a) [05:54:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [05:54:55] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264099 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a) [05:54:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [05:55:12] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586147096 (https://github.com/cluebotng/component-configs/commits/6295be2043be15b64a1f5a2667c2f9f11b4194b5) [05:55:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [05:55:26] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586147205 (https://github.com/cluebotng/component-configs/commits/6295be2043be15b64a1f5a2667c2f9f11b4194b5) [05:55:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [05:59:46] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264002 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a) [05:59:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [06:00:53] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264177 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a) [06:00:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [06:07:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [06:07:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [06:12:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [06:12:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [06:19:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [06:19:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [06:24:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [06:24:06] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [06:25:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [06:25:04] 10Cloud-VPS, 06tools-infrastructure-team, 06Test Platform (San Miguel de Allende 31): Evaluate virtio-net multiqueue for Cloud VPS instances - https://phabricator.wikimedia.org/T443396 (10Mhurd) 03NEW [06:25:51] 10Cloud-VPS, 06tools-infrastructure-team, 06Test Platform (San Miguel de Allende 31): Test/benchmark virtio multiqueue for integration-cache-proxy - https://phabricator.wikimedia.org/T440418#12507135 (10Mhurd) [06:25:54] 10Cloud-VPS, 06tools-infrastructure-team, 06Test Platform (San Miguel de Allende 31): Evaluate virtio-net multiqueue for Cloud VPS instances - https://phabricator.wikimedia.org/T443396#12507136 (10Mhurd) [06:26:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [06:29:06] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [06:31:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [06:31:39] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41590047860 (https://github.com/cluebotng/component-configs/commits/b727998497e3e9aad0bbeead0c3ab4000f61584c) [06:31:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [06:34:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [06:34:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [06:35:53] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41590277348 (https://github.com/cluebotng/component-configs/commits/be302650e64737354887b8a0d3765c836f16937f) [06:35:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [06:39:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [06:39:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:10:06] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:15:06] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:16:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [07:16:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:20:21] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:21:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [07:22:06] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:32:06] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:40:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [07:40:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:45:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [07:45:06] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:50:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [07:50:06] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:55:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [07:55:06] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:56:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [07:56:06] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:00:16] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:00:21] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:01:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:01:06] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:05:16] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:05:21] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:23:06] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:28:06] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:31:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:31:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:36:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:36:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:45:38] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41605215680 (https://github.com/cluebotng/component-configs/commits/874f44a0ab1b450c296132a9246bea96e0b7d25b) [08:45:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [08:47:06] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41605215687 (https://github.com/cluebotng/component-configs/commits/874f44a0ab1b450c296132a9246bea96e0b7d25b) [08:47:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [09:25:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [09:25:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:26:13] (03PS1) 10Btullis: ceph: Remove dummy keys for the unused osd.$HOSTNAME entities [labs/private] - 10https://gerrit.wikimedia.org/r/1361813 (https://phabricator.wikimedia.org/T437717) [09:30:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [09:30:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:40:34] FIRING: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [09:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [10:08:00] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#12507572 (10ForzaGreen) https://superset.wmcloud.org is currently down. Has it already been shut down? [10:09:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [10:09:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [10:14:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [10:14:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [10:14:23] 10Quarry, 06tools-platform-team: [quarry] started failing when results directory does not exist - https://phabricator.wikimedia.org/T443411 (10dcaro) 03NEW [10:16:15] (03update) 10renovatebot: Update Rust crate replica-pools to v0.1.3 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/94 [10:16:17] (03open) 10renovatebot: Update Rust crate replica-pools to v0.1.3 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/94 [10:16:50] 10Quarry, 06tools-platform-team: [quarry,infra] The cluster can't be scaled and tofu tries to get it to 3 workers - https://phabricator.wikimedia.org/T443413 (10dcaro) 03NEW [10:24:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [10:24:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [10:25:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:25:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [10:29:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [10:29:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [10:33:47] (03CR) 10Btullis: [V:03+2 C:03+2] ceph: Remove dummy keys for the unused osd.$HOSTNAME entities [labs/private] - 10https://gerrit.wikimedia.org/r/1361813 (https://phabricator.wikimedia.org/T437717) (owner: 10Btullis) [10:45:38] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: 2026-10-21 Toolforge and Cloud VPS down - https://phabricator.wikimedia.org/T441908#12507847 (10Nirmos) [10:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [10:51:17] 10Quarry, 06tools-platform-team: [quarry] started failing when results directory does not exist - https://phabricator.wikimedia.org/T443411#12507862 (10dcaro) p:05Triage→03Medium [10:51:27] 10Quarry, 06tools-platform-team: [quarry] started failing when results directory does not exist - https://phabricator.wikimedia.org/T443411#12507865 (10dcaro) a:05dcaro→03None [10:54:58] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41622611776 (https://github.com/cluebotng/component-configs/commits/5393f63d09ee5ee78540551dbd2532edbb754bdd) [10:55:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [11:08:43] (03open) 10dcaro: cli: load entry points as subcommands too [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/81 (https://phabricator.wikimedia.org/T430988) [11:09:26] (03open) 10dcaro: cli: add aliases [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/104 (https://phabricator.wikimedia.org/T430988) [11:10:07] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988#12507913 (10dcaro) Just pushed the MRs with the code I was working on for this a couple weeks back [11:15:03] 10Tools: templatetransclusioncheck - https://phabricator.wikimedia.org/T443148#12507936 (10Aklapper) https://toolsadmin.wikimedia.org/tools/id/templatetransclusioncheck does not list an issue tracker for that tool, so it's unclear where to report issues with it. CC'ing @Adithyak1997 and @Chameleon222 - clarifica... [11:15:21] 10Tools: templatetransclusioncheck displays "no healthy upstream" error - https://phabricator.wikimedia.org/T443148#12507940 (10Aklapper) [11:24:35] (03update) 10dcaro: cli: add aliases [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/104 (https://phabricator.wikimedia.org/T430988) [11:25:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:29:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [11:29:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [11:34:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [11:34:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [11:42:31] (03update) 10dcaro: cli: add aliases [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/104 (https://phabricator.wikimedia.org/T430988) [11:43:41] (03update) 10dcaro: cli: add aliases [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/104 (https://phabricator.wikimedia.org/T430988) [11:45:13] (03update) 10rabuma: books: Return 404 when a book is not found (T442783) [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/16 (https://phabricator.wikimedia.org/T442783) [11:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:45:48] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: Catalog update sets view_count to 0 when the pageviews API request fails - https://phabricator.wikimedia.org/T442788#12508106 (10Saiphani02) [11:48:17] 10Tool-lingua-datasets, 10Toolforge (Quota-requests), 06tools-platform-team: Setup Lingua-data.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12508132 (10fgiunchedi) >>! In T440689#12504616, @Yug wrote: > Hello @fgiunchedi , > I guessed you were focused on PHP2SHELL, all good. 👍🏼 > >... [11:49:59] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: Random sort can repeat or skip books between pages - https://phabricator.wikimedia.org/T443122#12508140 (10Saiphani02) [12:05:05] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616290 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828) [12:05:06] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616265 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828) [12:05:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [12:05:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [12:05:10] !log tools.cluebotng-editsets Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616324 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828) [12:05:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [12:05:14] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616245 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828) [12:05:14] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616142 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828) [12:05:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [12:05:16] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616232 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828) [12:05:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [12:05:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [12:07:18] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616438 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828) [12:07:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [12:08:25] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616200 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828) [12:08:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [12:08:37] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616293 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828) [12:08:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [12:23:17] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.576-20261102120046-7cfdc5d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1485 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:23:18] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.576-20261102120046-7cfdc5d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1485 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:23:28] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.576-20261102120046-7cfdc5d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1485 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:23:38] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.92-20261102120210-2bbef1d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1491 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:23:43] (03approved) 10raymond-ndibe: ingress-admission: bump to 0.0.92-20261102120210-2bbef1d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1491 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:23:50] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.92-20261102120210-2bbef1d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1491 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:23:53] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.92-20261102120210-2bbef1d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1491 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:24:07] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.91-20261102115833-5d4ddff4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1487 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:24:12] (03approved) 10raymond-ndibe: envvars-api: bump to 0.0.91-20261102115833-5d4ddff4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1487 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:24:19] (03update) 10dcaro: cli: add aliases [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/104 (https://phabricator.wikimedia.org/T430988) [12:24:23] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.91-20261102115833-5d4ddff4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1487 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:24:26] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.91-20261102115833-5d4ddff4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1487 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:24:37] (03merge) 10raymond-ndibe: ingress-admission: bump to 0.0.92-20261102120210-2bbef1d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1491 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:24:47] (03update) 10raymond-ndibe: envvars-admission: bump to 0.0.44-20261102115937-514d5978 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1489 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:24:53] (03approved) 10raymond-ndibe: envvars-admission: bump to 0.0.44-20261102115937-514d5978 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1489 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:25:19] (03update) 10raymond-ndibe: envvars-admission: bump to 0.0.44-20261102115937-514d5978 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1489 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:25:19] (03update) 10raymond-ndibe: envvars-admission: bump to 0.0.44-20261102115937-514d5978 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1489 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:25:37] (03update) 10raymond-ndibe: components-api: bump to 0.0.230-20261103030914-7d05f699 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1486 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:25:39] (03approved) 10raymond-ndibe: components-api: bump to 0.0.230-20261103030914-7d05f699 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1486 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:25:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:25:44] (03update) 10raymond-ndibe: components-api: bump to 0.0.230-20261103030914-7d05f699 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1486 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:25:51] (03merge) 10raymond-ndibe: envvars-admission: bump to 0.0.44-20261102115937-514d5978 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1489 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [12:27:06] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:27:09] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.91-20261102115833-5d4ddff4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1487 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:17] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.91-20261102115833-5d4ddff4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1487 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:40] (03merge) 10raymond-ndibe: envvars-api: bump to 0.0.91-20261102115833-5d4ddff4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1487 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:00] (03update) 10raymond-ndibe: components-api: bump to 0.0.230-20261103030914-7d05f699 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1486 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:05] (03update) 10raymond-ndibe: components-api: bump to 0.0.230-20261103030914-7d05f699 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1486 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:32] (03merge) 10raymond-ndibe: components-api: bump to 0.0.230-20261103030914-7d05f699 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1486 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:49] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.157-20261102115213-78a28912 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1483 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:53] (03approved) 10raymond-ndibe: builds-builder: bump to 0.0.157-20261102115213-78a28912 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1483 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:56] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.157-20261102115213-78a28912 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1483 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:31:33] (03merge) 10raymond-ndibe: builds-builder: bump to 0.0.157-20261102115213-78a28912 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1483 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:31:48] (03update) 10raymond-ndibe: builds-api: bump to 0.0.223-20261102121145-d6532fb9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1462 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:31:51] (03approved) 10raymond-ndibe: builds-api: bump to 0.0.223-20261102121145-d6532fb9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1462 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:32:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [12:32:03] (03update) 10raymond-ndibe: builds-api: bump to 0.0.223-20261102121145-d6532fb9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1462 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:32:06] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:33:27] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988#12508333 (10Raymond_Ndibe) a:05Raymond_Ndibe→03dcaro [12:34:22] (03merge) 10raymond-ndibe: builds-api: bump to 0.0.223-20261102121145-d6532fb9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1462 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:36:08] (03update) 10semajk: docs: fix errors and improve local setup instructions in README [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/8 [12:36:58] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.112-20261102121143-e1e5d0dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1484 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:36:58] (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.112-20261102121143-e1e5d0dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1484 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:37:06] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:37:17] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.112-20261102121143-e1e5d0dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1484 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:37:26] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.112-20261102121143-e1e5d0dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1484 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:37:49] (03update) 10raymond-ndibe: alerts-api: bump to 0.0.16-20261102121525-533da816 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1481 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:37:51] (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.112-20261102121143-e1e5d0dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1484 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:37:53] (03approved) 10raymond-ndibe: alerts-api: bump to 0.0.16-20261102121525-533da816 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1481 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:37:59] (03update) 10raymond-ndibe: alerts-api: bump to 0.0.16-20261102121525-533da816 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1481 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:38:29] (03update) 10dcaro: cli: add aliases [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/104 (https://phabricator.wikimedia.org/T430988) [12:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:47:46] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12508389 (10TLepage-WMF) Weekly progress summaries since the last one, from [[ https://app.asana.com/1/3758245663860/project/1219393658211416/overview/1219397469702937 | As... [12:55:00] (03update) 10dcaro: cli: load entry points as subcommands too [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/81 (https://phabricator.wikimedia.org/T430988) [12:56:35] (03update) 10dcaro: cli: add aliases [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/104 (https://phabricator.wikimedia.org/T430988) [12:59:33] (03approved) 10raymond-ndibe: jobs-emailer: bump to 0.0.100-20261103040136-f3db824c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1495 (https://phabricator.wikimedia.org/T443392) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:59:34] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.100-20261103040136-f3db824c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1495 (https://phabricator.wikimedia.org/T443392) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:59:36] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.100-20261103040136-f3db824c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1495 (https://phabricator.wikimedia.org/T443392) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:00:06] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-internal-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:00:22] (03merge) 10raymond-ndibe: alerts-api: bump to 0.0.16-20261102121525-533da816 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1481 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:00:52] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.100-20261103040136-f3db824c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1495 (https://phabricator.wikimedia.org/T443392) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:00:53] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.100-20261103040136-f3db824c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1495 (https://phabricator.wikimedia.org/T443392) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:01:34] (03merge) 10raymond-ndibe: jobs-emailer: bump to 0.0.100-20261103040136-f3db824c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1495 (https://phabricator.wikimedia.org/T443392) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:01:54] (03open) 10dcaro: functional-tests: use the new aliases for components-cli [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1496 (https://phabricator.wikimedia.org/T430988) [13:02:10] (03update) 10dcaro: functional-tests: use the new aliases for components-cli [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1496 (https://phabricator.wikimedia.org/T430988) [13:02:20] (03update) 10dcaro: cli: add aliases [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/104 (https://phabricator.wikimedia.org/T430988) [13:02:36] (03update) 10dcaro: functional-tests: use the new aliases for components-cli [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1496 (https://phabricator.wikimedia.org/T430988) [13:05:06] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-internal-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:06:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [13:06:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:15:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:16:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [13:16:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:17:12] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:20:34] (03update) 10raymond-ndibe: logs-api: bump to 0.0.60-20261102121901-64d4bc00 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1493 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:20:39] (03approved) 10raymond-ndibe: logs-api: bump to 0.0.60-20261102121901-64d4bc00 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1493 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:20:41] (03update) 10raymond-ndibe: logs-api: bump to 0.0.60-20261102121901-64d4bc00 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1493 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:20:45] (03update) 10raymond-ndibe: logs-api: bump to 0.0.60-20261102121901-64d4bc00 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1493 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:21:14] (03merge) 10raymond-ndibe: logs-api: bump to 0.0.60-20261102121901-64d4bc00 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1493 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:22:12] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:29:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [13:29:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:34:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [13:34:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:34:49] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12508674 (10Redmin) [13:35:06] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12508677 (10Redmin) [13:36:13] (03approved) 10dcaro: utils,README: clean up references to 'Needs review' label [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1494 (https://phabricator.wikimedia.org/T443238) (owner: 10raymond-ndibe) [13:37:42] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.79-20261102121143-888c00f7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1482 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:37:44] (03approved) 10raymond-ndibe: maintain-harbor: bump to 0.0.79-20261102121143-888c00f7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1482 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:37:45] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.79-20261102121143-888c00f7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1482 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:37:51] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.79-20261102121143-888c00f7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1482 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:38:18] (03merge) 10raymond-ndibe: maintain-harbor: bump to 0.0.79-20261102121143-888c00f7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1482 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:48:40] 10Tool-wsindex: Books API sends hundreds of database queries for one page of books - https://phabricator.wikimedia.org/T443443 (10RCodes) 03NEW [13:55:30] 10Cloud-VPS, 06tools-infrastructure-team: Copy requestctl known-fingerprints file to WMCS instance - https://phabricator.wikimedia.org/T441785#12508767 (10taavi) 05Open→03Resolved [14:14:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [14:14:56] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [14:15:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [14:16:25] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [14:25:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:26:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:34:45] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [14:41:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [14:58:38] (03approved) 10dcaro: models: use a type statement for the ComponentInfo alias [repos/cloud/toolforge/components-api] (add_component_parsing_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/206 (https://phabricator.wikimedia.org/T362077) (owner: 10raymond-ndibe) [14:58:48] (03update) 10dcaro: config: set component_type in the generated config [repos/cloud/toolforge/components-api] (use_type_statement_for_component_info) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/207 (https://phabricator.wikimedia.org/T362077) (owner: 10raymond-ndibe) [14:59:21] (03approved) 10dcaro: config: set component_type in the generated config [repos/cloud/toolforge/components-api] (use_type_statement_for_component_info) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/207 (https://phabricator.wikimedia.org/T362077) (owner: 10raymond-ndibe) [14:59:32] (03update) 10dcaro: config: warn about components without component_type [repos/cloud/toolforge/components-api] (set_component_type_in_generated_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T362077) (owner: 10raymond-ndibe) [15:00:01] (03update) 10zhaofjx: add support for URL parameters ?h and ?d to set the time window [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/33 (owner: 10chimaifeanyi29) [15:00:02] (03update) 10zhaofjx: add support for URL parameters ?h and ?d to set the time window [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/33 (owner: 10chimaifeanyi29) [15:04:29] (03approved) 10zhaofjx: add support for URL parameters ?h and ?d to set the time window [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/33 (owner: 10chimaifeanyi29) [15:04:43] (03merge) 10zhaofjx: add support for URL parameters ?h and ?d to set the time window [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/33 (owner: 10chimaifeanyi29) [15:07:25] 10Tool-wikinewsie, 07good first task, 13Patch-For-Review: Specify Wikinewsie time window in url - https://phabricator.wikimedia.org/T442918#12509210 (10ZhaoFJx) 05Open→03Resolved [15:08:05] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [15:09:06] (03approved) 10dcaro: config: warn about components without component_type [repos/cloud/toolforge/components-api] (set_component_type_in_generated_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T362077) (owner: 10raymond-ndibe) [15:09:15] (03update) 10dcaro: feat: add webservice component type support [repos/cloud/toolforge/components-api] (warn_on_missing_component_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/199 (https://phabricator.wikimedia.org/T362077) (owner: 10raymond-ndibe) [15:11:33] 10Tool-delintbot, 13Patch-For-Review: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12509247 (10Redmin) [15:13:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [15:13:05] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [15:18:54] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12509267 (10Sfaci) @Jdforrester-WMF `repos/data-engineering/growthbook` is just a pipeline we use to build our own docker image to be able... [15:19:54] (03merge) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.204-20261102120232-9501a8a8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1488 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:20:14] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.82-20261102120221-e37faecf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1492 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:20:16] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.82-20261102120221-e37faecf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1492 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:20:19] (03approved) 10raymond-ndibe: registry-admission: bump to 0.0.82-20261102120221-e37faecf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1492 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:20:23] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.82-20261102120221-e37faecf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1492 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:20:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:20:46] (03update) 10raymond-ndibe: volume-admission: bump to 0.0.90-20261102115905-9b89b648 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1490 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:24:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [15:24:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [15:24:15] (03approved) 10raymond-ndibe: volume-admission: bump to 0.0.90-20261102115905-9b89b648 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1490 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:25:14] (03merge) 10raymond-ndibe: volume-admission: bump to 0.0.90-20261102115905-9b89b648 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1490 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:25:40] (03update) 10raymond-ndibe: utils,README: clean up references to 'Needs review' label [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1494 (https://phabricator.wikimedia.org/T443238) [15:25:52] (03approved) 10raymond-ndibe: utils,README: clean up references to 'Needs review' label [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1494 (https://phabricator.wikimedia.org/T443238) [15:25:56] (03update) 10raymond-ndibe: utils,README: clean up references to 'Needs review' label [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1494 (https://phabricator.wikimedia.org/T443238) [15:26:08] (03merge) 10raymond-ndibe: utils,README: clean up references to 'Needs review' label [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1494 (https://phabricator.wikimedia.org/T443238) [15:27:05] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance project-proxy-puppetserver-1 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:28:27] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12509352 (10Jdforrester-WMF) >>! In T437512#12509267, @Sfaci wrote: > @Jdforrester-WMF `repos/data-engineering/growthbook` is just a pipel... [15:28:33] (03open) 10r4356th: Replace Bengali numerals in width values [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/46 (https://phabricator.wikimedia.org/T421530) [15:28:43] (03PS1) 10David Caro: node22: add new pre-built image [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T442893) [15:29:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [15:29:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [15:30:26] (03open) 10dcaro: node22: add new pre-built image [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/25 (https://phabricator.wikimedia.org/T442893) [15:30:51] (03CR) 10David Caro: "Built it locally, will build it also in docker-regsitry and test in lima-kilo, but ready to review." [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T442893) (owner: 10David Caro) [15:32:05] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance project-proxy-puppetserver-1 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:32:10] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 10Wikimedia-Developer-Portal, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Create an extended property for UI friendly API module descriptions - https://phabricator.wikimedia.org/T434253#12509404 (10KineticPelagic) 05Open→03... [15:32:47] (03update) 10dcaro: node22: add new pre-built image [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/25 (https://phabricator.wikimedia.org/T442893) [15:32:54] 10Striker, 06tools-platform-team, 10ChangeProp, 10Charts, and 12 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12509410 (10Jdforrester-WMF) [15:34:06] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12509420 (10Jdforrester-WMF) [15:35:10] 10Striker, 06tools-platform-team, 10ChangeProp, 10Charts, and 12 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12509444 (10Jdforrester-WMF) [15:35:14] 10Toolforge, 06tools-platform-team: [alerts-api] Count the restarts for each pod in ToolFrequentRestarts - https://phabricator.wikimedia.org/T441348#12509449 (10DamianZaremba) The current behaviour is indeed quite annoying, cluebotng alerted for a single component restarting (due to libera.chat being flaky), t... [15:36:29] 06tools-platform-team: [all] first of the month dependency upgrades for 03-11-2026 - https://phabricator.wikimedia.org/T443392#12509461 (10Raymond_Ndibe) [15:39:28] (03merge) 10r4356th: Replace Bengali numerals in width values [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/46 (https://phabricator.wikimedia.org/T421530) [15:40:03] 10Tool-delintbot, 13Patch-For-Review: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12509489 (10Redmin) [15:42:54] (03open) 10atitkov: Draft: Embed the WikiRun web game in the native apps [toolforge-repos/wikirun-game] - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/4 (https://phabricator.wikimedia.org/T442998) [15:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:46:03] 10Tool-wmf-openapi-linter, 10Wikidata, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 06Wikidata-Omega (Triaging): Unable to upgrade Wikibase to api-testing 1.8.0 - `fs.readFileSync is not a function` - https://phabricator.wikimedia.org/T438947#12509567 (10HCoplin-WMF) 05Open→03Resolv... [15:50:01] 10Striker, 06tools-platform-team, 10ChangeProp, 10Charts, and 12 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12509600 (10HCoplin-WMF) [15:55:35] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12509647 (10JSengupta-WMF) Is it cropping a video or shortening a video? The name will be based on that. If it's former call it "Crop video" and if latter call... [16:02:12] 06tools-platform-team: [all] first of the month dependency upgrades for 03-11-2026 - https://phabricator.wikimedia.org/T443392#12509679 (10Raymond_Ndibe) [16:02:49] 06tools-platform-team: [all] first of the month dependency upgrades for 03-11-2026 - https://phabricator.wikimedia.org/T443392#12509681 (10Raymond_Ndibe) 05In progress→03Resolved a:03Raymond_Ndibe [16:15:34] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12510673 (10Redmin) [16:15:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:16:28] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12510878 (10Doc_James) I am still seeing the videocuttool separate from "Media actions" on Commons {F111902873} I do see Media Actions have moved up, which is... [16:17:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [16:17:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [16:21:19] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12511044 (10Amdrel) > I am still seeing the videocuttool separate from "Media actions" on Commons I //think// that's a pre-existing gadget. I have my own versi... [16:21:29] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12511045 (10Doc_James) Oh and the VideoCutTool is able to BOTH trim and crop and rotate video. As well as remove audio or adjust audio volumn. [16:22:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [16:22:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [16:35:15] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12511156 (10JSengupta-WMF) In that case I'd go with Edit Video [16:39:32] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12511197 (10mikez-WMF) Sounds good! Let's move any discussion about reorganizing into the spinoff task since that also might involve regrouping: https://phabric... [16:40:01] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12511200 (10Doc_James) Great. Added it here to my Common.js {F111904805} And now it appears here as it should {F111904861} And I have turned off VideoCutToo... [16:44:00] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project lingua-data-s4 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/372 (https://phabricator.wikimedia.org/T440447) [16:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:46:06] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12511256 (10Doc_James) Right now under preferences -> Gadgets we have both CropTool and VideoCutTool {F111905478} When we add "Media actions" I am not sure if... [16:46:46] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project lingua-data-s3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/373 (https://phabricator.wikimedia.org/T440447) [16:51:25] !log volans@cloudcumin1001 lingua-data-s3 START - Cookbook wmcs.vps.create_project for s3-only project lingua-data-s3 in eqiad1 (T440447) [16:51:26] volans@cloudcumin1001: Unknown project "lingua-data-s3" [16:51:26] T440447: Request creation of VPS project : LinguaData - https://phabricator.wikimedia.org/T440447 [16:52:03] !log volans@cloudcumin1001 lingua-data-s3 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for s3-only project lingua-data-s3 in eqiad1 (T440447) [16:52:03] volans@cloudcumin1001: Unknown project "lingua-data-s3" [16:52:34] 10Cloud-VPS (Project-requests), 10Language-Diversity-Hub, 13Patch-For-Review: Request creation of VPS project : LinguaData - https://phabricator.wikimedia.org/T440447#12511320 (10Volans) Based on the discussion on the related task T440689, I'm creating the project as "object storage only" and will add quota... [16:53:31] !log volans@cloudcumin1001 lingua-data-s3 START - Cookbook wmcs.vps.create_project for s3-only project lingua-data-s3 in eqiad1 (T440447) [16:53:31] volans@cloudcumin1001: Unknown project "lingua-data-s3" [16:53:47] !log volans@cloudcumin1001 lingua-data-s3 END (ERROR) - Cookbook wmcs.vps.create_project (exit_code=97) for s3-only project lingua-data-s3 in eqiad1 (T440447) [16:53:47] volans@cloudcumin1001: Unknown project "lingua-data-s3" [16:54:14] (03update) 10volans: projects: added project lingua-data-s4 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/372 (https://phabricator.wikimedia.org/T440447) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [16:54:15] (03close) 10volans: projects: added project lingua-data-s4 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/372 (https://phabricator.wikimedia.org/T440447) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [16:54:31] (03update) 10volans: projects: added project lingua-data-s3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/373 (https://phabricator.wikimedia.org/T440447) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [16:54:32] (03close) 10volans: projects: added project lingua-data-s3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/373 (https://phabricator.wikimedia.org/T440447) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [16:55:46] !log volans@cloudcumin1001 lingua-data-s3 START - Cookbook wmcs.vps.create_project for s3-only project lingua-data-s3 in eqiad1 (T440447) [16:55:46] volans@cloudcumin1001: Unknown project "lingua-data-s3" [16:56:33] (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project lingua-data-s3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/374 (https://phabricator.wikimedia.org/T440447) [16:56:33] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project lingua-data-s3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/374 (https://phabricator.wikimedia.org/T440447) [16:58:36] (03merge) 10volans: projects: added project lingua-data-s3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/374 (https://phabricator.wikimedia.org/T440447) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [16:59:34] !log volans@cloudcumin1001 lingua-data-s3 END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for s3-only project lingua-data-s3 in eqiad1 (T440447) [16:59:35] volans@cloudcumin1001: Unknown project "lingua-data-s3" [16:59:36] T440447: Request creation of VPS project : LinguaData - https://phabricator.wikimedia.org/T440447 [17:08:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:10:10] 10Tool-delintbot: Edit across all namespaces in one run based on a sensible list - https://phabricator.wikimedia.org/T415890#12511502 (10Redmin) [17:10:31] 10Tool-delintbot: Edit across all namespaces in one run based on a sensible list - https://phabricator.wikimedia.org/T415890#12511504 (10Redmin) [17:20:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [17:28:39] 10Tool-athar: Impliment the Export function - https://phabricator.wikimedia.org/T443486 (10Bonnjalal) 03NEW [17:32:41] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: 2026-10-21 Toolforge and Cloud VPS taken offline for "PHP2shell" security response - https://phabricator.wikimedia.org/T441908#12511700 (10bd808) [17:32:54] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12511698 (10Sfaci) >>! In T437512#12509352, @Jdforrester-WMF wrote: >>>! In T437512#12509267, @Sfaci wrote: >> @Jdforrester-WMF `repos/dat... [17:36:06] 10Toolforge, 06tools-platform-team: [builds-api] returns incorrect parameter values for use_latest_versions & use_deprecated_versions causing [components-api] to not re-use existing builds - https://phabricator.wikimedia.org/T442364#12511765 (10dcaro) p:05Triage→03Medium [17:39:50] 10Toolforge, 06tools-platform-team: [components-api] leaks internal authentication header requirement on bad client request - https://phabricator.wikimedia.org/T443490 (10DamianZaremba) 03NEW [17:40:55] (03update) 10damian: Store build options as annotations [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/169 [17:41:45] 10Cloud-VPS, 06tools-infrastructure-team, 06Test Platform (San Miguel de Allende 31): Test/benchmark virtio multiqueue for integration-cache-proxy - https://phabricator.wikimedia.org/T440418#12511820 (10bd808) [17:44:41] (03CR) 10Volans: [C:03+2] "TEsted with test-cookbook for lingua-data-s3" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1362059 (owner: 10Volans) [17:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [17:47:29] (03Merged) 10jenkins-bot: CloudVPS create_project: add a --s3-only flag [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1362059 (owner: 10Volans) [17:50:10] 10Tool-extjsonuploader: Unbreak extjsonupdater by upgrading its PHP - https://phabricator.wikimedia.org/T443492 (10taavi) 03NEW [17:55:59] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12511939 (10Sfaci) [17:56:20] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41688162297 (https://github.com/cluebotng/component-configs/commits/7f183bdf8d9d62f3092978941b91e04a9a5531c6) [17:56:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [17:56:55] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12511948 (10Sfaci) [18:14:47] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691194793 (https://github.com/cluebotng/component-configs/commits/7c77c0270dd527e186687a6d6eb0aff11d0ad80c) [18:14:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [18:15:16] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691194741 (https://github.com/cluebotng/component-configs/commits/7c77c0270dd527e186687a6d6eb0aff11d0ad80c) [18:15:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [18:15:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [18:16:43] (03update) 10renovatebot: Update Rust crate replica-pools to v0.1.3 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/94 [18:16:55] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/93 [18:16:59] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/93 [18:17:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:17:05] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:17:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/65 [18:18:44] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/40 [18:18:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/40 [18:19:24] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691894105 (https://github.com/cluebotng/component-configs/commits/4d3b69effc2827247ca575cf060dce20483d3103) [18:19:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [18:19:55] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691894047 (https://github.com/cluebotng/component-configs/commits/4d3b69effc2827247ca575cf060dce20483d3103) [18:19:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [18:22:05] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:25:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:26:19] (03CR) 10David Caro: "Tested it in lima-kilo:" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T442893) (owner: 10David Caro) [18:26:51] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693129959 (https://github.com/cluebotng/component-configs/commits/0d19841535066d42c60bc747e02829434852f914) [18:26:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [18:27:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:27:05] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:27:14] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693130037 (https://github.com/cluebotng/component-configs/commits/0d19841535066d42c60bc747e02829434852f914) [18:27:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [18:28:26] 10Cloud-VPS (Project-requests), 10Language-Diversity-Hub: Request creation of VPS project : LinguaData - https://phabricator.wikimedia.org/T440447#12512251 (10Volans) 05Open→03Resolved a:03Volans Created the project `lingua-data-s3` without quota for VMs (`--s3-only` flag) and increased default quot... [18:28:58] 10Toolforge, 06tools-platform-team: [image-config,pre-built] add node22 pre-built image - https://phabricator.wikimedia.org/T443499 (10dcaro) 03NEW [18:29:14] (03PS2) 10David Caro: node22: add new pre-built image [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T443499) [18:29:41] (03update) 10dcaro: node22: add new pre-built image [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/25 (https://phabricator.wikimedia.org/T442893 https://phabricator.wikimedia.org/T443499) [18:30:04] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693580691 (https://github.com/cluebotng/component-configs/commits/9ff9bae00f7241eef432f91586a0cc47e459c4e7) [18:30:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [18:30:20] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [image-config,pre-built] add node22 pre-built image - https://phabricator.wikimedia.org/T443499#12512282 (10dcaro) p:05Triage→03Medium [18:30:22] 10Tool-lingua-datasets, 10Toolforge (Quota-requests), 06tools-platform-team: Setup Lingua-data.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12512283 (10Volans) 05Open→03Resolved a:03Volans See T440447#12512251 for the object storage part. Tentatively resolving this task too... [18:32:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:32:05] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693580770 (https://github.com/cluebotng/component-configs/commits/9ff9bae00f7241eef432f91586a0cc47e459c4e7) [18:32:05] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:32:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:32:35] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693690557 (https://github.com/cluebotng/component-configs/commits/2350c7e21a6306b68d1249b6635e7820e90f8d65) [18:32:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [18:37:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:37:05] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:38:05] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41694203768 (https://github.com/cluebotng/component-configs/commits/50df9beed114de84489c4353abb75acc5cc118bd) [18:38:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [18:45:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [18:55:33] 10Tool-extjsonuploader: Unbreak extjsonuploader by upgrading its PHP - https://phabricator.wikimedia.org/T443492#12512417 (10taavi) [18:56:50] 10Tool-wsindex: Books API sends hundreds of database queries for one page of books - https://phabricator.wikimedia.org/T443443#12512455 (10RCodes) @Bodhisattwa @Saiphani02 @Muguro Could this go under the Outreachy 33 task (T436628)? [19:15:11] (03update) 10renovatebot: Update Rust crate replica-pools to v0.1.3 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/94 [19:15:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/93 [19:15:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/65 [19:15:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:16:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/40 [19:17:12] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:21:05] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [19:22:12] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:26:05] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [19:31:05] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [19:36:05] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [19:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:48:36] 10Tool-community-activity-alerts-system: Add cross-project activity comparison - https://phabricator.wikimedia.org/T443509 (10Swayam_Agrahari) 03NEW [19:51:07] 10Tool-community-activity-alerts-system: Provide research-ready API access to activity data - https://phabricator.wikimedia.org/T443511 (10Swayam_Agrahari) 03NEW [19:55:52] (03PS1) 10Southparkfan: write_config: Add deployment-prep repositories [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1362171 (https://phabricator.wikimedia.org/T401839) [20:04:50] (03open) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Düzenleme özetinde çıkarılan kategoriyi çizgisiz yaz [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/10 [20:20:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [20:21:03] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#12512835 (10ForzaGreen) Thank you very much @Dev and @KCVelaga for volunteering to maintain Superset. That's a great job ! Concerning the outage related to php vulnerability, indeed Super... [20:24:36] (03PS1) 10Lokal Profil: api: Rename Language to RegionNames [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362179 [20:32:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [20:32:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:37:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [20:37:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [20:48:00] (03approved) 10raymond-ndibe: gen: update toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/204 (https://phabricator.wikimedia.org/T362077) [20:48:06] (03merge) 10raymond-ndibe: gen: update toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/204 (https://phabricator.wikimedia.org/T362077) [20:48:11] (03update) 10raymond-ndibe: tests: test the parsing of components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T362077) [20:53:26] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.231-20261103204825-5fcac860 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1497 (https://phabricator.wikimedia.org/T362077) [20:53:26] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.231-20261103204825-5fcac860 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1497 (https://phabricator.wikimedia.org/T362077) [21:08:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:18:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [21:18:05] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [21:20:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [21:23:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [21:23:05] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [21:27:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [21:27:05] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [21:32:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [21:32:05] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [21:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [21:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:55:17] 10Cloud-VPS (Project-requests): Request creation of deorph VPS project - https://phabricator.wikimedia.org/T442348#12513240 (10Bancherx) Hi Volans, Yes, we should be able to reduce memory usage through refactoring, though we still need to verify whether it fits below 8 GB and measure the performance impact. [22:13:07] (03PS1) 10Lokal Profil: maintenance: Share the script bootstrap [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362210 [22:13:07] (03PS1) 10Lokal Profil: Refresh the countries cache after each harvest [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362211 [22:25:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [22:26:26] 10Cloud-Services, 10WM-Bot: wm-bot.wmflabs.org now incorrectly redirects to wm-bot.wmcloud.org - https://phabricator.wikimedia.org/T443531 (10Addshore) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and... [22:27:26] 10Cloud-Services, 10WM-Bot: wm-bot.wmflabs.org now incorrectly redirects to wm-bot.wmcloud.org - https://phabricator.wikimedia.org/T443531#12513394 (10Addshore) [22:28:06] 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrades for November 2026 - https://phabricator.wikimedia.org/T443532 (10gitlab-toolforge-bot) 03NEW [22:28:33] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/7 (https://phabricator.wikimedia.org/T443532) [22:28:42] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/15 (https://phabricator.wikimedia.org/T443532) [22:28:45] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/15 (https://phabricator.wikimedia.org/T443532) [22:28:56] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/21 (https://phabricator.wikimedia.org/T443532) [22:28:59] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/21 (https://phabricator.wikimedia.org/T443532) [22:29:05] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/52 (https://phabricator.wikimedia.org/T443532) [22:29:13] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/52 (https://phabricator.wikimedia.org/T443532) [22:29:20] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/105 (https://phabricator.wikimedia.org/T443532) [22:29:26] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/211 (https://phabricator.wikimedia.org/T443532) [22:29:30] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/105 (https://phabricator.wikimedia.org/T443532) [22:29:38] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/211 (https://phabricator.wikimedia.org/T443532) [22:29:39] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/66 (https://phabricator.wikimedia.org/T443532) [22:29:49] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/157 (https://phabricator.wikimedia.org/T443532) [22:29:54] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/66 (https://phabricator.wikimedia.org/T443532) [22:30:03] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/157 (https://phabricator.wikimedia.org/T443532) [22:30:10] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/386 (https://phabricator.wikimedia.org/T443532) [22:30:13] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/124 (https://phabricator.wikimedia.org/T443532) [22:30:30] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/386 (https://phabricator.wikimedia.org/T443532) [22:30:35] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/124 (https://phabricator.wikimedia.org/T443532) [22:30:39] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/113 (https://phabricator.wikimedia.org/T443532) [22:30:44] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/80 (https://phabricator.wikimedia.org/T443532) [22:30:51] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/82 (https://phabricator.wikimedia.org/T443532) [22:30:53] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/113 (https://phabricator.wikimedia.org/T443532) [22:30:56] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/80 (https://phabricator.wikimedia.org/T443532) [22:31:02] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/82 (https://phabricator.wikimedia.org/T443532) [22:31:05] 10Cloud-VPS, 06tools-infrastructure-team, 10WM-Bot: wm-bot.wmflabs.org now incorrectly redirects to wm-bot.wmcloud.org - https://phabricator.wikimedia.org/T443531#12513420 (10Addshore) [22:37:14] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Improving toolforge NFS patrol - https://phabricator.wikimedia.org/T439494#12513437 (10Volans) Today the alert fired again, I run the cleanup in https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull that freed a... [22:45:01] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [22:45:06] FIRING: PuppetAgentNoResources: No Puppet resources found on instance extdist-07 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [22:46:05] FIRING: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:47:05] FIRING: PuppetAgentNoResources: No Puppet resources found on instance paws-puppetserver-1 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:49:05] FIRING: PuppetAgentNoResources: No Puppet resources found on instance scratch-2 on project cloudinfra-nfs - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:50:05] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance meta-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:51:39] 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrades for November 2026 - https://phabricator.wikimedia.org/T443532#12513503 (10Raymond_Ndibe) 05Open→03In progress [22:51:51] 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrades for November 2026 - https://phabricator.wikimedia.org/T443532#12513504 (10Raymond_Ndibe) a:03Raymond_Ndibe [22:53:05] FIRING: PuppetAgentNoResources: No Puppet resources found on instance k8s-nfs on project quarry - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:55:05] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:57:33] (03PS1) 10Lokal Profil: tests: Mock Intuition rather than hand-writing a stub [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362224 [22:58:05] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cvn-apache12 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:00:31] (03PS1) 10Lokal Profil: api: Let FormatHtmllist use the shared output loop [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362226 [23:07:30] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-docker-cxserver02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436467#12513563 (10Southparkfan) 05Open→03Resolved a:03Southparkfan [23:07:31] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-docker-citoid02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436466#12513566 (10Southparkfan) 05Open→03Resolved a:03Southparkfan [23:08:05] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache12 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:10:01] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [23:10:05] FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-puppetserver-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:10:05] FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:12:05] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:12:06] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [23:13:05] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache12 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:13:05] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance k8s-nfs on project quarry - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:16:25] (03update) 10renovatebot: Update Rust crate replica-pools to v0.1.3 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/94 [23:16:29] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/93 [23:16:54] 10Tool-athar: Impliment the Export function - https://phabricator.wikimedia.org/T443486#12513598 (10Bonnjalal) [23:17:06] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [23:17:09] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/65 [23:18:14] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/40 [23:25:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [23:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [23:50:01] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse