[00:06:22] FIRING: [2x] PrometheusK8sCertExpirySoon: Prometheus k8s certificate is about to expire - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusK8sCertExpirySoon [00:37:34] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [00:48:33] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [01:13:22] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/22 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:13:25] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/22 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:13:35] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/22 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:13:38] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/48 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:13:42] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/48 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:13:52] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/48 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:14:00] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/167 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:14:03] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/167 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:14:16] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/167 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:14:16] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/295 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:14:20] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/295 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:14:29] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/295 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:15:18] (03update) 10raymond-ndibe: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) [01:16:38] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.22-20260505011402-7e549897 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1238 [01:16:45] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.83-20260505011420-664eb496 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1239 [01:16:46] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.83-20260505011420-664eb496 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1239 [01:18:33] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.195-20260505011444-db9e6885 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1240 [01:19:40] (03update) 10raymond-ndibe: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) [01:19:41] (03approved) 10raymond-ndibe: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) [01:20:14] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.487-20260505011456-7aef964e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1241 [01:20:15] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.487-20260505011456-7aef964e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1241 [01:20:22] (03merge) 10raymond-ndibe: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) [01:20:23] (03update) 10raymond-ndibe: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) [01:21:19] (03update) 10raymond-ndibe: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) [01:23:27] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.488-20260505012033-d89f1826 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1241 (https://phabricator.wikimedia.org/T415322) [01:23:40] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.488-20260505012033-d89f1826 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1241 (https://phabricator.wikimedia.org/T415322) [01:25:51] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [01:35:10] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [01:35:22] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [01:38:12] (03update) 10raymond-ndibe: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) [01:45:31] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [01:46:31] 06cloud-services-team, 10Data-Services: imagelinks table missing in pplwiki, several other tables missing in testcommonswiki - https://phabricator.wikimedia.org/T425384 (10AKhatun_WMF) 03NEW [01:48:13] (03update) 10raymond-ndibe: logs-api: bump to 0.0.22-20260505011402-7e549897 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1238 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:48:14] (03approved) 10raymond-ndibe: logs-api: bump to 0.0.22-20260505011402-7e549897 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1238 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:48:18] (03merge) 10raymond-ndibe: logs-api: bump to 0.0.22-20260505011402-7e549897 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1238 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:48:46] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [01:48:59] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.83-20260505011420-664eb496 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1239 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:49:12] !log raymond-ndibe@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-emailer [01:49:33] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [01:53:32] (03update) 10raymond-ndibe: images.py: add tests for image variant matching [repos/cloud/toolforge/jobs-api] (refactor_image_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/286 (https://phabricator.wikimedia.org/T415322) [02:03:23] (03close) 10raymond-ndibe: [jobs-api] validate command string [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T356267) [02:03:57] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [02:06:30] (03close) 10raymond-ndibe: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [02:08:14] (03close) 10raymond-ndibe: [envvars-cli] print error string and not dict [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/81 (https://phabricator.wikimedia.org/T360147) [02:09:40] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [02:14:12] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [02:24:09] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [02:29:11] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [02:34:19] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.83-20260505011420-664eb496 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1239 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:34:21] (03approved) 10raymond-ndibe: jobs-emailer: bump to 0.0.83-20260505011420-664eb496 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1239 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:34:31] (03merge) 10raymond-ndibe: jobs-emailer: bump to 0.0.83-20260505011420-664eb496 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1239 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:35:30] (03update) 10raymond-ndibe: components-api: bump to 0.0.195-20260505011444-db9e6885 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1240 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:35:56] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [02:52:39] 10Cloud-VPS (Quota-requests): Quota increase for development metrics - https://phabricator.wikimedia.org/T425387 (10thcipriani) 03NEW [02:54:11] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [02:56:06] (03update) 10raymond-ndibe: components-api: bump to 0.0.195-20260505011444-db9e6885 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1240 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:56:15] (03approved) 10raymond-ndibe: components-api: bump to 0.0.195-20260505011444-db9e6885 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1240 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:56:34] (03merge) 10raymond-ndibe: components-api: bump to 0.0.195-20260505011444-db9e6885 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1240 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:57:15] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [03:09:11] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [03:11:02] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [03:23:13] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [03:25:25] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.488-20260505012033-d89f1826 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1241 (https://phabricator.wikimedia.org/T415322) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:25:27] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.488-20260505012033-d89f1826 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1241 (https://phabricator.wikimedia.org/T415322) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:25:28] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.488-20260505012033-d89f1826 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1241 (https://phabricator.wikimedia.org/T415322) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:25:59] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.488-20260505012033-d89f1826 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1241 (https://phabricator.wikimedia.org/T415322) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:26:13] (03update) 10raymond-ndibe: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) [03:26:19] (03update) 10raymond-ndibe: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) [03:26:20] (03approved) 10raymond-ndibe: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) [03:26:49] (03merge) 10raymond-ndibe: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) [03:26:50] (03update) 10raymond-ndibe: images.py: add tests for image variant matching [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/286 (https://phabricator.wikimedia.org/T415322) [03:29:23] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.489-20260505032700-9f8ba59e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1242 (https://phabricator.wikimedia.org/T415322) [03:29:30] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.489-20260505032700-9f8ba59e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1242 (https://phabricator.wikimedia.org/T415322) [03:35:13] (03update) 10raymond-ndibe: istio-system: Disable X-Envoy-Peer-Metadata headers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) (owner: 10taavi) [03:36:41] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [03:48:11] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [03:50:16] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [04:03:12] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [04:14:25] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.489-20260505032700-9f8ba59e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1242 (https://phabricator.wikimedia.org/T415322) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [04:14:27] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.489-20260505032700-9f8ba59e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1242 (https://phabricator.wikimedia.org/T415322) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [04:14:32] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.489-20260505032700-9f8ba59e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1242 (https://phabricator.wikimedia.org/T415322) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [04:15:00] (03update) 10raymond-ndibe: images.py: add tests for image variant matching [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/286 (https://phabricator.wikimedia.org/T415322) [04:15:05] (03update) 10raymond-ndibe: images.py: add tests for image variant matching [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/286 (https://phabricator.wikimedia.org/T415322) [04:15:14] (03update) 10raymond-ndibe: images.py: add tests for image variant matching [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/286 (https://phabricator.wikimedia.org/T415322) [04:15:16] (03approved) 10raymond-ndibe: images.py: add tests for image variant matching [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/286 (https://phabricator.wikimedia.org/T415322) [04:17:15] (03merge) 10raymond-ndibe: images.py: add tests for image variant matching [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/286 (https://phabricator.wikimedia.org/T415322) [04:17:18] (03update) 10raymond-ndibe: replace job images with web images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/263 (https://phabricator.wikimedia.org/T415322) [04:19:57] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.490-20260505041730-92b6f702 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1243 (https://phabricator.wikimedia.org/T415322) [04:20:06] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.490-20260505041730-92b6f702 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1243 (https://phabricator.wikimedia.org/T415322) [04:20:38] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [04:31:11] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [04:35:29] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [04:49:15] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [05:18:10] (03update) 10countcount: Highlight inherited node prefixes [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/1 [05:19:53] (03merge) 10countcount: Highlight inherited node prefixes [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/1 [05:33:00] (03open) 10countcount: Set minimum prefix lengths to MediaWiki block limits (IPv4 /16, IPv6 /19) [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/8 [05:42:11] (03update) 10countcount: Set minimum prefix lengths to MediaWiki block limits (IPv4 /16, IPv6 /19) [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/8 [05:42:18] (03update) 10countcount: Set minimum prefix lengths to MediaWiki block limits (IPv4 /16, IPv6 /19) [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/8 [05:45:15] (03update) 10countcount: Set minimum prefix lengths to MediaWiki block limits (IPv4 /16, IPv6 /19) [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/8 [05:45:44] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.490-20260505041730-92b6f702 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1243 (https://phabricator.wikimedia.org/T415322) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [05:45:46] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.490-20260505041730-92b6f702 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1243 (https://phabricator.wikimedia.org/T415322) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [05:45:53] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.490-20260505041730-92b6f702 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1243 (https://phabricator.wikimedia.org/T415322) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [05:48:55] (03open) 10countcount: Add documentation consistency guidelines to AGENTS.md [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/9 [05:49:16] (03merge) 10countcount: Add documentation consistency guidelines to AGENTS.md [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/9 [05:49:52] (03update) 10countcount: Set minimum prefix lengths to MediaWiki block limits (IPv4 /16, IPv6 /19) [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/8 [05:50:42] (03merge) 10countcount: Set minimum prefix lengths to MediaWiki block limits (IPv4 /16, IPv6 /19) [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/8 [08:27:56] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#11888585 (10fgiunchedi) [08:47:39] (03open) 10countcount: Sync README with actual behavior [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/10 [08:48:00] (03merge) 10countcount: Sync README with actual behavior [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/10 [09:02:36] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Set external url for prometheus/thanos metricsinfra web interface - https://phabricator.wikimedia.org/T424312#11888744 (10taavi) 05Open→03Resolved a:03taavi [09:38:39] (03open) 10kineticpelagic: feat(linter): Finalize OpenAPI description validation mechanism for structured objects (T422600) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/19 [09:42:53] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work, 13Patch-For-Review: Add the remaining linting rules - https://phabricator.wikimedia.org/T422600#11888873 (10KineticPelagic) 05In progress→03Resolved @AGhirelli-WMF , feel... [10:22:41] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:25:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:58:43] 10Tool-wikicordo: Self-nomination filter for deletion requests in Wikicordo - https://phabricator.wikimedia.org/T425361#11889202 (10Ladsgroup) I implemented the reporting part so the data will be there once I deploy the new version but I still need to implement the GUI filtering too. While I was there, I made an... [12:21:56] FIRING: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:25:23] 06cloud-services-team: Alert on openstack resources close to running out - https://phabricator.wikimedia.org/T425412 (10fgiunchedi) 03NEW [12:26:52] 06cloud-services-team, 10Cloud-VPS: Alert on openstack resources close to running out - https://phabricator.wikimedia.org/T425412#11889253 (10taavi) [12:29:56] 10tool-wscontest, 03Wikimedia-Hackathon-2026: Enhance Wikisource Contest Tool into a scalable, real-time, and user-centric campaign platform - https://phabricator.wikimedia.org/T425084#11889257 (10Aklapper) [12:41:10] 10Tool-itwiki, 06Commons, 03Wikimedia-Hackathon-2026: Auto-calculate the bearing (compass) from Wikimedia Commons' photos from two pairs of coordinates (Location + Object location) - https://phabricator.wikimedia.org/T363052#11889294 (10valerio.bozzolan) OK absolutely nothing was done for this during wmhack... [12:52:44] 06cloud-services-team, 06Trust-and-Safety: Account recovery help needed for Developer account akanji-wmf - https://phabricator.wikimedia.org/T425213#11889330 (10Aklapper) 05Open→03Invalid Closing per last comment. [13:02:28] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002#11889383 (10KBach) [13:02:50] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)): Add tests to ensure consistency between OAD example and OpenAPI linter - https://phabricator.wikimedia.org/T419576#11889384 (10KBach) [13:17:47] 10Cloud-VPS (Quota-requests), 06Content-Transform-Team: Increase quota for wikitextexp project - https://phabricator.wikimedia.org/T425420 (10Jgiannelos) 03NEW [13:18:26] 10Cloud-VPS (Quota-requests), 06Content-Transform-Team: Increase quota for wikitextexp project - https://phabricator.wikimedia.org/T425420#11889507 (10Jgiannelos) [13:18:38] 10Cloud-VPS (Quota-requests), 06Content-Transform-Team: Increase quota for wikitextexp project - https://phabricator.wikimedia.org/T425420#11889508 (10Jgiannelos) [13:41:19] 10Cloud-VPS (Project-requests): Request creation of thelounge Cloud VPS project - https://phabricator.wikimedia.org/T424443#11889634 (10taavi) +1, with the condition that all users of this service must be individually identifiable by the operators of the IRC networks they are connecting to. In practice this mean... [14:04:14] 10Cloud-VPS (Project-requests): Request creation of thelounge Cloud VPS project - https://phabricator.wikimedia.org/T424443#11889745 (10fnegri) a:03fnegri [14:12:01] 10Cloud-VPS (Project-requests): Request creation of thelounge Cloud VPS project - https://phabricator.wikimedia.org/T424443#11889767 (10Jony) Hi @taavi, Thank you for the approval! I completely understand the conditions regarding IP reputation and user identification. Once the instance is provisioned, I will:... [14:16:56] FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:20:09] 10Toolforge, 06tools-platform-team: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#11889806 (10fnegri) a:05dcaro→03None [14:37:43] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 07OKR-Work, 13Patch-For-Review: Add the remaining linting rules - https://phabricator.wikimedia.org/T422600#11889895 (10KineticPelagic) [14:58:58] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 07OKR-Work: [SPIKE] Add security scheme information to MediaWiki REST API description - https://phabricator.wikimedia.org/T423552#11889981 (10OWresch-WMF) [15:02:17] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002#11890023 (10BPirkle) [15:18:44] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 05MW-1.46-notes (1.46.0-wmf.26; 2026-04-28), 07OKR-Work: OpenAPI linting: Add missing OpenAPI spec elements to Response Components - https://phabricator.wikimedia.org/T422739#11890134 (10BPirkle) [15:20:29] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 07Essential-Work: Linter: Split rules into more logical groupings - https://phabricator.wikimedia.org/T415914#11890142 (10BPirkle) [15:21:05] 10Tool-wmf-openapi-linter, 07OKR-Work: Fix linter issues discovered during implementation of the OAD example - https://phabricator.wikimedia.org/T414974#11890146 (10BPirkle) [15:22:16] 10Tool-wmf-openapi-linter, 07OKR-Work: Improve linting - unify example validation - https://phabricator.wikimedia.org/T423411#11890149 (10BPirkle) [15:52:48] 10Tool-refill: Automatic linking of work/publisher - https://phabricator.wikimedia.org/T352394#11890347 (10Pppery) [15:59:46] (03update) 10aghirelli: feat(linter): Finalize OpenAPI description validation mechanism for structured objects (T422600) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/19 (owner: 10kineticpelagic) [15:59:50] (03update) 10aghirelli: feat(linter): Finalize OpenAPI description validation mechanism for structured objects (T422600) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/19 (owner: 10kineticpelagic) [16:21:56] FIRING: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:43:34] 10Tool-itwiki, 06Commons: Auto-calculate the bearing (compass) from Wikimedia Commons' photos from two pairs of coordinates (Location + Object location) - https://phabricator.wikimedia.org/T363052#11890583 (10Izno) [17:21:51] (03CR) 10MusikAnimal: [C:03+2] Use tabs not spaces in twig files and SQL queries [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276362 (owner: 10Alien4444) [17:22:53] (03Merged) 10jenkins-bot: Use tabs not spaces in twig files and SQL queries [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276362 (owner: 10Alien4444) [17:29:21] 10Cloud-VPS (Project-requests), 10WISE: Request creation of wise VPS project - https://phabricator.wikimedia.org/T424068#11890907 (10Gopavasanth) [17:34:23] 06cloud-services-team, 10Data-Services, 06Data-Persistence: imagelinks table missing in pplwiki, several other tables missing in testcommonswiki - https://phabricator.wikimedia.org/T425384#11890961 (10AKhatun_WMF) [17:35:07] (03PS1) 10MusikAnimal: 3.24.4 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283057 [17:36:02] (03update) 10kineticpelagic: feat(linter): Finalize OpenAPI description validation mechanism for structured objects (T422600) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/19 [17:36:21] (03CR) 10MusikAnimal: [C:03+2] 3.24.4 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283057 (owner: 10MusikAnimal) [17:36:58] (03Merged) 10jenkins-bot: 3.24.4 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283057 (owner: 10MusikAnimal) [17:39:10] 10Cloud-VPS (Quota-requests), 06Content-Transform-Team: Increase quota for wikitextexp project - https://phabricator.wikimedia.org/T425420#11891013 (10Volans) +1 for me [17:39:11] (03open) 10sweil: Add Tesseract script models to model list [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/6 [17:41:02] 10Cloud-VPS (Quota-requests): Quota increase for development metrics - https://phabricator.wikimedia.org/T425387#11891038 (10Volans) +1 for me [17:42:03] (03update) 10kineticpelagic: feat(linter): Finalize OpenAPI description validation mechanism for structured objects (T422600) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/19 [17:45:45] (03approved) 10aghirelli: feat(linter): Finalize OpenAPI description validation mechanism for structured objects (T422600) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/19 (owner: 10kineticpelagic) [17:46:01] (03merge) 10aghirelli: feat(linter): Finalize OpenAPI description validation mechanism for structured objects (T422600) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/19 (owner: 10kineticpelagic) [17:46:49] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 07OKR-Work, 13Patch-For-Review: Add the remaining linting rules - https://phabricator.wikimedia.org/T422600#11891104 (10AGhirelli-WMF) 05In progress→03Resolved [17:47:05] 06cloud-services-team, 10Data-Services, 06Data-Persistence: imagelinks table missing in pplwiki, several other tables missing in testcommonswiki - https://phabricator.wikimedia.org/T425384#11891115 (10Ladsgroup) testcommonswiki is intentional (T421914) commonswiki will follow soon. pplwiki I'm not sure. @Zab... [17:50:40] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 07OKR-Work, 13Patch-For-Review: Add the remaining linting rules - https://phabricator.wikimedia.org/T422600#11891147 (10KineticPelagic) [18:07:56] (03merge) 10samwilson: Add Tesseract script models to model list [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/6 (owner: 10sweil) [18:16:56] FIRING: [2x] SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:56:21] (03update) 10sweil: Add new OCR parameter to normalize the result text [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/3 [19:07:53] (03open) 10sweil: Update GitLab CI from Debian bookworm to Debian trixie [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/7 [19:36:47] (03PS1) 10CDanis: Add a maintenance script cron secret [labs/private] - 10https://gerrit.wikimedia.org/r/1283085 [19:37:25] 10Tool-inteGraality: Stream live progress during web updates - https://phabricator.wikimedia.org/T425477 (10JeanFred) 03NEW [19:37:32] (03CR) 10CDanis: [V:03+2 C:03+2] Add a maintenance script cron secret [labs/private] - 10https://gerrit.wikimedia.org/r/1283085 (owner: 10CDanis) [20:08:57] 10Toolforge, 06tools-platform-team: fix bugs in istio-gateway deployment - https://phabricator.wikimedia.org/T425479 (10Raymond_Ndibe) 03NEW [20:13:54] (03update) 10raymond-ndibe: Draft: fix istio issue [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1244 (https://phabricator.wikimedia.org/T425479) [20:14:32] (03update) 10raymond-ndibe: fix istio issue [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1244 (https://phabricator.wikimedia.org/T425479) [20:15:31] (03update) 10raymond-ndibe: [istio-gateway, utils]: fix istio bugs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1244 (https://phabricator.wikimedia.org/T425479) [20:16:56] RESOLVED: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:16:56] FIRING: [2x] SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:17:40] 10Toolforge, 06tools-platform-team: Remove all reference to ingress-nginx in lima-kilo - https://phabricator.wikimedia.org/T425480 (10Raymond_Ndibe) 03NEW [20:20:54] 10Toolforge, 06tools-platform-team: remove ingress-nginx deployment - https://phabricator.wikimedia.org/T425480#11891816 (10Raymond_Ndibe) [20:21:42] (03open) 10raymond-ndibe: main.yaml: remove the deployment of ingress-nginx [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/321 (https://phabricator.wikimedia.org/T425480) [20:25:49] (03update) 10raymond-ndibe: main.yaml: remove the deployment of ingress-nginx [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/321 (https://phabricator.wikimedia.org/T425480) [20:26:05] (03update) 10raymond-ndibe: main.yaml, kind.yaml: remove the deployment of ingress-nginx [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/321 (https://phabricator.wikimedia.org/T425480) [20:28:14] 10Tool-etherpad-backup, 03Wikimedia-Hackathon-2026: Setup proof of concept storage and retrieval from WMCS object storage - https://phabricator.wikimedia.org/T422958#11891851 (10bd808) 05In progress→03Resolved [20:43:13] (03update) 10danyya: Draft: Home page [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/33 [20:53:13] 10Tool-humaniki-2: Link to an FAQ - https://phabricator.wikimedia.org/T425482 (10Danya) 03NEW [20:54:29] 10Tool-humaniki-2: Link to report a bug - https://phabricator.wikimedia.org/T425483 (10Danya) 03NEW [20:54:35] 10Tool-humaniki-2: Link to suggest a new feature - https://phabricator.wikimedia.org/T425484 (10Danya) 03NEW [20:57:48] 10Tool-humaniki-2: Link to the Phabricator board - https://phabricator.wikimedia.org/T425485 (10Danya) 03NEW [20:59:17] 10Tool-humaniki-2: Aggregate gender data in the database instead of client - https://phabricator.wikimedia.org/T425219#11891971 (10Danya) [20:59:26] 10Tool-humaniki-2: Use vector tiles in map - https://phabricator.wikimedia.org/T425288#11891974 (10Danya) p:05Medium→03Low [20:59:33] 10Tool-humaniki-2: Link to the Phabricator board - https://phabricator.wikimedia.org/T425485#11891975 (10Danya) [20:59:38] 10Tool-humaniki-2: Link to suggest a new feature - https://phabricator.wikimedia.org/T425484#11891976 (10Danya) [20:59:45] 10Tool-humaniki-2: Link to report a bug - https://phabricator.wikimedia.org/T425483#11891977 (10Danya) [20:59:49] 10Tool-humaniki-2: Link to an FAQ - https://phabricator.wikimedia.org/T425482#11891978 (10Danya) [20:59:54] 10Tool-humaniki-2: Link to the Phabricator board - https://phabricator.wikimedia.org/T425485#11891980 (10Danya) p:05Triage→03Low [20:59:59] 10Tool-humaniki-2: Link to suggest a new feature - https://phabricator.wikimedia.org/T425484#11891982 (10Danya) p:05Triage→03Low [21:00:03] 10Tool-humaniki-2: Link to report a bug - https://phabricator.wikimedia.org/T425483#11891984 (10Danya) p:05Triage→03Low [21:00:06] 10Tool-humaniki-2: Link to an FAQ - https://phabricator.wikimedia.org/T425482#11891987 (10Danya) p:05Triage→03Low [21:02:24] 10Tool-humaniki-2: Use dumps? - https://phabricator.wikimedia.org/T423012#11891990 (10Danya) [[https://www.mediawiki.org/wiki/Wikidata_Toolkit|Wikidata Toolkit]] may be useful here. This is what Humaniki [[https://www.mediawiki.org/wiki/Humaniki/FAQ#Why_humaniki_doesn’t_reflect_editing_I_did_yesterday?|was using]]. [21:04:23] 10Tool-humaniki-2: Create a contribution guide - https://phabricator.wikimedia.org/T425486 (10Danya) 03NEW [21:04:29] 10Tool-humaniki-2: Create a contribution guide - https://phabricator.wikimedia.org/T425486#11892005 (10Danya) [21:04:37] 10Tool-humaniki-2: Create a contribution guide - https://phabricator.wikimedia.org/T425486#11892007 (10Danya) p:05Triage→03Low [21:05:44] (03update) 10raymond-ndibe: istio-system: Disable X-Envoy-Peer-Metadata headers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) (owner: 10taavi) [21:07:10] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [21:10:40] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11892037 (10jrbs) >>! In T423547#11828822, @SD0001 wrote: > *** `./cli --election config/ucoc-2026.yaml gener... [21:14:16] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [21:15:04] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] (fix_istio_issue) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [21:24:23] 10Tool-etherpad-backup, 03Wikimedia-Hackathon-2026: Archive etherpads mentioned in hackathon/techconf tasks - https://phabricator.wikimedia.org/T417207#11892073 (10bd808) Let's see how many of the new batch are actually new: `lang=shell-session I have no name!@shell-1778013300:~$ etherpad-backup ls | cut -f 3... [21:47:21] 10Tool-humaniki-2: Migrate site to Wikimedia Cloud Services? - https://phabricator.wikimedia.org/T425489 (10Danya) 03NEW [21:47:27] 10Tool-humaniki-2: Migrate site to Wikimedia Cloud Services? - https://phabricator.wikimedia.org/T425489#11892123 (10Danya) [22:50:59] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 07OKR-Work: [SPIKE] Add security scheme information to MediaWiki REST API description - https://phabricator.wikimedia.org/T423552#11892281 (10BPirkle) Two ways of doing this come to mind: ====Narrow==== One option i... [23:29:44] 10Tool-ghanasupremecases1: ghanasupremecases.toolforge.org violates Privacy Policy - https://phabricator.wikimedia.org/T425079#11892341 (10SincerelySaeed5000) 05Open→03Resolved The issue has been fixed. I removed the external Cloudflare Font Awesome CDN reference and replaced it with the Toolforge-hoste... [23:55:56] 10Tool-upload-workbench: Status indicator dot turns green before required fields are filled - https://phabricator.wikimedia.org/T425496 (10Daanvr) 03NEW p:05Triage→03Medium