[00:00:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.937% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:00:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:10:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:05:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.842% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [02:38:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:43:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:15:25] 06cloud-services-team: Update make-toolforge-user-list.py - https://phabricator.wikimedia.org/T411545#11864062 (10komla) The `make-toolforge-user-list.py` script has been superseded by the new process for the cloud services survey. I'm working on updating the new process on wiki. I will also push the changes ma... [05:16:27] 10Toolforge, 06tools-platform-team: Update make-toolforge-user-list.py - https://phabricator.wikimedia.org/T411545#11864063 (10komla) a:03komla [05:20:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:26:42] 10Toolforge, 06tools-platform-team: Document updated process for cloud services survey - https://phabricator.wikimedia.org/T424609 (10komla) 03NEW [05:27:53] 10Toolforge, 06tools-platform-team: Update make-toolforge-user-list.py - https://phabricator.wikimedia.org/T411545#11864091 (10komla) 05Open→03Resolved I've opened a subtask so I will close this one. [05:35:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:00:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:21:12] 10Tool-centralnotice-banner-editor, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: CentralNotice Banner Editor: What's new and What's next - https://phabricator.wikimedia.org/T424074#11864189 (10Oyelola_Victoria) [06:24:09] (03approved) 10booksmurf: Fix up MINT_URL env config variable [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/10 (https://phabricator.wikimedia.org/T206436) (owner: 10samwilson) [06:25:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:38:03] 10Toolforge, 06tools-platform-team: Document updated process for cloud services survey - https://phabricator.wikimedia.org/T424609#11864202 (10komla) [07:42:29] (03merge) 10samwilson: Fix up MINT_URL env config variable [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/10 (https://phabricator.wikimedia.org/T206436) [08:18:17] !log godog@r5 admin START - Cookbook wmcs.openstack.rack_resources on cluster 'codfw1dev' [08:18:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:18:48] !log godog@r5 admin END (PASS) - Cookbook wmcs.openstack.rack_resources (exit_code=0) on cluster 'codfw1dev' [08:18:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:19:10] !log godog@r5 admin START - Cookbook wmcs.openstack.rack_resources on cluster 'codfw1dev' [08:19:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:19:21] !log godog@r5 admin END (PASS) - Cookbook wmcs.openstack.rack_resources (exit_code=0) on cluster 'codfw1dev' [08:19:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:28:55] !log godog@r5 admin START - Cookbook wmcs.openstack.rack_resources on cluster 'codfw1dev' [08:29:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:29:09] !log godog@r5 admin END (PASS) - Cookbook wmcs.openstack.rack_resources (exit_code=0) on cluster 'codfw1dev' [08:29:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:32:29] !log godog@r5 admin START - Cookbook wmcs.openstack.rack_resources on cluster 'codfw1dev' [08:32:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:32:45] !log godog@r5 admin END (PASS) - Cookbook wmcs.openstack.rack_resources (exit_code=0) on cluster 'codfw1dev' [08:32:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:41:25] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Quota increase for development metrics - https://phabricator.wikimedia.org/T424564#11864491 (10fnegri) +1 [08:42:13] !log godog@r5 admin START - Cookbook wmcs.openstack.rack_resources on cluster 'codfw1dev' [08:42:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:42:46] !log godog@r5 admin END (PASS) - Cookbook wmcs.openstack.rack_resources (exit_code=0) on cluster 'codfw1dev' [08:42:48] 10Cloud-VPS (Project-requests): Request creation of thelounge Cloud VPS project - https://phabricator.wikimedia.org/T424443#11864496 (10taavi) How do you plan to manage user access to the service? [08:42:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:56:04] !log volans@cloudcumin1001 development-metrics START - Cookbook wmcs.openstack.quota_increase by 8 cores, 34816 ram [08:56:12] !log volans@cloudcumin1001 development-metrics END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 8 cores, 34816 ram [08:56:59] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Quota increase for development metrics - https://phabricator.wikimedia.org/T424564#11864537 (10Volans) 05Open→03Resolved a:03Volans +1 executed `cookbook wmcs.openstack.quota_increase --cores 8 --ram 34G --project development-metrics --cluster-name... [09:00:44] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Test Platform, 07OKR-Work: Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#11864545 (10Peter) How long do the test take today to run if you run it on your machine? What about start adding it to core so we can see how it works?... [09:29:43] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/45 (https://phabricator.wikimedia.org/T422184) [09:50:12] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/292 (https://phabricator.wikimedia.org/T422184) [09:52:55] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/91 (https://phabricator.wikimedia.org/T422184) [09:55:46] 10Cloud-VPS (Project-requests): Request creation of thelounge Cloud VPS project - https://phabricator.wikimedia.org/T424443#11864948 (10Jony) Hi @bd808 and @taavi, thanks for reviewing the request. @bd808, You are right, the Cloud VPS web proxy service is perfectly fine. I originally asked for a floating IP and... [10:06:03] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Controlled cloudsw down tests for F4 - https://phabricator.wikimedia.org/T419658#11864978 (10fgiunchedi) I crunched some numbers today to see the resource distribution across racks: ` +------+-------+-----+-----------------+-------+-------------------+--... [10:09:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:24:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:32:27] (03update) 10samwilson: Remove phpunit-bridge [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/9 [10:32:34] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:38:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:42:51] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/45 (https://phabricator.wikimedia.org/T422184) [10:44:11] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:44:25] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:57:13] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:58:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:12:22] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.484-20260421000258-fe5dffd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1223 (https://phabricator.wikimedia.org/T423870) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:12:24] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.484-20260421000258-fe5dffd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1223 (https://phabricator.wikimedia.org/T423870) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:12:39] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.484-20260421000258-fe5dffd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1223 (https://phabricator.wikimedia.org/T423870) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:13:31] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.484-20260421000258-fe5dffd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1223 (https://phabricator.wikimedia.org/T423870) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:16:20] (03open) 10raymond-ndibe: upgrade_blubber_to_v0_24_0 [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/84 (https://phabricator.wikimedia.org/T422184) [11:20:26] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/45 (https://phabricator.wikimedia.org/T422184) [11:22:32] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [11:22:59] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [11:29:37] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/292 (https://phabricator.wikimedia.org/T422184) [11:32:48] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/91 (https://phabricator.wikimedia.org/T422184) [11:42:12] (03open) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/153 (https://phabricator.wikimedia.org/T422184) [12:17:59] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [12:20:06] (03open) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T422184) [12:22:32] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:28:45] (03open) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T422184) [12:33:49] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T422184) [12:35:44] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T422184) [12:53:25] 10VPS-project-Codesearch: Make it possible to use negative regex assertions in Codesearch - https://phabricator.wikimedia.org/T424655 (10A_smart_kitten) 03NEW [12:54:17] 10VPS-project-Codesearch: Make it possible to use negative regex assertions in Codesearch - https://phabricator.wikimedia.org/T424655#11865711 (10A_smart_kitten) Upstream issue [[https://github.com/hound-search/hound/issues/305|hound-search/hound#305]] seems potentially relevant here, and a comment on that issue... [12:56:22] (03open) 10volans: [toolforge] fix python version configuration [toolforge-repos/pywikibot-buildservice] (toolforge) - 10https://gitlab.wikimedia.org/toolforge-repos/pywikibot-buildservice/-/merge_requests/2 [12:57:03] (03approved) 10fnegri: [toolforge] fix python version configuration [toolforge-repos/pywikibot-buildservice] (toolforge) - 10https://gitlab.wikimedia.org/toolforge-repos/pywikibot-buildservice/-/merge_requests/2 (owner: 10volans) [12:57:57] (03merge) 10volans: [toolforge] fix python version configuration [toolforge-repos/pywikibot-buildservice] (toolforge) - 10https://gitlab.wikimedia.org/toolforge-repos/pywikibot-buildservice/-/merge_requests/2 [13:01:13] 06cloud-services-team, 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T423790#11865762 (10Volans) 05Open→03Resolved a:03Volans New version of pywikibot has been deployed to toolforge. [13:07:21] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T422184) [13:10:11] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T422184) [13:12:35] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658 (10fgiunchedi) 03NEW [13:19:19] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#11865892 (10fgiunchedi) [13:19:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:20:14] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [13:24:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:28:35] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T423789#11865945 (10Volans) 05Open→03Resolved a:03Volans Pywikibot has been updated in paws. The deployment was timing out for me, it worked for Francesco, but then paws hub... [13:32:24] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#11865980 (10Andrew) A minor subtask of this is to add some kind of annotation to the grafana dashboard reminding us that 66% ram usage is actually... [13:46:43] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 13Patch-For-Review: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#11866031 (10HakanIST) We ran into an issue with a Python webservice (lingua-libre) where OAuth callbacks were failing with `upstream connect e... [13:54:59] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [logs-api,jobs-cli] `toolforge jobs logs` has inconsistent ordering - https://phabricator.wikimedia.org/T401552#11866067 (10Raymond_Ndibe) a:03Raymond_Ndibe [13:55:03] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [general] upgrade all python repos to python >=3.13 - https://phabricator.wikimedia.org/T422184#11866068 (10Raymond_Ndibe) a:03Raymond_Ndibe [13:56:09] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#11866072 (10fgiunchedi) [14:05:37] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/20 [14:05:47] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/15 [14:06:00] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/78 [14:06:10] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/78 [14:06:18] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/165 [14:06:22] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/165 [14:06:25] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/154 [14:06:51] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/21 [14:06:55] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/87 [14:07:07] 10VPS-project-Codesearch: Make it possible to use negative regex assertions in Codesearch - https://phabricator.wikimedia.org/T424655#11866124 (10A_smart_kitten) [14:07:08] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/79 [14:07:09] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/154 [14:07:13] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/133 [14:07:13] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/21 [14:07:13] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/46 [14:07:25] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/103 [14:07:34] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/87 [14:07:46] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/166 [14:08:05] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/293 [14:08:06] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/107 [14:08:29] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/46 [14:08:31] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/47 [14:08:32] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/134 [14:08:38] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/79 [14:08:46] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/73 [14:08:55] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/158 [14:08:58] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/92 [14:09:00] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/294 [14:09:01] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/320 [14:09:02] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/89 [14:09:03] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/108 [14:09:05] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/133 [14:09:11] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/65 [14:09:14] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/103 [14:09:20] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/166 [14:09:21] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/64 [14:09:26] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/93 [14:09:30] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/65 [14:09:34] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/293 [14:09:38] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/107 [14:09:50] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/47 [14:10:02] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/134 [14:10:06] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/73 [14:10:10] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/158 [14:10:14] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/92 [14:10:18] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/294 [14:10:23] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/89 [14:10:27] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/320 [14:10:31] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/108 [14:10:35] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/65 [14:10:39] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/64 [14:10:43] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/93 [14:10:47] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/65 [14:27:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:27:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [14:29:20] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/165 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:29:21] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/165 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:29:37] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/165 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:30:00] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/65 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:30:01] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/65 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:30:10] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/65 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:30:16] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/93 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:30:17] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/93 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:30:21] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/93 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:30:28] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/64 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:30:34] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/64 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:30:35] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/64 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:11] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/65 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:12] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/65 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:15] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/65 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:25] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/108 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:26] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/108 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:31] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/108 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:38] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/89 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:39] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/89 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:44] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/89 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:50] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/320 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:51] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/320 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:31:57] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/320 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:32:02] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/64 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:32:04] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/294 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:32:11] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/294 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:32:17] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/294 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:32:19] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/92 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:32:22] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/92 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:32:32] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/92 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:32:32] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:32:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [14:32:46] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/158 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:32:49] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/158 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:32:57] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/158 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:32:58] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/73 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:33:01] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/73 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:33:05] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/73 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:33:12] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/134 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:33:18] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/134 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:33:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate [14:33:28] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/134 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:33:28] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/47 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:33:33] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/47 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:33:45] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/47 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:33:46] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/107 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:33:52] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,designate [14:33:53] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/107 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:34:20] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/107 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:34:49] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-builder: bump to 0.0.148-20260428143158-a5e19815 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1227 [14:34:50] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/103 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:34:54] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/107 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:35:02] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.193-20260428142955-5938760e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1228 [14:35:07] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.92-20260428143040-1ee9495a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1229 [14:35:19] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/103 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:35:23] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.70-20260428143129-1fde83a4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1230 [14:35:27] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-builder: bump to 0.0.148-20260428143158-a5e19815 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1227 [14:35:31] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/103 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:35:35] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.193-20260428142955-5938760e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1228 [14:35:39] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/92 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:35:43] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.92-20260428143040-1ee9495a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1229 [14:35:49] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/46 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:36:01] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.70-20260428143129-1fde83a4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1230 [14:36:12] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/46 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:36:30] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/46 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:36:38] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/133 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:36:46] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/133 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:36:50] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/107 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:36:54] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/133 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:36:58] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/133 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:37:11] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/79 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:37:15] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/79 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:37:23] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/79 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:37:25] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/87 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:37:29] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/87 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:37:42] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/87 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:37:44] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/21 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:37:52] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/21 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:37:56] FIRING: [3x] SystemdUnitDown: The service unit designate-producer.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:38:04] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/154 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:38:05] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/21 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:38:11] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/154 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:38:20] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/46 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:38:21] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.93-20260428143555-d3e3148d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1229 [14:38:25] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/154 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:38:25] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/78 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:38:39] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/78 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:38:56] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.81-20260428143413-b29ea7c6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1231 [14:39:02] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.93-20260428143555-d3e3148d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1229 [14:39:05] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.81-20260428143413-b29ea7c6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1231 [14:39:11] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/78 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:39:23] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/78 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:39:30] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/133 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:40:14] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/15 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:40:17] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/15 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:40:23] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/15 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:40:56] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.195-20260428143817-6fc04f02 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1232 [14:41:17] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.195-20260428143817-6fc04f02 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1232 [14:41:18] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/20 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:41:19] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/78 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:41:28] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.82-20260428143906-b0b51e37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1231 [14:41:28] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.20-20260428143842-eeee2645 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1233 [14:41:30] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.20-20260428143842-eeee2645 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1233 [14:41:45] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/20 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:41:49] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.82-20260428143906-b0b51e37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1231 [14:41:50] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/20 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:42:22] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/293 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:43:24] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-api: bump to 0.0.84-20260428143331-13957a07 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1234 [14:43:30] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-api: bump to 0.0.84-20260428143331-13957a07 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1234 [14:44:24] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/20 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:46:34] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/293 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:46:46] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.21-20260428144444-f2a67a66 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1233 [14:46:59] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.21-20260428144444-f2a67a66 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1233 [14:47:42] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/166 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:55:26] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002#11866320 (10AGhirelli-WMF) Note for estimation: - T423411 (currently in progress) unifies example validation for parameters, requestBody, and... [14:56:10] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002#11866326 (10AGhirelli-WMF) [15:04:32] FIRING: InstanceDown: Project metricsinfra instance metricsinfra-thanos-fe-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:04:32] FIRING: [2x] TargetDown: Job thanos-query is unreachable in project metricsinfra instance metricsinfra-thanos-fe-2 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [15:22:56] FIRING: [4x] SystemdUnitDown: The service unit designate-producer.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:24:46] (03open) 10isaranto: channels: update wikimedia-ml tags to match quarter variants [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/69 [15:25:28] (03update) 10isaranto: channels: update wikimedia-ml tags to match quarter variants [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/69 [15:36:22] FIRING: [3x] HAProxyBackendUnavailable: HAProxy service designate-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:41:22] RESOLVED: [3x] HAProxyBackendUnavailable: HAProxy service designate-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:41:52] 10Cloud-Services: Migrate Tools Platform Envoy TLS proxy services to the 2026 discovery intermediate - https://phabricator.wikimedia.org/T424675 (10MoritzMuehlenhoff) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/prof... [15:42:56] FIRING: [4x] SystemdUnitDown: The service unit designate-producer.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:45:55] (03open) 10taavi: istio-system: Disable X-Envoy-Peer-Metadata headers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) [15:46:01] (03update) 10taavi: istio-system: Disable X-Envoy-Peer-Metadata headers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) [15:46:50] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-system [15:47:05] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system [15:49:44] FIRING: [2x] IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [15:53:10] (03update) 10taavi: istio-system: Disable X-Envoy-Peer-Metadata headers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) [15:53:24] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-system [15:53:32] 10Cloud-VPS, 06tools-infrastructure-team: Migrate Tools Platform Envoy TLS proxy services to the 2026 discovery intermediate - https://phabricator.wikimedia.org/T424675#11866716 (10taavi) [15:53:39] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system [15:54:44] FIRING: [5x] IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [15:59:15] 10Toolforge, 06tools-platform-team: Cannot output "Ερευνητής Αλήθειας" - https://phabricator.wikimedia.org/T367663#11866737 (10Raymond_Ndibe) 05Open→03In progress [15:59:44] RESOLVED: [5x] IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [16:01:55] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 13Patch-For-Review: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#11866752 (10taavi) >>! In T392356#11866031, @HakanIST wrote: > Issue in T424029 with a Python webservice (lingua-libre) where OAuth callbacks... [16:06:51] 10Tool-refill: massive error.log file spam due to LabsDB.php PHP fatal error - https://phabricator.wikimedia.org/T389917#11866794 (10Novem_Linguae) 05Open→03Resolved a:03Novem_Linguae I deleted the log files, then made some changes to the front end code (and deleted extraneous front end code). It's bee... [16:08:41] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:09:58] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: Toolforge on bare metal POC - https://phabricator.wikimedia.org/T407296#11866822 (10Andrew) 05Open→03Stalled Update on this task: Due to further discussion + budget and staffing changes, we are putting this project on indefinite hold. There is a gene... [16:12:39] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11866840 (10aputhin) a:05dcaro→03None [16:13:58] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[56] - https://phabricator.wikimedia.org/T419892#11866850 (10Jclark-ctr) a:03Jclark-ctr [16:15:49] 10Toolforge, 06tools-platform-team: [Toolforge]: Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#11866875 (10aputhin) 05Open→03In progress [16:17:02] 10Toolforge, 06tools-platform-team: Document updated process for cloud services survey - https://phabricator.wikimedia.org/T424609#11866884 (10aputhin) 05Open→03In progress [16:17:56] RESOLVED: [3x] SystemdUnitDown: The service unit designate-producer.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:18:11] 10Toolforge, 06tools-platform-team: [Toolforge]: User Activity (based on successful SSH logins) - https://phabricator.wikimedia.org/T423228#11866895 (10aputhin) 05In progress→03Resolved First iteration is done, automation work will follow on T423549 [16:24:26] FIRING: [4x] SystemdUnitDown: The service unit designate-producer.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:26:36] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#11866938 (10fnegri) 05Open→03In progress a:03fgiunchedi [16:28:17] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#11866945 (10fnegri) Assigning to @fgiunchedi as he's actively working on this. This [announcement](https://lists.wikimedia.org/hyperkitty/li... [16:29:02] 10VPS-project-Phabricator, 06collaboration-services, 06Infrastructure-Foundations, 10Mail, 10Phabricator: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance - https://phabricator.wikimedia.org/T422559#11866948 (10A_smart_kitten) From [[https://we.phorge... [16:38:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:43:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:53:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:58:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:04:20] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[56] - https://phabricator.wikimedia.org/T419892#11867127 (10Jclark-ctr) @Andrew This server can’t go in D5 that switch only supports 1G/10G. @ayounsi Can you confirm I’m right? I believe the only WMCS ra... [17:09:32] RESOLVED: InstanceDown: Project metricsinfra instance metricsinfra-thanos-fe-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:09:32] RESOLVED: [2x] TargetDown: Job thanos-query is unreachable in project metricsinfra instance metricsinfra-thanos-fe-2 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [17:24:49] 10VPS-project-Codesearch: Make it possible to use negative regex assertions in Codesearch - https://phabricator.wikimedia.org/T424655#11867251 (10A_smart_kitten) [17:46:03] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/166 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:52:31] 10Cloud-VPS (Project-requests): Request creation of thelounge Cloud VPS project - https://phabricator.wikimedia.org/T424443#11867422 (10bd808) [17:52:53] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/166 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:52:54] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/166 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:53:02] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/166 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:53:05] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/166 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:53:50] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/293 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:53:52] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/293 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:53:56] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/293 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:55:49] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.194-20260428175314-8fe67579 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1228 [17:56:01] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.194-20260428175314-8fe67579 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1228 [17:57:24] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.486-20260428175408-acc0a0f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1236 [17:57:30] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.486-20260428175408-acc0a0f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1236 [18:10:35] 10Tool-curator: Curator: Uploads by one specific user fail all the time - https://phabricator.wikimedia.org/T424757 (10PantheraLeo1359531) 03NEW [18:16:56] FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:21:34] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T422184) [18:21:35] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T422184) [18:37:35] (03open) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/22 (https://phabricator.wikimedia.org/T422184) [18:46:57] (03open) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/91 (https://phabricator.wikimedia.org/T422184) [18:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:48:56] (03open) 10somerandomdeveloper: Ignore V+1 from Cindy The Browser Test Bot [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/70 (https://phabricator.wikimedia.org/T421334) [18:53:22] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/91 (https://phabricator.wikimedia.org/T422184) [18:54:36] (03update) 10somerandomdeveloper: Ignore V+1 from Cindy The Browser Test Bot [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/70 (https://phabricator.wikimedia.org/T421334) [18:55:26] 10Wikibugs, 13Patch-For-Review: Wikibugs should ignore V+1 from Cindy The Browser Test Bot - https://phabricator.wikimedia.org/T421334#11868323 (10SomeRandomDeveloper) a:03SomeRandomDeveloper [18:56:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:57:30] (03update) 10raymond-ndibe: logs-api: bump to 0.0.21-20260428144444-f2a67a66 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1233 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:57:32] (03approved) 10raymond-ndibe: logs-api: bump to 0.0.21-20260428144444-f2a67a66 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1233 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:57:35] (03merge) 10raymond-ndibe: logs-api: bump to 0.0.21-20260428144444-f2a67a66 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1233 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:57:56] (03update) 10somerandomdeveloper: Ignore V+1 from Cindy The Browser Test Bot [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/70 (https://phabricator.wikimedia.org/T421334) [18:58:04] (03update) 10raymond-ndibe: components-api: bump to 0.0.194-20260428175314-8fe67579 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1228 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:58:16] (03update) 10raymond-ndibe: components-api: bump to 0.0.194-20260428175314-8fe67579 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1228 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:58:20] (03approved) 10raymond-ndibe: components-api: bump to 0.0.194-20260428175314-8fe67579 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1228 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:58:32] (03merge) 10raymond-ndibe: components-api: bump to 0.0.194-20260428175314-8fe67579 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1228 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:59:19] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.486-20260428175408-acc0a0f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1236 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:59:19] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.486-20260428175408-acc0a0f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1236 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:59:24] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.486-20260428175408-acc0a0f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1236 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [19:01:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:07:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:12:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:22:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:22:52] (03open) 10raymond-ndibe: use python3.13 [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/16 (https://phabricator.wikimedia.org/T422184) [19:23:02] (03update) 10raymond-ndibe: use python3.13 [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/16 (https://phabricator.wikimedia.org/T422184) [19:32:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:32:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:33:21] (03open) 10raymond-ndibe: use python3.13 [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/26 (https://phabricator.wikimedia.org/T422184) [19:33:30] (03update) 10raymond-ndibe: use python3.13 [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/16 (https://phabricator.wikimedia.org/T422184) [19:37:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:39:22] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.82-20260428143906-b0b51e37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1231 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [19:39:26] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.82-20260428143906-b0b51e37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1231 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [19:39:32] (03approved) 10raymond-ndibe: jobs-emailer: bump to 0.0.82-20260428143906-b0b51e37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1231 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [19:39:55] (03merge) 10raymond-ndibe: jobs-emailer: bump to 0.0.82-20260428143906-b0b51e37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1231 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [19:41:33] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/153 (https://phabricator.wikimedia.org/T422184) [19:43:05] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [19:43:09] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [19:51:02] (03open) 10kineticpelagic: Draft: Add rules for operations and paths (T422504) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/17 [19:57:17] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.212-20260428143311-7d2493f4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1237 [19:57:18] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.212-20260428143311-7d2493f4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1237 [20:56:21] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.486-20260428175408-acc0a0f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1236 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:56:23] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.486-20260428175408-acc0a0f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1236 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:56:32] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.486-20260428175408-acc0a0f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1236 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:56:57] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.195-20260428143817-6fc04f02 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1232 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:57:01] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.486-20260428175408-acc0a0f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1236 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:57:12] (03approved) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.195-20260428143817-6fc04f02 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1232 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:57:13] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.195-20260428143817-6fc04f02 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1232 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:57:40] (03merge) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.195-20260428143817-6fc04f02 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1232 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:58:36] 10Wikibugs, 13Patch-For-Review: Wikibugs should ignore V+1 from Cindy The Browser Test Bot - https://phabricator.wikimedia.org/T421334#11868688 (10bd808) 05Open→03In progress [21:18:50] (03update) 10danyya: Draft: Country stats [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/20 [22:06:00] (03update) 10somerandomdeveloper: Ignore V+1 from Cindy The Browser Test Bot [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/70 (https://phabricator.wikimedia.org/T421334) [22:13:39] 10Tool-curator: Curator: Uploads by one specific user fail all the time - https://phabricator.wikimedia.org/T424757#11868885 (10Albinfo) Loging out and loging in again to Curator helped (as suggested by DaxServer). Sorry, I didn't try this before. [22:16:37] (03approved) 10bd808: Ignore V+1 from Cindy The Browser Test Bot [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/70 (https://phabricator.wikimedia.org/T421334) (owner: 10somerandomdeveloper) [22:16:41] (03update) 10bd808: Ignore V+1 from Cindy The Browser Test Bot [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/70 (https://phabricator.wikimedia.org/T421334) (owner: 10somerandomdeveloper) [22:17:11] FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [22:23:11] 10Wikibugs: Update repo to work with 24 builder stack - https://phabricator.wikimedia.org/T424779 (10bd808) 03NEW [22:23:52] 10Wikibugs: Update repo to work with 24 builder stack - https://phabricator.wikimedia.org/T424779#11868958 (10bd808) [22:23:56] 10Wikibugs, 13Patch-For-Review: Wikibugs should ignore V+1 from Cindy The Browser Test Bot - https://phabricator.wikimedia.org/T421334#11868957 (10bd808) [22:24:18] 10Wikibugs: Update repo to work with 24 builder stack - https://phabricator.wikimedia.org/T424779#11868960 (10bd808) p:05Triage→03High [22:41:41] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 13Patch-For-Review: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#11868990 (10Arendpieter) Just to make sure: this issue only affects Toolforge, i.e., Cloud Services, right? Is ingress-nginx also used on the... [23:19:57] 10Tool-gitlab-account-approval: Recognize and follow Developer account -> Phabricator links in LDAP - https://phabricator.wikimedia.org/T424783 (10bd808) 03NEW [23:21:03] (03open) 10komla: Add 'community' section to notes template [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/3 [23:39:34] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Icebox, 07Epic: Plan a replacement for wiki replicas that is better suited to typical OLAP use cases than the MediaWiki OLTP schema - https://phabricator.wikimedia.org/T215858#11869177 (10ttaylor) One day..... [23:40:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [23:41:31] (03merge) 10samwilson: Check temp file before adding picture to Zip file [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/8 (https://phabricator.wikimedia.org/T415479) [23:45:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors