[01:00:36] 06tools-infrastructure-team: Track toolforge virtualization footprint: ram/cores/instance &c - https://phabricator.wikimedia.org/T436275#12305900 (10Andrew) As for resource usage within toolforge, we already have the 'capacity' panels on this dash: https://grafana.wmcloud.org/d/8GiwHDL4k/infra-kubernetes-cluste... [01:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:45:09] (03CR) 10AntiCompositeNumber: "Hello," [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1312290 (owner: 10Amanojaku) [01:45:48] 06tools-infrastructure-team, 10wikitech.wikimedia.org: wikitech-static: get better at cleaning up old containers and/or establish a cadence for rebuilding the host - https://phabricator.wikimedia.org/T432391#12306028 (10Andrew) ` admin@ip-172-31-36-12:/$ uptime 01:33:01 up 7 min, 1 user, load average: 0.18,... [02:05:55] 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12306081 (10Ottomata) Semi related: {T437541} [03:02:42] 10Tool-projo: Projo: Move NPP snapshot state into MariaDB and remove NFS publication locking - https://phabricator.wikimedia.org/T437544 (10Audiodude) 03NEW p:05Triage→03Low [04:13:50] 10Tool-projo: Use selection picker on check tone page - https://phabricator.wikimedia.org/T437550 (10Audiodude) 03NEW [04:15:01] 10Tool-projo: Make jobs page the same width as the rest of the site (minor) - https://phabricator.wikimedia.org/T437551 (10Audiodude) 03NEW [04:19:38] 10Tool-projo: Re-use the queued/reopen UX on the Improve page - https://phabricator.wikimedia.org/T437552 (10Audiodude) 03NEW [05:00:03] 10Tool-projo: Consider moving topic based generation to job queue - https://phabricator.wikimedia.org/T437553 (10Audiodude) 03NEW [05:04:32] 10Tool-projo: Consider moving topic based generation to job queue - https://phabricator.wikimedia.org/T437553#12306322 (10Audiodude) To see the failure currently, try the topic `chess`. [05:09:14] 10Tool-projo: Consider moving topic based generation to job queue - https://phabricator.wikimedia.org/T437553#12306324 (10Audiodude) Seems like the prudent temporary solution is to disable vector re-ranking. See: https://gitlab.wikimedia.org/tmoney/projo/-/merge_requests/11 [06:05:25] FIRING: SystemdUnitFailed: prometheus-node-textfile-cloudvps_project_usage.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:49:43] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12306430 (10fgiunchedi) +tools-infra for visibility, thank you @Peter for the detailed task ! [07:44:31] 10Cloud-VPS, 06tools-infrastructure-team, 10Cumin, 06Infrastructure-Foundations: Enable cumin hostfile backend on cloudcumin hosts - https://phabricator.wikimedia.org/T433916#12306547 (10Volans) @Andrew how are you generating the list of hosts? On all cumin hosts, given the dependency on clustershell, we d... [07:46:55] 10Tool-wikinewsie, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S3 2026 September): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12306550 (10Nikerabbit) [07:47:11] 10Tool-wikinewsie, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S3 2026 September): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12306551 (10Nikerabbit) a:03Nikerabbit [08:11:28] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12306593 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloud... [08:13:59] 10Cloud-VPS (Project-requests), 06MediaWiki-Media-Platform-Team: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088#12306597 (10Ladsgroup) Thanks! [08:14:46] 10Tool-wikinewsie, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Ongoing maintenance), and 2 others: Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12306599 (10Nikerabbit) [08:46:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:51:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:55:57] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12306764 (10elukey) @Andrew I tried to dig a bit more into it and these are the disks and their size as seen by d-i: ` sda: rota=0 s... [10:00:33] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12306979 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt... [10:05:32] 10Tool-gitlab-bot: Create a skeltal GitLab bot that can respond to messages in merge request comments - https://phabricator.wikimedia.org/T408445#12307007 (10dcaro) I'm the worst, sorry for missing this task, I started creating https://gitlab.wikimedia.org/toolforge-repos/cloud-glab-bot in my volunteer time, tha... [10:05:39] FIRING: SystemdUnitFailed: prometheus-node-textfile-cloudvps_project_usage.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:09:48] 10Cloud Services Proposals, 06cloud-services-team, 06tools-platform-team: Decision request - Update and expand on coding standards for the tools-platform team - https://phabricator.wikimedia.org/T408782#12307026 (10dcaro) [10:10:56] (03update) 10raymond-ndibe: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [10:12:19] 10Cloud Services Proposals, 06tools-platform-team: Decision request - tools-platform internal review workflow - https://phabricator.wikimedia.org/T437573 (10dcaro) 03NEW [10:12:25] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12307040 (10elukey) @Andrew I took a look with Filippo today and these are our findings: * older cloudcephosd have two SSDs and a so... [10:13:06] 10Cloud Services Proposals, 06tools-platform-team: Decision request - tools-platform internal review workflow - https://phabricator.wikimedia.org/T437573#12307043 (10dcaro) [10:13:44] 10Cloud Services Proposals, 06tools-platform-team: Decision request - tools-platform internal review workflow - https://phabricator.wikimedia.org/T437573#12307046 (10dcaro) [10:14:00] 10Cloud Services Proposals, 06tools-platform-team: Decision request - tools-platform internal review workflow - https://phabricator.wikimedia.org/T437573#12307049 (10dcaro) p:05Triage→03High [10:14:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [10:14:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [10:14:44] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [10:14:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [10:14:57] 10Cloud Services Proposals, 06tools-platform-team: Decision request - tools-platform internal review workflow - https://phabricator.wikimedia.org/T437573#12307051 (10dcaro) [10:14:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [10:14:58] 10Cloud Services Proposals, 06cloud-services-team, 06tools-platform-team: Decision request - Update and expand on coding standards for the tools-platform team - https://phabricator.wikimedia.org/T408782#12307052 (10dcaro) [10:14:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [10:15:59] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [10:16:00] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [10:17:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [10:17:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [10:30:01] 10Tool-wikinewsie, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S3 2026 September): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12307078 (10Nikerabbit) [11:13:46] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [11:13:53] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [11:13:59] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [11:14:52] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [11:15:29] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [11:31:09] 14cloud-services-team (FY2025/2026-Q1-Q2), 10Cloud-VPS, 14Toolforge (Toolforge iteration 22), 06tools-infrastructure-team, and 2 others: 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures - https://phabricator.wikimedia.org/T399281#12307263 (10MLechvien-WMF) [11:31:46] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 07Wikimedia-Incident: 2024-11-26 Toolforge DNS incident - https://phabricator.wikimedia.org/T380844#12307287 (10MLechvien-WMF) [11:31:49] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12307290 (10Marostegui) clouddb1024 is now running x4 - it is not yet ready to hold views, I need to make sure ever... [11:33:19] 14cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-VPS, 06tools-infrastructure-team, 05Cloud-Services-Origin-User, and 5 others: Cloud Ceph outage 2023-02-13 - https://phabricator.wikimedia.org/T329535#12307326 (10MLechvien-WMF) [11:34:03] 14cloud-services-team (Kanban), 10Toolforge, 06tools-platform-team, 07Wikimedia-Incident: The following container images did not match any of the allowed registries ([['docker-registry.tools.wmflabs.org']]) - https://phabricator.wikimedia.org/T320446#12307349 (10MLechvien-WMF) [11:37:08] 14cloud-services-team (Kanban), 10Toolforge, 06tools-platform-team, 07Wikimedia-Incident: login.toolforge.org ssh broken due to LDAP lookup failures - https://phabricator.wikimedia.org/T262347#12307407 (10MLechvien-WMF) [11:37:43] 10Toolforge, 06tools-platform-team, 07Wikimedia-Incident: https://tools.wmflabs.org/ is redirecting to itself - https://phabricator.wikimedia.org/T247501#12307423 (10MLechvien-WMF) [11:37:55] 14cloud-services-team (Kanban), 10Cloud-VPS, 06tools-infrastructure-team, 07Wikimedia-Incident: CloudVPS: m5-master databases for openstack may require re-encoding - https://phabricator.wikimedia.org/T234830#12307428 (10MLechvien-WMF) [11:38:26] 14cloud-services-team (Kanban), 10Cloud-VPS, 10Tools, 06tools-infrastructure-team, 07Wikimedia-Incident: Resolve keystone schema errors - https://phabricator.wikimedia.org/T235070#12307441 (10MLechvien-WMF) [11:38:29] 14cloud-services-team (Kanban), 10Cloud-VPS, 10Tools, 06tools-infrastructure-team, 07Wikimedia-Incident: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07 - https://phabricator.wikimedia.org/T234834#12307443 (10MLechvien-WMF) [11:38:46] 14cloud-services-team (Kanban), 10Toolforge, 06tools-platform-team, 07Wikimedia-Incident: Audit tools project puppet CA certs to ensure that they are all consistent - https://phabricator.wikimedia.org/T232772#12307451 (10MLechvien-WMF) [11:40:18] 10Cloud-VPS, 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 07Wikimedia-Incident: contintcloud instance refuses to launch due to "Maximum number of fixed ips exceeded - https://phabricator.wikimedia.org/T171158#12307494 (10MLechvien-WMF) [12:08:22] FIRING: HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-x4 backend clouddb1025.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [12:09:22] FIRING: [2x] HAProxyWikiReplicaSectionUnavailable: Wiki replica section x4 has no available servers on cloudlb1001:9900 - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyWikiReplicaSectionUnavailable [12:13:36] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12307642 (10fgiunchedi) >>! In T419892#12307040, @elukey wrote: > @Andrew I took a look with Filippo today and these are our findings... [12:13:42] 10Tool-wikinewsie, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S3 2026 September): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12307645 (10abi_) 05Open→03Resolved Exports: https://gitlab.wikimedia.org/toolforge-re... [12:13:56] 10Tool-wikinewsie, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S3 2026 September): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12307649 (10abi_) [12:14:07] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [12:14:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [12:14:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [12:15:40] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [12:16:53] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12307653 (10fgiunchedi) 05Resolved→03Open a:05VRiley-WMF→03fgiunchedi I'll take this until 1075 is fully back in service (... [12:17:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [13:02:31] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12307868 (10dcaro) a:05dcaro→03None [13:11:37] (03update) 10dcaro: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) [13:14:49] (03merge) 10dcaro: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) [13:18:26] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.214-20260910131459-703c82b0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1415 (https://phabricator.wikimedia.org/T434496) [13:22:19] (03CR) 10Krinkle: [C:03+2] write_config: Add mediawiki/vendor to "Everything" preset [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1336608 (https://phabricator.wikimedia.org/T227704) (owner: 10Krinkle) [13:24:48] (03CR) 10CI reject: [V:04-1] write_config: Add mediawiki/vendor to "Everything" preset [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1336608 (https://phabricator.wikimedia.org/T227704) (owner: 10Krinkle) [13:44:08] 10Openstack-Magnum, 06tools-infrastructure-team: Replace all heat-based magnum clusters - https://phabricator.wikimedia.org/T437496#12308060 (10Andrew) [13:46:20] 10Cloud-VPS, 06tools-infrastructure-team: Upgrade cloud-vps openstack to version 'Gazpacho' - https://phabricator.wikimedia.org/T437595 (10Andrew) 03NEW [13:46:42] 10Cloud-VPS, 06tools-infrastructure-team: Upgrade cloud-vps openstack to version 'Gazpacho' - https://phabricator.wikimedia.org/T437595#12308076 (10Andrew) [13:46:44] 10Openstack-Magnum, 06tools-infrastructure-team: Replace all heat-based magnum clusters - https://phabricator.wikimedia.org/T437496#12308077 (10Andrew) [13:47:45] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [13:52:16] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [13:55:37] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12308106 (10bking) Per Slack conversation with @brouberol , I'm starting the work on this now. [13:57:35] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12308108 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2003 for hos... [13:57:41] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [13:59:31] (03approved) 10dcaro: Fix missing space in "No deployments found" output [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T437212) (owner: 10coderanimeshpathak04092007) [13:59:36] (03merge) 10dcaro: Fix missing space in "No deployments found" output [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T437212) (owner: 10coderanimeshpathak04092007) [14:00:47] 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 06The-Wikipedia-Library, 07Epic: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402053#12308112 (10jsn.sherman) 05Open→03Resolved a:03jsn.sherman Meant to close this with the last subtask. whoops... [14:02:09] (03open) 10dcaro: d/changelog: bump to 0.0.21 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/99 (https://phabricator.wikimedia.org/T437212) [14:02:50] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [14:03:06] (03approved) 10dcaro: components-api: bump to 0.0.214-20260910131459-703c82b0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1415 (https://phabricator.wikimedia.org/T434496) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:03:17] (03merge) 10dcaro: components-api: bump to 0.0.214-20260910131459-703c82b0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1415 (https://phabricator.wikimedia.org/T434496) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:03:32] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-cli [14:03:40] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [components-api] add the option to the tool configuration bulid section to allow passing built time envvars - https://phabricator.wikimedia.org/T434496#12308127 (10dcaro) 05In progress→03Resolved [14:06:25] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli [14:06:43] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-cli [14:07:12] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: More granular 'wmcs' clusters for better observability - https://phabricator.wikimedia.org/T437272#12308158 (10fgiunchedi) [14:09:57] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli [14:11:28] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12308171 (10Peter) [14:13:39] (03approved) 10dcaro: d/changelog: bump to 0.0.21 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/99 (https://phabricator.wikimedia.org/T437212) [14:13:45] (03merge) 10dcaro: d/changelog: bump to 0.0.21 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/99 (https://phabricator.wikimedia.org/T437212) [14:20:25] RESOLVED: SystemdUnitFailed: prometheus-node-textfile-cloudvps_project_usage.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:30:50] (03approved) 10dcaro: emailer: include pod status message in failure emails [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/58 (https://phabricator.wikimedia.org/T384252) (owner: 10mahveotm) [14:31:00] (03merge) 10dcaro: emailer: include pod status message in failure emails [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/58 (https://phabricator.wikimedia.org/T384252) (owner: 10mahveotm) [14:31:29] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [14:31:29] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-emailer [14:33:15] (03update) 10dcaro: envvars-admission: Drop old ElasticSearch-related variables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1394 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi) [14:34:00] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [14:34:06] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.95-20260910143110-3c3ee543 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1416 (https://phabricator.wikimedia.org/T384252) [14:34:08] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.95-20260910143110-3c3ee543 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1416 (https://phabricator.wikimedia.org/T384252) [14:51:54] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [14:54:03] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [14:54:30] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [15:02:39] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12308559 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2003 for host cl... [15:14:16] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [15:14:24] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [15:14:24] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [15:14:37] (03update) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [15:14:39] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [15:15:35] (03update) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [15:21:04] (03approved) 10dcaro: jobs-emailer: bump to 0.0.95-20260910143110-3c3ee543 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1416 (https://phabricator.wikimedia.org/T384252) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:21:14] (03merge) 10dcaro: jobs-emailer: bump to 0.0.95-20260910143110-3c3ee543 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1416 (https://phabricator.wikimedia.org/T384252) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:21:37] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12308698 (10Andrew) > cloudcephosd[1048-52] have a single nvme disk (no hw raid of any kind) Those hosts have a BOSS card which is a... [15:22:05] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-emailer] If the pod is in error status, try to get the status.message field in the email, otherwise just 'error' is not that useful - https://phabricator.wikimedia.org/T384252#12308701 (10dcaro) 05In progress→03Reso... [15:32:10] (03update) 10dcaro: envvars-admission: Drop old ElasticSearch-related variables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1394 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi) [15:32:12] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [15:35:26] (03update) 10dcaro: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [15:35:59] (03update) 10dcaro: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [15:36:02] (03update) 10dcaro: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [15:36:07] (03update) 10dcaro: models.py: expose state and webservice_defaults in images endpoint [repos/cloud/toolforge/jobs-api] (add_webservice_endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679) (owner: 10raymond-ndibe) [15:36:10] (03update) 10dcaro: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [15:36:14] (03update) 10dcaro: support --webservice option [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [15:36:19] (03update) 10dcaro: Expand `source` support in `ToolConfig` [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T402764) (owner: 10damian) [15:41:10] 10Toolforge, 06tools-platform-team, 07good first task: Missing space in "no deployments found" message - https://phabricator.wikimedia.org/T437212#12308854 (10dcaro) 05Open→03Resolved [15:42:13] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli] Use the new `status` fields instead of the old short/long statuses - https://phabricator.wikimedia.org/T430904#12308861 (10dcaro) I don't see any code reviews open, bumping to in progress [15:42:23] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [15:42:38] (03update) 10dcaro: gen: update toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/188 (https://phabricator.wikimedia.org/T436485) (owner: 10raymond-ndibe) [15:43:27] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [15:50:47] (03approved) 10dcaro: gen: update toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/188 (https://phabricator.wikimedia.org/T436485) (owner: 10raymond-ndibe) [15:51:01] 06tools-platform-team, 13Patch-For-Review: components-api: update toolforge models - https://phabricator.wikimedia.org/T436485#12308908 (10dcaro) 05Open→03In progress [15:51:55] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [toollabs-images] stop hardcoding port 8000 in the webservice-runner scripts - https://phabricator.wikimedia.org/T432078#12308912 (10dcaro) The MR has been approved already. [15:52:36] (03update) 10dcaro: jobs: Add timeout support for one-off jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/373 (https://phabricator.wikimedia.org/T401110) (owner: 10mahveotm) [15:54:11] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [15:57:06] (03approved) 10dcaro: envvars-admission: Drop old ElasticSearch-related variables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1394 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi) [15:57:19] (03merge) 10dcaro: envvars-admission: Drop old ElasticSearch-related variables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1394 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi) [15:58:32] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api,jobs-cli] Support `--timeout` for one-off jobs - https://phabricator.wikimedia.org/T401110#12308952 (10dcaro) Being reviewed by @Raymond_Ndibe [16:00:03] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [components-api] split cancel endpoint to avoid duplicated operation id - https://phabricator.wikimedia.org/T434499#12308966 (10dcaro) Being reviewed by @Raymond_Ndibe [16:08:37] FIRING: HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-x4 backend clouddb1025.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:09:22] FIRING: [2x] HAProxyWikiReplicaSectionUnavailable: Wiki replica section x4 has no available servers on cloudlb1001:9900 - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyWikiReplicaSectionUnavailable [16:13:26] 10Toolforge, 06tools-platform-team: [components-cli] releases CI job stopped working - https://phabricator.wikimedia.org/T437624 (10dcaro) 03NEW [16:13:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [16:13:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [16:14:04] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [16:15:00] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [16:15:36] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [16:19:16] 06cloud-services-team, 10Quarry, 06tools-platform-team, 06RoadToWiki, and 3 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12309031 (10github-toolforge-bot) david-caro opened https://github.com/toolforge/quarry/pull/103 [16:32:05] 10Quarry, 06tools-platform-team: [quarry] Deployment fails to run from bastion - https://phabricator.wikimedia.org/T437626 (10dcaro) 03NEW [16:32:45] 06cloud-services-team, 10Quarry, 06tools-platform-team, 06RoadToWiki, and 3 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12309088 (10dcaro) Quarry is failing to deploy it seems :/ {T437626} [16:32:49] 10Quarry, 06tools-platform-team: [quarry] Deployment fails to run from bastion - https://phabricator.wikimedia.org/T437626#12309093 (10dcaro) [16:32:58] 06cloud-services-team, 10Quarry, 06tools-platform-team, 06RoadToWiki, and 3 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12309092 (10dcaro) [16:33:03] 10Quarry, 06tools-platform-team: [quarry] Deployment fails to run from bastion - https://phabricator.wikimedia.org/T437626#12309097 (10taavi) dupe of {T395238}? [16:33:48] 10Quarry, 06tools-platform-team: [quarry] Deployment fails to run from bastion - https://phabricator.wikimedia.org/T437626#12309106 (10dcaro) >>! In T437626#12309097, @taavi wrote: > dupe of {T395238}? Thanks! I'll merge into that one [16:34:19] 10Quarry, 06tools-platform-team: [quarry] Deployment fails to run from bastion - https://phabricator.wikimedia.org/T437626#12309109 (10dcaro) →14Duplicate dup:03T395238 [16:34:20] 10Quarry, 06tools-platform-team: quarry deployment fails with S3 403 - https://phabricator.wikimedia.org/T395238#12309111 (10dcaro) [16:34:33] 10Quarry, 06tools-platform-team: quarry deployment fails with S3 403 - https://phabricator.wikimedia.org/T395238#12309115 (10dcaro) [16:34:42] 06cloud-services-team, 10Quarry, 06tools-platform-team, 06RoadToWiki, and 3 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12309114 (10dcaro) [16:35:29] 06cloud-services-team, 10Quarry, 06tools-platform-team, 06RoadToWiki, and 3 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12309118 (10dcaro) 05Open→03In progress [16:35:31] 06cloud-services-team, 10Quarry, 06tools-platform-team, 06RoadToWiki, and 3 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12309119 (10dcaro) 05In progress→03Stalled [16:37:06] 10Cloud Services Proposals, 06tools-platform-team: Decision request - tools-platform internal review workflow - https://phabricator.wikimedia.org/T437573#12309132 (10dcaro) @aputhin If I understood your comments today, you are taking this one over? [16:39:12] 10Quarry, 06tools-platform-team, 07good first task, 13Patch-For-Review: Add nofollow attribute in some links to prevent bots from following unnecessary ones - https://phabricator.wikimedia.org/T197488#12309137 (10dcaro) @Osps7 thanks! there's some changes requested in the PR [16:39:50] 10Quarry, 06tools-platform-team, 07good first task, 13Patch-For-Review: Add nofollow attribute in some links to prevent bots from following unnecessary ones - https://phabricator.wikimedia.org/T197488#12309141 (10dcaro) [16:39:53] 10Quarry, 06tools-platform-team: quarry deployment fails with S3 403 - https://phabricator.wikimedia.org/T395238#12309142 (10dcaro) [16:41:56] (03open) 10dcaro: d/changelog: bump to 0.0.29 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/152 [16:42:35] (03update) 10dcaro: d/changelog: bump to 0.0.29 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/152 (https://phabricator.wikimedia.org/T435793) [16:43:11] 10Toolforge, 06tools-platform-team: [builds-cli] remove wrong help message for `--ref` - https://phabricator.wikimedia.org/T435793#12309159 (10dcaro) Deploying [16:43:26] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-cli [16:44:28] (03update) 10dcaro: cli: preserve option separators for subcommands [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/74 (https://phabricator.wikimedia.org/T370184) (owner: 10mahveotm) [16:45:04] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli [16:45:22] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-cli [16:47:54] 06tools-infrastructure-team: Track toolforge virtualization footprint: ram/cores/instance &c - https://phabricator.wikimedia.org/T436275#12309189 (10Andrew) Here is a simple monochromatic dashboard showing project usage vs. total deployment usage https://grafana-rw.wikimedia.org/d/antxhhs/ [16:49:31] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli [16:49:34] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Complete upgrading WMCS bare metal hosts to Trixie - https://phabricator.wikimedia.org/T375217#12309195 (10bking) [16:49:41] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12309191 (10bking) 05Open→03Resolved The above reimage was successful, and I... [16:49:46] (03approved) 10dcaro: d/changelog: bump to 0.0.29 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/152 (https://phabricator.wikimedia.org/T435793) [16:49:51] (03merge) 10dcaro: d/changelog: bump to 0.0.29 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/152 (https://phabricator.wikimedia.org/T435793) [16:50:51] 10Toolforge, 06tools-platform-team: [builds-cli] remove wrong help message for `--ref` - https://phabricator.wikimedia.org/T435793#12309220 (10dcaro) 05In progress→03Resolved [16:54:01] 10Toolforge, 06tools-platform-team: [components-cli] releases CI job stopped working - https://phabricator.wikimedia.org/T437624#12309237 (10dcaro) Builds-cli worked :/ https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/jobs/957050 [17:05:16] 10Tool-phab-ban, 10Toolforge, 06tools-platform-team, 07Sustainability (Incident Followup): Intermittent timeouts on Phab-ban tool - https://phabricator.wikimedia.org/T431967#12309284 (10dcaro) 05Open→03Resolved a:03dcaro @RhinosF1 this does not seem to be an issue anymore, I'll optimistically clo... [17:09:39] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 07Incident Severity 3, 07Wikimedia-Incident: 2026-07-11 Incident - Toolforge haproxies flapping - https://phabricator.wikimedia.org/T431946#12309298 (10dcaro) [17:10:01] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 07Incident Severity 3, 07Wikimedia-Incident: 2026-07-11 Incident - Toolforge haproxies flapping - https://phabricator.wikimedia.org/T431946#12309300 (10dcaro) @MLechvien-WMF Not sure if that's enough, linked to the document with more de... [17:10:32] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team, 06Tech-Docs-Team, 07OKR-Work: Fix description and example issues for page, revision, search, and file paths in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428153#12309301 (10aaron) [17:16:27] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [17:16:35] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [17:16:42] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [17:17:25] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [17:33:13] 10Tool-Pageviews: Add PagePile as a source for Massviews - https://phabricator.wikimedia.org/T437011#12309334 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [17:42:12] 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad, 06SRE: Physical move - https://phabricator.wikimedia.org/T437631 (10VRiley-WMF) 03NEW [17:42:38] 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad, 06SRE: Physical move - https://phabricator.wikimedia.org/T437631#12309372 (10VRiley-WMF) 05Open→03Resolved This has been completed. [17:46:18] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [17:46:42] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [19:08:44] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12309680 (10BLiviero-WMF) Hi Peter, thank you for the details! Is there also any kind of view that shows how... [22:09:13] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-mwlog02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436469#12310236 (10Southparkfan) 05In progress→03Resolved a:03Southparkfan https://gerrit.wikimedia.org/r/c/operations... [22:10:48] FIRING: PuppetFailure: Puppet has failed on clouddb1024:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [22:14:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [22:14:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [22:14:28] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [22:14:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [22:14:44] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [22:14:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [22:15:25] FIRING: [2x] SystemdUnitFailed: wmf-pt-kill@s4.service on clouddb1024:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:15:52] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [22:15:54] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [22:17:27] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [22:17:28] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [23:14:14] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16