[02:22:25] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:33:20] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12078721 (10Papaul) 05Open→03Resolved Complete [02:34:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12078723 (10Papaul) [02:35:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: rack A8 maintenance 2026-07-01 10:00 am CT - https://phabricator.wikimedia.org/T429856#12078724 (10Papaul) 05Open→03Resolved Complete [02:36:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12078726 (10Papaul) [06:22:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:08:01] 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909 (10ayounsi) 03NEW p:05Triage→03Medium [08:08:33] 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12079216 (10ayounsi) [08:08:37] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12079217 (10ayounsi) [08:10:19] 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Diffscan: investigate IPv6 support and explore other scanning tooling - https://phabricator.wikimedia.org/T265329#12079221 (10taavi) a:03taavi [08:10:57] 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Diffscan: investigate IPv6 support and explore other scanning tooling - https://phabricator.wikimedia.org/T265329#12079222 (10taavi) a:05taavi→03None [08:13:39] 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12079236 (10ayounsi) [08:25:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12079304 (10ayounsi) [08:26:21] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12079312 (10ayounsi) [08:53:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12079442 (10ayounsi) [09:12:34] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918 (10ayounsi) 03NEW p:05Triage→03Medium The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phab... [09:22:24] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079602 (10ayounsi) @cmooney @Papaul can one of you take that upgrade ? And update the date/time to anything that suits you in that week. [09:24:21] topranks: https://github.com/aerleon/aerleon/releases/tag/1.17.0 [09:25:03] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079609 (10CWilliams-WMF) [09:25:22] XioNoX: hah nice! [09:25:31] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079614 (10CWilliams-WMF) [09:25:40] when I looked at the previous release cadence yesterday I wasn't too confident it would be quick [09:27:30] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12079630 (10ayounsi) [09:27:41] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12079635 (10ayounsi) [09:27:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12079637 (10ayounsi) [09:28:05] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079636 (10ayounsi) [09:28:51] already on https://pypi.org/project/aerleon/ too [09:29:41] nice, I'll take a look a cutting a new Homer release [09:30:09] I guess I just re-run the process, do I need to increment the version number? [09:30:21] no [09:30:45] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079648 (10CWilliams-WMF) [09:30:59] topranks: start at "Now generate the new wheels" from https://wikitech.wikimedia.org/wiki/Homer#In_the_homer-deploy_repository [09:31:12] ok cool thanks [09:31:20] and you can check the version in the frozen-requirements.txt [09:42:22] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079752 (10Marostegui) db2160 needs just a downtime. dbproxy will fail during the maintenance (just irc notification) and they'll recover on their... [09:49:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12079790 (10ayounsi) [09:56:49] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12079820 (10ayounsi) [10:00:26] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12079859 (10ayounsi) [10:00:43] 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12079866 (10ayounsi) a:03ayounsi [10:04:13] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079891 (10cmooney) a:03cmooney [10:06:12] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079906 (10cmooney) [10:18:26] 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12079937 (10jcrespo) backup2014: should be ok without depooling for a short period ms-backup2004 will require manual stopping of backup processes... [10:19:01] 10netops, 10Cloud-VPS, 06Data-Platform-SRE, 10Data-Services, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12079944 (10fgiunchedi) Ok so I spent some time getting a better understanding of NFS and how Linux clients reacts to a server... [10:20:36] 10SRE-tools, 06Infrastructure-Foundations: Cookbook for rack depool - https://phabricator.wikimedia.org/T327300#12079954 (10ayounsi) [10:22:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:56:24] o/, I have a few packages that need to be getting built for bookworm and aren't. Would anyone be up for showing me how to fix that? [10:57:05] (namely python3-imagecatalog and charlie, blocking deployment server upgrade) [10:57:58] sure, I need to wrap some puppet server reboots now, will have a look when I'm done [10:58:10] thank you! [12:23:52] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#12080468 (10cmooney) [12:27:05] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12080487 (10cmooney) [12:52:25] RESOLVED: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:53:25] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:06:25] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026) - https://phabricator.wikimedia.org/T429386#12080641 (10cmooney) 05Resolved→03Open Probably a little premature to close. I've opened ticket 2026-0616-761841 to confirm exactly what Jun... [13:18:25] RESOLVED: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:20:25] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:25:27] 10netops, 10Cloud-VPS, 06Data-Platform-SRE, 10Data-Services, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12080727 (10ayounsi) >>! In T411248#12076020, @fgiunchedi wrote: >>>! In T411248#12074721, @ayounsi wrote: >> @fgiunchedi maybe... [14:06:45] 10netops, 06Infrastructure-Foundations, 06SRE: Blackbox probe for TLS cert expriy failing on multiple eqiad SR-Linux nodes - https://phabricator.wikimedia.org/T429242#12081032 (10cmooney) Nokia came back to advise this is a known bug in 24.10.4. ` Hello Cathal, The gRPC server stops responding on SR-Linux?... [14:48:44] 10netops, 06Infrastructure-Foundations, 06Traffic, 06cloud-services-team (FY2025/2026-Q3-Q4): LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12081268 (10cmooney) I merged the patch to add the lvs public IPs to the healtcheck list now, an... [14:54:04] 10netops, 06Data-Persistence, 06DC-Ops, 06Infrastructure-Foundations, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12081319 (10Gehel) [14:57:19] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12081351 (10Gehel) [15:22:37] 10CAS-SSO, 10Cloud-VPS, 06tools-infrastructure-team, 07Security: Allowed callback URL for keystone CAS service is too wide - https://phabricator.wikimedia.org/T430957#12081558 (10taavi) 05Open→03Resolved a:03taavi [17:20:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:42:24] 10netops, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE: Lumen transport eqiad codfw down July 2026 - https://phabricator.wikimedia.org/T430874#12082394 (10VRiley-WMF) Swapped out the fiber and it looks like it came back up [17:43:42] 10netops, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE: Lumen transport eqiad codfw down July 2026 - https://phabricator.wikimedia.org/T430874#12082397 (10cmooney) 05Open→03Resolved a:03cmooney @VRiley-WMF thanks! Yep traffic back on the link looks good now cheers. ` cmooney@re... [21:20:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed