[06:05:48] <wikibugs>	 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12093069 (10MoritzMuehlenhoff)
[12:39:46] <wikibugs>	 10netops, 10Cloud-VPS, 06Data-Platform-SRE, 10Data-Services, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12094337 (10fgiunchedi) >>! In T411248#12084087, @fgiunchedi wrote: >>>! In T411248#12080727, @ayounsi wrote: >>>>! In T411248#...
[12:56:33] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026) - https://phabricator.wikimedia.org/T429386#12094364 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=99c0df32-80dc-453a-920f-5d3957387031) set by cmooney@cumin1003 for 1...
[12:57:13] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026) - https://phabricator.wikimedia.org/T429386#12094370 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8f5cefde-0060-47c2-8575-70bfa00d21f7) set by cmooney@cumin1003 for 1...
[12:59:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:23:08] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026) - https://phabricator.wikimedia.org/T429386#12094443 (10cmooney) Upgraded cr3-eqsin also as PIC needed bouncing.  And created some instructions to do the traffic drain without 'graceful-shu...
[13:59:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:32:36] <wikibugs>	 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, and 2 others: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12094907 (10Scott_French) Thanks @ayounsi - I'll aim to have this complete during my day today and will follow up here once it's safe to proceed...
[14:34:58] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: OOB IPV6 down - https://phabricator.wikimedia.org/T430599#12094934 (10Papaul) @cmooney see below email from DR thanks  ` From our investigation, we confirmed that the source network (2607:fb58:9000:7::/64) is reachable within ou...
[14:53:52] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: OOB IPV6 down - https://phabricator.wikimedia.org/T430599#12095011 (10cmooney) @papaul hmm it looks like it's working different now: ` cmooney@mr1-ulsfo> traceroute 2620:0:861:3:208:80:154:78 routing-instance oob-transit wait 2...
[19:32:46] <wikibugs>	 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12096752 (10ssingh) >>! In T430909#12096739, @Scott_French wrote: > **Status** >  > Alright, the (updated) procedure in T430909#12091874 is compl...