[00:05:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:05:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:05:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:51:53] 10netops, 06Infrastructure-Foundations, 06SRE: Blackbox probe for TLS cert expriy failing on multiple eqiad SR-Linux nodes - https://phabricator.wikimedia.org/T429242#12064314 (10ayounsi) 05Resolved→03Open Alerts are back for eqiad C/D (+spines) - https://alerts.wikimedia.org/?q=scope%3Dnetwork&q=alertna... [10:07:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12064630 (10cmooney) p:05Medium→03High [10:19:07] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: bird bfd session with 172.20.1.1 down - Bad packet from 172.20.1.1 - unknown session id - https://phabricator.wikimedia.org/T427202#12064697 (10cmooney) 05Open→03Resolved a:03cmooney Gonna close this one for now, if it be... [10:42:32] 10CAS-SSO, 06Infrastructure-Foundations, 10Phabricator, 06SRE: Add logout.d script for Phabricator - https://phabricator.wikimedia.org/T286904#12064836 (10MoritzMuehlenhoff) >>! In T286904#12041211, @Aklapper wrote: > Hi, I myself am not sure what to add apart from T286904#11091482. Please elaborate if any... [11:46:38] 10CAS-SSO, 06Infrastructure-Foundations: Logging out of idm does not log me out - https://phabricator.wikimedia.org/T392350#12065024 (10SLyngshede-WMF) Testing on IDM/Bitu is appears that the Python Social Auth library have no functionality which allows back channel logout. However newer versions of the librar... [12:05:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:10:48] FIRING: PuppetFailure: Puppet has failed on build2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [12:20:48] RESOLVED: PuppetFailure: Puppet has failed on build2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [12:55:06] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux: applying analytics-in acl to irb sub-interface blocks ARP - https://phabricator.wikimedia.org/T429499#12065297 (10cmooney) FWIW this behaviour is not evident on SR-Linux v25.10.1 (tested on lswtest-d8-eqiad). It seems to be yet another 24.x SR-Linux b... [12:59:17] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Terminal configuration for cookbooks - https://phabricator.wikimedia.org/T429129#12065322 (10MoritzMuehlenhoff) p:05Triage→03Low [13:41:25] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12065635 (10Papaul) [13:48:36] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12065694 (10Papaul) [13:51:39] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12065734 (10Papaul) [13:54:06] 10netops, 06Infrastructure-Foundations, 06SRE: Blackbox probe for TLS cert expriy failing on multiple eqiad SR-Linux nodes - https://phabricator.wikimedia.org/T429242#12065756 (10cmooney) >>! In T429242#12064314, @ayounsi wrote: > Alerts are back for eqiad C/D (+spines) - https://alerts.wikimedia.org/?q=scop... [14:52:16] 10CFSSL-PKI, 06Infrastructure-Foundations, 10Observability-Metrics: cfssl: export metricts from sql database - https://phabricator.wikimedia.org/T327768#12066309 (10LSobanski) 05Open→03Declined Not planning to work on this. [15:20:18] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Deploy wmflib 3.0.0 to production - https://phabricator.wikimedia.org/T430552 (10MoritzMuehlenhoff) 03NEW [15:20:28] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Deploy wmflib 3.0.0 to production - https://phabricator.wikimedia.org/T430552#12066517 (10MoritzMuehlenhoff) p:05Triage→03Medium [16:05:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:05:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:30:46] 10CFSSL-PKI, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 06SRE, 07Puppet (Puppet 7.0): Create dynamic CRL - https://phabricator.wikimedia.org/T340543#12068190 (10jhathaway) while investigating whether this task could be resolved, I found... We seem to be checking the CRL, based on the agent l... [22:00:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo: ULSFO: OOB IPV6 down - https://phabricator.wikimedia.org/T430599 (10Papaul) 03NEW [22:00:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo: ULSFO: OOB IPV6 down - https://phabricator.wikimedia.org/T430599#12068267 (10Papaul) p:05Triage→03Medium [23:40:34] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12068484 (10BCornwall) What is the duration of the maintenance?