[07:52:29] 10SRE-tools, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Fix Pypi twine setup for pywmflib - https://phabricator.wikimedia.org/T430620#12074632 (10elukey) 05Open→03Resolved a:03elukey Ok this was a pebcak due to me not knowing how `bdist_wheel` works. I thought I needed to add the `w... [07:53:50] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Deploy wmflib 3.0.0 to production - https://phabricator.wikimedia.org/T430552#12074644 (10elukey) 3.0.0 deployed fleetwide, and 3.1.0 is ready to go. @MoritzMuehlenhoff we can do the 3.1.0 rollout next week, hopefully this will give enough time to people to f... [08:17:55] 10netops, 06Infrastructure-Foundations, 06Traffic, 06cloud-services-team (FY2025/2026-Q3-Q4), 13Patch-For-Review: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12074704 (10ayounsi) This service is using LVS with IPIP so I don't unders... [08:18:17] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Deploy wmflib 3.0.0 to production - https://phabricator.wikimedia.org/T430552#12074705 (10MoritzMuehlenhoff) >>! In T430552#12074635, @elukey wrote: > 3.0.0 deployed fleetwide, and 3.1.0 is ready to go. @MoritzMuehlenhoff we can do the 3.1.0 rollout next week... [08:22:36] 10netops, 10Cloud-VPS, 06Data-Platform-SRE, 10Data-Services, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12074721 (10ayounsi) @fgiunchedi maybe silly question but to make sure we're preserving public IPs, why can't we use the same I... [12:37:53] 10netops, 06Infrastructure-Foundations: GSHUT (and other?) community matching/actions not working on SR-Linux - https://phabricator.wikimedia.org/T430810#12075732 (10Aklapper) [12:57:12] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12075843 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0e1e62e2-283a-46e8-8264-f9dc495aa361) set by... [13:31:11] FIRING: PfwCoreBGPDown: ... [13:31:11] Fundraising Firewall core BGP session down between pfw1-eqiad and cr1-eqiad (208.80.154.200) - group Production - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=pfw1-eqiad:9804&var-bgp_group=Production&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [13:35:33] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12075997 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c1065c24-e5fc-46f8-8e8c-c08a491f7877) set by... [13:36:22] RESOLVED: PfwCoreBGPDown: ... [13:36:28] Fundraising Firewall core BGP session down between pfw1-eqiad and cr1-eqiad (208.80.154.200) - group Production - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=pfw1-eqiad:9804&var-bgp_group=Production&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [13:37:08] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12076012 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c9fa9a87-4c81-4974-9c0a-6a8c8205b869) set by... [13:37:25] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:39:07] 10netops, 10Cloud-VPS, 06Data-Platform-SRE, 10Data-Services, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12076020 (10fgiunchedi) >>! In T411248#12074721, @ayounsi wrote: > @fgiunchedi maybe silly question but to make sure we're pres... [13:41:10] 10netops, 06Infrastructure-Foundations, 06SRE: GSHUT (and other?) community matching/actions not working on SR-Linux - https://phabricator.wikimedia.org/T430810#12076032 (10cmooney) [13:57:25] RESOLVED: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:02:25] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:03:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12076313 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=aab5730e-3995-411e-aa0c-e48fc42c7ad5) set by... [15:00:43] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: OOB IPV6 down - https://phabricator.wikimedia.org/T430599#12076680 (10Papaul) Information sent to DR [15:12:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: rack A8 maintenance 2026-07-01 10:00 am CT - https://phabricator.wikimedia.org/T429856#12076785 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c0d4701e-c375-434f-b42c-5115f14a1c21) set by pt1979@cumin1003 fo... [15:12:34] FIRING: [32x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:37:25] FIRING: [25x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:42:25] FIRING: [25x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:06:55] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12077048 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e7fbbad5-cec3-439b-bf2e-a0a0bbae2b6b) set by pt1979@cumin1003... [16:47:21] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12077348 (10FCeratto-WMF) This triggered the alert `ProbeDown: Service wikikube-ctrl2004...` - the probe indicated a downtime from 16:32:0... [16:51:45] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12077406 (10FCeratto-WMF) It impacted db2226 https://grafana.wikimedia.org/goto/bfqtb4uge6i9sf?orgId=default and db2203 https://grafana.wi... [18:58:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12078004 (10cmooney) Going to re-schedule this for next Wednesday July 8th at 14:00 UTC. Fairly confident things went ok... [19:28:02] 10netops, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE: Lumen transport eqiad codfw down July 2026 - https://phabricator.wikimedia.org/T430874 (10cmooney) 03NEW p:05Triage→03Medium [19:42:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:17:25] FIRING: [2x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:22:25] FIRING: [2x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed