[00:05:25] RESOLVED: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:32:07] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12068530 (10Papaul) @BCornwall about 40 minutes from adding the image to reboot [04:14:17] good morning, I sent a patch allowing the webproxies to tunnel traffic to port 25000, that is the port used by WMCS OpenStack Keystone API : https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306161/2/hieradata/common/profile/installserver/proxy.yaml [04:15:38] we did a similar thing previously to reach out to port 6443, the port to reach out to OpenStack Magnum API (to drive Kubernertes clusters there) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1174842 [05:24:42] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: OOB IPV6 down - https://phabricator.wikimedia.org/T430599#12068734 (10ayounsi) FYI it works from my home internet: ` laptop:~$ ping -6 mr1-ulsfo.oob.wikimedia.org PING mr1-ulsfo.oob.wikimedia.org (2607:fb58:9000:7::2) 56 data by... [07:02:42] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Deploy wmflib 3.0.0 to production - https://phabricator.wikimedia.org/T430552#12068877 (10elukey) Changelog here: https://github.com/wikimedia/operations-software-pywmflib/blob/master/CHANGELOG.rst I am going to rollout the changes for Debian 11, then I'll d... [07:07:25] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Deploy wmflib 3.0.0 to production - https://phabricator.wikimedia.org/T430552#12068878 (10MoritzMuehlenhoff) Sounds good! [07:20:17] !log upgrade all bullseye hosts to pywmflib 3.0 - T430552 [07:20:17] elukey: Not expecting to hear !log here [07:20:18] T430552: Deploy wmflib 3.0.0 to production - https://phabricator.wikimedia.org/T430552 [07:21:19] yep you are right [08:21:36] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Fix Pypi twine setup for pywmflib - https://phabricator.wikimedia.org/T430620 (10elukey) 03NEW [08:24:10] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Fix Pypi twine setup for pywmflib - https://phabricator.wikimedia.org/T430620#12069181 (10Volans) The build on `build2004` is only for the debian package, the PyPi release is all based on the local artifacts created locally. The reported version `3.1.1.dev0+g... [08:31:00] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Fix Pypi twine setup for pywmflib - https://phabricator.wikimedia.org/T430620#12069199 (10elukey) >>! In T430620#12069181, @Volans wrote: > The build on `build2004` is only for the debian package, the PyPi release is all based on the local artifacts created l... [09:12:14] 10netops, 10Cloud-VPS, 06Data-Platform-SRE, 10Data-Services, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12069417 (10fgiunchedi) [11:56:11] 10netops, 06Infrastructure-Foundations, 06Traffic, 06cloud-services-team (FY2025/2026-Q3-Q4): LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651 (10fgiunchedi) 03NEW [11:57:55] 10netops, 06Infrastructure-Foundations, 06Traffic, 06cloud-services-team (FY2025/2026-Q3-Q4), 13Patch-For-Review: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12070204 (10fgiunchedi) https://gerrit.wikimedia.org/r/c/operations/puppet... [12:07:57] 10netops, 06Infrastructure-Foundations, 06SRE: Blackbox probe for TLS cert expriy failing on multiple eqiad SR-Linux nodes - https://phabricator.wikimedia.org/T429242#12070253 (10cmooney) lsw1-d2-eqiad has been left in a "bad" state and I opened 05827811 against it with Nokia. [12:25:49] 10netops, 06Infrastructure-Foundations, 06Traffic, 06cloud-services-team (FY2025/2026-Q3-Q4), 13Patch-For-Review: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12070301 (10MoritzMuehlenhoff) All the IPs allowed by LOAD_BALANCER_HEALTH... [12:26:11] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 06SRE: Preserve SSH host key when re-imaging hosts - https://phabricator.wikimedia.org/T129180#12070303 (10elukey) 05Open→03Declined Declining, please re-open if needed! [12:29:53] 10SRE-tools: Spicerack cookbooks TODO list - https://phabricator.wikimedia.org/T203943#12070317 (10elukey) 05Open→03Resolved a:03elukey Resoling this, since most of the work is done and only a couple of tasks are left to do. [12:31:05] 10SRE-tools, 06Infrastructure-Foundations, 06SRE, 07Python3-Porting: Puppet tox: properly lint both Py2 and Py3 files - https://phabricator.wikimedia.org/T184435#12070337 (10elukey) 05Open→03Resolved a:03elukey Pretty sure this is not an issue anymore, resolving, please re-open if I am mistaken. [12:41:18] 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, 06tools-infrastructure-team: Upgrade cloudsw1-e4-eqiad - https://phabricator.wikimedia.org/T429013#12070397 (10ayounsi) Unfortunately there is no easy way to perform the maintenance without impacting the devices connected to that switch. [12:44:16] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12070412 (10cmooney) I spoke to @VRiley-WMF about this and we are going to schedule the change for Wed July 1st starting a... [12:59:27] 10netops, 10Cloud-VPS, 06Data-Platform-SRE, 10Data-Services, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12070478 (10fgiunchedi) [13:09:38] 10netops, 10Cloud-VPS, 06Data-Platform-SRE, 10Data-Services, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12070530 (10fgiunchedi) I'll be starting the trials on `toolsbeta` and specifically `toolsbeta-test-k8s-worker-nfs-8.toolsbeta`... [13:33:03] 10netops, 06Infrastructure-Foundations, 06Traffic, 06cloud-services-team (FY2025/2026-Q3-Q4), 13Patch-For-Review: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12070626 (10cmooney) @fgiunchedi thanks for the task. I think @MoritzMueh... [15:09:25] FIRING: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:09:25] RESOLVED: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:10:22] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026) - https://phabricator.wikimedia.org/T429386#12073435 (10cmooney) 05Open→03Resolved [20:27:14] 10netops, 06Infrastructure-Foundations, 06SRE: Create alerting for saturation on sub-rated interfaces - https://phabricator.wikimedia.org/T374614#12073531 (10cmooney) We now have the //CoreRouterInterfaceDropPercent// alerts, which to an extent covers this scenario. It's not exactly the same, but we are in...