[00:12:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[02:12:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[02:14:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:14:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:00:21] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166353 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a746a25c-706b-4c61-bdf4-214a1c78ff90) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their servi...
[09:12:04] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166372 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cce20762-fa19-4ada-b944-1a93ea8ed59f) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi...
[10:11:54] <wikibugs>	 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12166586 (10ayounsi)
[10:14:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:14:42] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166601 (10ayounsi) 05Open→03Resolved a:03ayounsi Done.
[11:00:52] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490 (10cmooney) 03NEW p:05Triage→03Medium
[11:03:34] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490#12166714 (10cmooney) Case 2026-0729-825879 created
[11:26:25] <wikibugs>	 10CFSSL-PKI, 06Infrastructure-Foundations, 10Cloud-VPS (Debian Bullseye Deprecation): Rebuild VMs in PKI cloud-vps project - https://phabricator.wikimedia.org/T405017#12166774 (10elukey) 05Open→03Resolved a:03elukey Old VMs deleted!
[13:31:29] <wikibugs>	 10netops, 06Infrastructure-Foundations, 07Sustainability (Incident Followup): Review thresholds for paging on DDoS alerts - https://phabricator.wikimedia.org/T431683#12167158 (10MLechvien-WMF)
[13:39:05] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12167187 (10cmooney)
[14:11:32] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12167393 (10Papaul)
[14:14:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:33:49] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12167497 (10jhathaway) >>! In T383047#12165518, @Perryprog wrote: Out of curiosity,...
[15:14:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[15:17:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[15:55:21] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12167963 (10jhathaway) >>! In T383047#12165505, @Scott_French wrote: > Very nice fin...
[16:28:28] <wikibugs>	 10SRE-tools, 06Infrastructure-Foundations, 06SRE: sre.ganeti.makevm: Create machine types - https://phabricator.wikimedia.org/T344972#12168164 (10LSobanski) p:05Medium→03Low
[16:31:02] <wikibugs>	 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations: Cookbook sre.hardware.upgrade-firmware fails to get firmwares from Dell's website - https://phabricator.wikimedia.org/T357756#12168189 (10LSobanski) Untagging #infrastructure-foundations. @wiki_willy could you take a look at updating the documentation ment...
[17:44:08] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12168699 (10cmooney) @VRiley to confirm what the next steps are here of most importance to unblock my side of the work:  # Cable the mgmt port on each of the...
[18:22:05] <wikibugs>	 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations: Cookbook sre.hardware.upgrade-firmware fails to get firmwares from Dell's website - https://phabricator.wikimedia.org/T357756#12168795 (10wiki_willy) a:03RobH Chatted with @RobH, who will follow up on this one.  Thanks, Willy
[19:17:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[19:19:31] <wikibugs>	 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations: Cookbook sre.hardware.upgrade-firmware fails to get firmwares from Dell's website - https://phabricator.wikimedia.org/T357756#12168996 (10RobH) I've made a first run at this to remove the wholly inaccurate info so its not there anymore but will keep this t...
[23:17:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[23:57:52] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12169750 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0fbc87b8-270e-465d-9d10-b229a17d4554) set by pt1979@cumin1003 for 2:00:00...