[06:13:15] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#12404668 (10ayounsi) ulsfo too, with {T434068} [09:02:35] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade to CAS 8.0.X - https://phabricator.wikimedia.org/T439509#12405129 (10Aklapper) [11:46:07] should we wait for new patches to land in spicerack (but I don't see much in https://gerrit.wikimedia.org/r/q/project:operations/software/spicerack+status:open ) or should we cut a new release for the netbox-next change? [12:09:13] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:43:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12406199 (10cmooney) >>! In T418018#12403321, @Jclark-ctr wrote: > @cmooney @RobH Finished A8 LSW, spine management, and SCS cabling. Updated the SCS... [13:45:33] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406205 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b92fad2c-3918-4020-9934-ce2871a5d804) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:03:46] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406857 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=973a1f0e-7b13-4b84-a662-c464cd8b1c39) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:04:02] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406858 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=3dad56c8-3d29-4732-8cdc-888cbff5cd56) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:05:00] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406878 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=5b1d769e-793d-41ab-9f6d-8ffbbf1b0f05) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:05:30] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406881 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=84bef5d3-d308-4bff-9956-cd6ef184cc99) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:06:24] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406887 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8edef278-f04e-4382-b245-c0737e83964a) set by pt1979@cumin1004 for 2:00:00 on 1 host(s) and their services with reason: J... [15:07:07] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406898 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=06d6d295-b237-4905-b49b-224383a6da43) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:07:33] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406900 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=559e6f53-818c-4947-be1b-1e17f06db0ca) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:08:25] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406912 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=08788b61-9f61-4307-acc1-9beec96b1b26) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:09:45] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406934 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=1dc6ec98-e477-4fe8-b41e-e03982222d6b) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:10:48] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406938 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8457b4c5-c815-4920-9fe5-0c1a08027db8) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:11:12] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406942 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7926f577-2f16-46c7-8af4-b515985fe43b) set by pt1979@cumin1004 for 2:00:00 on 2 host(s) and their services with reason: J... [15:48:02] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12407308 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cc8c6577-fecc-488f-ba8f-baf0394868ba) set by pt1979@cumin1004 for 1:00:00 on 3 host(s) and their services with reason: J... [15:48:32] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12407313 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=9feb482a-36e5-4ef0-bd31-e2381cbba1be) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:51:14] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12407331 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ef3a779c-6b51-41af-927e-15706021fa93) set by cmooney@cumin1004 for 2:00:00 on 4 host(s) and their services with reason:... [16:09:24] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:01:49] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12408084 (10Papaul) [17:02:49] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12408094 (10Papaul) Cr2-codfw Junos upgrade finished. Tout many steps to move traffic away from the router. Thanks @cmooney for the tips [18:04:13] RESOLVED: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:09:10] 10netops, 06Infrastructure-Foundations: eqsin-codfw HE packet loss/latency - oct 2026 - https://phabricator.wikimedia.org/T439947#12408474 (10wiki_willy) To close the loop, we received a credit back for $269.17