[01:48:40] FIRING: SystemdUnitFailed: wmf_auto_restart_kerberos_rsync.service on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:11:41] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [03:14:40] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [04:11:41] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [05:48:40] FIRING: SystemdUnitFailed: wmf_auto_restart_kerberos_rsync.service on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:52:09] 10netops, 06Infrastructure-Foundations: asw1-b12-drmrs stopped reporting metrics - https://phabricator.wikimedia.org/T413181#11581852 (10ayounsi) We're currently troubleshooting why we can't see troubleshooting logs. But it can maybe be the root cause for the metrics issues. **TL;DR; we should upgrade to 23.4... [06:56:39] 10netops, 06Infrastructure-Foundations: drmrs: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416441 (10ayounsi) 03NEW [06:59:29] 10netops, 06Infrastructure-Foundations: magru: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416442 (10ayounsi) 03NEW [07:04:23] 10netops, 10Cloud-Services, 06Infrastructure-Foundations: codfw: Upgrade cloudsw1-b1-codfw (2026) - https://phabricator.wikimedia.org/T416443 (10ayounsi) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/... [07:06:25] 10netops, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444 (10ayounsi) 03NEW [07:06:39] 10netops, 06Infrastructure-Foundations: drmrs: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416441#11581979 (10ayounsi) [07:06:40] 10netops, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#11581980 (10ayounsi) [07:07:32] 10netops, 06Infrastructure-Foundations: asw1-b12-drmrs stopped reporting metrics - https://phabricator.wikimedia.org/T413181#11581996 (10ayounsi) [07:07:34] 10netops, 06Infrastructure-Foundations: drmrs: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416441#11581997 (10ayounsi) [07:08:04] 10netops, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#11581999 (10ayounsi) [07:08:06] 10netops, 06Infrastructure-Foundations: magru: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416442#11582000 (10ayounsi) [07:08:08] 10netops, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#11582001 (10ayounsi) [07:09:36] 10netops, 06Infrastructure-Foundations, 06tools-infrastructure-team: codfw: Upgrade cloudsw1-b1-codfw (2026) - https://phabricator.wikimedia.org/T416443#11582002 (10ayounsi) [07:09:47] 10netops, 06Infrastructure-Foundations, 06tools-infrastructure-team: codfw: Upgrade cloudsw1-b1-codfw (2026) - https://phabricator.wikimedia.org/T416443#11582003 (10ayounsi) [07:09:49] 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#11582004 (10ayounsi) [07:14:40] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [07:49:58] ^ fixing the wmf_auto_restart_kerberos_rsync.service alert [08:03:25] RESOLVED: SystemdUnitFailed: wmf_auto_restart_kerberos_rsync.service on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:02:35] 10netops, 06Infrastructure-Foundations: esams: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416450 (10ayounsi) 03NEW [09:40:55] 10netops, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#11582323 (10ayounsi) [09:40:56] 10netops, 06Infrastructure-Foundations: esams: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416450#11582322 (10ayounsi) [09:41:26] 10netops, 06Infrastructure-Foundations: esams: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416450#11582324 (10ayounsi) [09:41:28] 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#11582325 (10ayounsi)