[04:45:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [05:00:25] FIRING: [2x] MirrorHighLag: Mirrors - /srv/mirrors/debian synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [06:00:25] FIRING: [2x] MirrorHighLag: Mirrors - /srv/mirrors/debian synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [06:09:25] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:25:09] 10netops, 06Infrastructure-Foundations, 06Traffic: Upgrade End Of Support Junos - https://phabricator.wikimedia.org/T390813#11166256 (10ayounsi) There is no particular rush, let's say before the end of 2025 ? [08:45:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/debian synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [09:04:28] 10Packaging, 10bacula, 10Data-Persistence-Backup, 10Infrastructure Security, 06Infrastructure-Foundations: Trixie bacula-fd package incompatible with our bacula installation - https://phabricator.wikimedia.org/T404114#11166342 (10MoritzMuehlenhoff) I built a forward port of Bacula 9.6.7 for trixie-wikime... [09:25:27] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: Updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11166358 (10ayounsi) > We need to allow port number 48 on the Nokias, but not port number 0 as they start from 1 We already (and lazily) do : `min_value=0, max_value=48` which... [09:45:04] 10Packaging, 10bacula, 10Data-Persistence-Backup, 10Infrastructure Security, 06Infrastructure-Foundations: Trixie bacula-fd package incompatible with our bacula installation - https://phabricator.wikimedia.org/T404114#11166448 (10jcrespo) Thank you, that was fast! I sent a command, it may take some time... [10:09:40] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:09:25] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:00:00] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: Updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11167203 (10ayounsi) [13:10:02] 10netops, 06Infrastructure-Foundations, 06Traffic: Upgrade End Of Support Junos - https://phabricator.wikimedia.org/T390813#11167264 (10ssingh) >>! In T390813#11166256, @ayounsi wrote: > There is no particular rush, let's say before the end of 2025 ? OK thanks for sharing, we will triage and plan accordingly! [15:08:18] 10Packaging, 10bacula, 10Data-Persistence-Backup, 10Infrastructure Security, 06Infrastructure-Foundations: Trixie bacula-fd package incompatible with our bacula installation - https://phabricator.wikimedia.org/T404114#11167992 (10jcrespo) The backup went through according to logs, so that's good. But I w... [15:22:46] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11168027 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=eb9214cb-2708-4519-b3c6-38d4f0f7cf7d) set by pt1979@cumin1002 for 1:00:00 o... [15:26:27] 10Packaging, 10bacula, 10Data-Persistence-Backup, 10Infrastructure Security, and 2 others: Trixie bacula-fd package incompatible with our bacula installation - https://phabricator.wikimedia.org/T404114#11168044 (10MoritzMuehlenhoff) >>! In T404114#11167992, @jcrespo wrote: > The backup went through accordi... [15:28:17] 10Packaging, 10bacula, 10Data-Persistence-Backup, 10Infrastructure Security, and 3 others: Trixie bacula-fd package incompatible with our bacula installation - https://phabricator.wikimedia.org/T404114#11168066 (10fgiunchedi) +SRE for visibility [16:18:08] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11168372 (10Papaul) 05Open→03Resolved mr1-eqsin and cr2/3-eqsin are now running BGP for the management network. Resolving this task. Thanks @ay... [17:01:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/debian synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [18:10:58] 10Packaging, 10bacula, 10Data-Persistence-Backup, 10Infrastructure Security, and 3 others: Trixie bacula-fd package incompatible with our bacula installation - https://phabricator.wikimedia.org/T404114#11169125 (10Dzahn) @jcrespo I tested a restore on people1005. Just selected 3 image files from my own hom... [18:35:52] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox: Updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11169333 (10cmooney) >>! In T404146#11166358, @ayounsi wrote: > We already (and lazily) do : `min_value=0, max_value=48` which was to accommodate both SON... [18:58:15] 10netops, 06Infrastructure-Foundations, 06SRE: Renumber link addressing to lsw1-e1-eqiad and lsw1-f1-eqiad - https://phabricator.wikimedia.org/T404248 (10cmooney) 03NEW p:05Triage→03Low [20:46:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/debian synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [21:50:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [21:55:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag