[03:11:05] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:11:05] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:52:56] FIRING: MaxConntrack: Max conntrack at 80.95% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [05:47:56] RESOLVED: MaxConntrack: Max conntrack at 80.65% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [06:36:46] 10netops, 06Infrastructure-Foundations, 06SRE: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11105337 (10ayounsi) Note we need to keep in mind that the main goal here is to move the mgmt routers to use BGP instead of OSPF. It's fine to do some light recabling if it m... [07:16:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: new structured cabling needed between cages to eqiad 2025/6 switch refresh - https://phabricator.wikimedia.org/T402432#11105392 (10ayounsi) [07:18:49] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: new structured cabling needed between cages to eqiad 2025/6 switch refresh - https://phabricator.wikimedia.org/T402432#11105396 (10ayounsi) Sounds good ! It would also be fine to route temporarily through the core routers depend... [13:27:34] 10netops, 06Infrastructure-Foundations, 06SRE: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11106403 (10Papaul) Understood [14:24:05] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [15:15:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [15:29:05] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [15:30:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [16:32:56] FIRING: MaxConntrack: Max conntrack at 81.99% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [16:37:55] RESOLVED: MaxConntrack: Max conntrack at 81.85% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [18:22:50] 10netops, 06Infrastructure-Foundations, 06SRE: Homer: Add Python modules to configure Nokia SR Linux switches - https://phabricator.wikimedia.org/T402577 (10cmooney) 03NEW p:05Triage→03Medium [18:22:59] 10netops, 06Infrastructure-Foundations, 06SRE: Homer: Add Python modules to configure Nokia SR Linux switches - https://phabricator.wikimedia.org/T402577#11108196 (10cmooney) [19:52:07] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad: row C/D switch refresh configuration task - https://phabricator.wikimedia.org/T402588 (10cmooney) 03NEW p:05Triage→03Medium [20:00:06] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Eqiad: row C/D switch refresh configuration task - https://phabricator.wikimedia.org/T402588#11108541 (10cmooney) [20:01:22] 10netops, 06Infrastructure-Foundations, 06SRE: codfw expansion: configure new Nokia switches in rows E/F - https://phabricator.wikimedia.org/T402590 (10cmooney) 03NEW p:05Triage→03Medium [20:01:38] 10netops, 06Infrastructure-Foundations, 06SRE: codfw expansion: configure new Nokia switches in rows E/F - https://phabricator.wikimedia.org/T402590#11108572 (10cmooney) [21:12:22] 07Puppet, 10Beta-Cluster-Infrastructure: /usr/local/bin/puppetserver-deploy-code emits scary looking error messages during a `git rebase` operation - https://phabricator.wikimedia.org/T397877#11108801 (10bd808) 05In progress→03Resolved [22:42:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/debian synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag