[00:20:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [00:45:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [04:44:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/debian synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [05:46:11] 10netbox, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:cr* router power not balance on all 4 PEM's - https://phabricator.wikimedia.org/T401937#11100913 (10ayounsi) a:03Papaul [08:44:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/debian synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [09:02:51] 10netops, 10Ganeti, 06Infrastructure-Foundations: magru: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T402372 (10ayounsi) 03NEW [09:07:54] 10netops, 10Ganeti, 06Infrastructure-Foundations, 13Patch-For-Review: magru: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T402372#11101245 (10ayounsi) vlan `sandbox1-b3-magru` deleted as well as its switch IRB. IP ranges converted to the `virtual-machines` role: https://netbox.wik... [09:12:09] 10netops, 10Ganeti, 06Infrastructure-Foundations, 13Patch-For-Review: magru: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T402372#11101278 (10ayounsi) [09:12:47] 10netops, 10Ganeti, 06Infrastructure-Foundations, 13Patch-For-Review: magru: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T402372#11101279 (10ayounsi) [09:12:53] 10netops, 10Ganeti, 06Infrastructure-Foundations, 13Patch-For-Review: magru: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T402372#11101280 (10ayounsi) [14:49:01] 10netops, 10Ganeti, 06Infrastructure-Foundations, 13Patch-For-Review: magru: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T402372#11102664 (10MoritzMuehlenhoff) p:05Triage→03Medium [14:52:25] FIRING: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:54:26] ^ we restarted gitlab (which fixed CI/CD pipelines) [14:57:25] RESOLVED: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:21:56] FIRING: MaxConntrack: Max conntrack at 80.97% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [15:26:55] RESOLVED: MaxConntrack: Max conntrack at 80.97% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [16:38:26] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11103043 (10cmooney) @papaul just FYI the diagram you did is now not accessible for other users for some reason [19:14:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: new structured cabling needed between cages to eqiad 2025/6 switch refresh - https://phabricator.wikimedia.org/T402432 (10cmooney) 03NEW p:05Triage→03Medium [19:33:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: new structured cabling needed between cages to eqiad 2025/6 switch refresh - https://phabricator.wikimedia.org/T402432#11103889 (10cmooney) [19:38:15] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: new structured cabling needed between cages to eqiad 2025/6 switch refresh - https://phabricator.wikimedia.org/T402432#11103905 (10cmooney) Ok so I spoke to traffic and while they are close to ditching the need for L2 adjacency... [19:49:07] 10Mail, 06Fundraising-Backlog, 10fundraising-tech-ops, 10Infrastructure Security, and 2 others: Re-opening our DMarcian Trial - https://phabricator.wikimedia.org/T394788#11103984 (10Jgreen) 05Open→03Resolved a:03Jgreen trial is in progress [22:26:25] FIRING: SystemdUnitFailed: update-tails-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:11:58] 07Puppet, 06SRE, 03Readers Essential Work 2025: Certain mobile devices are (possibly) not being redirected to our mobile site - https://phabricator.wikimedia.org/T388032#11104804 (10Jdlrobson-WMF) I'll keep an eye on this during the mobile domain switchover. [23:26:25] RESOLVED: SystemdUnitFailed: update-tails-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed