[01:33:25] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:41:34] FIRING: DiskSpace: Disk space serpens:9100:/ 6.567% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=serpens - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:11:41] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:11:41] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [05:34:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11422574 (10Papaul) @ayounsi @cmooney please see below the steps to replace the loopback IPs on cr3/4-ulsfo and mr1-ulsfo If all this looks good, I will setup... [06:41:49] FIRING: DiskSpace: Disk space serpens:9100:/ 6.442% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=serpens - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:11:34] RESOLVED: DiskSpace: Disk space serpens:9100:/ 3.755% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=serpens - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:24:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11422756 (10ayounsi) Awesome, thx! The loopbacks are also in Puppet : https://github.com/search?q=repo%3Awikimedia%2Foperations-puppet%20198.35.26.193&type=co... [12:02:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Remove lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11423243 (10cmooney) >>! In T405499#11411590, @VRiley-WMF wrote: > Hey @cmooney It has been reused for that purpose, however it's still being worked on... [12:05:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: ssw1-d8-eqiad cross-rack links incorrect in Netbox - https://phabricator.wikimedia.org/T411480 (10cmooney) 03NEW p:05Triage→03Medium [12:51:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11423398 (10cmooney) @papaul as @ayounsi mentions you need to change it in puppet where it is also. Principally to change what IPs the hosts doing BGP are goi... [13:45:48] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D servers need to boot/reimage in UEFI mode - https://phabricator.wikimedia.org/T410910#11423686 (10cmooney) 05Open→03Resolved Thanks to the awesome work of @jhathaway this is no longer a requirement. We can use `--no82` with a host in BIOS... [15:31:40] 10netops, 07Puppet, 06Infrastructure-Foundations, 06serviceops: network::constants::mw_appserver_networks is out of date (or named poorly?) - https://phabricator.wikimedia.org/T411508 (10taavi) 03NEW [15:41:24] 07Puppet, 06serviceops: network::constants::mw_appserver_networks is out of date (or named poorly?) - https://phabricator.wikimedia.org/T411508#11424250 (10taavi) [15:49:49] 07Puppet, 06serviceops: network::constants::mw_appserver_networks is out of date (or named poorly?) - https://phabricator.wikimedia.org/T411508#11424318 (10Clement_Goubert) p:05Triage→03Low Probably a good task to pair on with @Blake or @jasmine_ on tracking down what's used where in puppet. [16:12:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11424434 (10BCornwall) @cmooney Yes, that looks good to me. We can still go for Dec 3 -... [16:13:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11424463 (10BCornwall) @cmooney Yes, that looks good to me. We can still go for Dec 4 -... [16:59:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11424827 (10Papaul) @ayounsi @cmooney thanks for the feedback. [17:02:33] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11424870 (10Papaul) @ssingh We are planning on doing the first phase(loopback IP change on core routers and management router) of the ULSFO refresh next week D... [17:21:37] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: ssw1-d8-eqiad cross-rack links incorrect in Netbox - https://phabricator.wikimedia.org/T411480#11425017 (10VRiley-WMF) 05Open→03Resolved Updated cable paths for the new switches in D8 to E1 and F1 [17:29:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Remove lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11425063 (10VRiley-WMF) 05Open→03In progress removing and updating cables [17:41:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Remove lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11425144 (10VRiley-WMF) 05In progress→03Resolved a:03VRiley-WMF this is completed [17:42:17] 10SRE-tools, 06Infrastructure-Foundations, 06serviceops, 13Patch-For-Review: Add a --rack flag to sre.k8s.pool-depool-node - https://phabricator.wikimedia.org/T410537#11425151 (10MLechvien-WMF) 05Open→03Resolved [18:41:56] 07Puppet, 06cloud-services-team, 10Horizon: Allow providing a commit message for hieradata changes - https://phabricator.wikimedia.org/T250623#11425504 (10taavi) p:05Triage→03Low [18:46:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11425543 (10ssingh) >>! In T408892#11424869, @Papaul wrote: > @ssingh We are planning on doing the first phase(loopback IP change on core routers and managemen... [23:24:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11426444 (10Papaul) @ssingh yes we have to depool the site, yes 10 AM CT