[03:19:16] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11776183 (10Papaul) @SLyngshede-WMF thank you very much. [03:25:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11776198 (10Papaul) [03:55:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#11776217 (10Papaul) @VRiley-WMF @Jclark-ctr when you are next onsite, can you please look for 1 QFX-SFP-1GE-T and plug it into mr1-eqiad ge-0/0/7? Thank you https://www.fs.... [08:11:25] 10netops, 06Infrastructure-Foundations, 06SRE: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421704#11776508 (10MLechvien-WMF) [10:35:25] 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 4 others: Increase visibility of kubernetes network status - https://phabricator.wikimedia.org/T356877#11777204 (10JMeybohm) [13:02:25] FIRING: SystemdUnitFailed: prometheus-nginx-exporter.service on urldownloader1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:12:25] FIRING: [2x] SystemdUnitFailed: prometheus-nginx-exporter.service on urldownloader1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:22:25] ^ these are leftovers from the early hcaptcha deploy, I'm removing promtheus-nginx-exporter frm these [13:27:25] RESOLVED: [2x] SystemdUnitFailed: prometheus-nginx-exporter.service on urldownloader1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:47:00] jhathaway: hey. all of the patches fixing validation issues for firewall::service still on ferm have now been fixed. the last PCC run was showing some issues with virtual resources and dependencies, I have sent two fixes which seem to work (1260721 and 1266205), but am all ears if you have better suggestions [15:09:36] 10netops, 06Infrastructure-Foundations: Create public vlan on eqiad and codfw pods E/F - https://phabricator.wikimedia.org/T422043 (10ayounsi) 03NEW [15:56:18] 10netops, 06Infrastructure-Foundations: Create public vlan on eqiad and codfw pods E/F - https://phabricator.wikimedia.org/T422043#11778904 (10cmooney) If we are going to have one public-enabled rack per "pod" then should we not have just one vlan assigned for codfw row E/F (and then one also for a/b and c/d)?... [23:25:10] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: mr1-eqiad: move from OSPF to BGP - https://phabricator.wikimedia.org/T421238#11780563 (10Papaul) All the BGP sessions are up ` mr1-eqiad# run show bgp summary group Production Threading mode: BGP I/O Default eBGP mode: advertise - accept, receive... [23:26:38] 10netops, 06DC-Ops, 06Infrastructure-Foundations: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#11780565 (10Papaul) @VRiley-WMF any update on this: I will like to move the OOB tomorrow during the maintenance window. Thanks