[04:09:38] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-c2-eqiad BGP maintenance/ Tuesday 17th at 9:30 CDT - https://phabricator.wikimedia.org/T420158 (10Papaul) 03NEW [04:16:11] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-c7-eqiad BGP maintenance/ Thursday 19th at 10:00 am CDT - https://phabricator.wikimedia.org/T420159 (10Papaul) 03NEW [07:25:12] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-c2-eqiad BGP maintenance/ Tuesday 17th at 9:30 CDT - https://phabricator.wikimedia.org/T420158#11712110 (10ayounsi) ` an-master1003: skipping host (Make sure the redundant master is active.) an-worker1220: skipping host (no depool... [07:44:13] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-c7-eqiad BGP maintenance/ Thursday 19th at 10:00 am CDT - https://phabricator.wikimedia.org/T420159#11712154 (10ayounsi) ` alert1002: Couldn't get or parse depool Hiera key an-worker1151: skipping host (no depool needed) an-worker... [07:57:36] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-c2-eqiad BGP maintenance/ Tuesday 17th at 9:30 CDT - https://phabricator.wikimedia.org/T420158#11712178 (10MoritzMuehlenhoff) [08:09:25] FIRING: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:14:25] RESOLVED: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:36:29] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-c2-eqiad BGP maintenance/ Tuesday 17th at 9:30 CDT - https://phabricator.wikimedia.org/T420158#11712298 (10taavi) [08:36:59] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-c7-eqiad BGP maintenance/ Thursday 19th at 10:00 am CDT - https://phabricator.wikimedia.org/T420159#11712299 (10taavi) [09:22:32] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Eqiad: lsw1-c2-eqiad BGP maintenance/ Tuesday 17th at 9:30 CDT - https://phabricator.wikimedia.org/T420158#11712410 (10Gehel) [09:22:41] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Eqiad: lsw1-c7-eqiad BGP maintenance/ Thursday 19th at 10:00 am CDT - https://phabricator.wikimedia.org/T420159#11712412 (10Gehel) [09:49:07] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Eqiad: lsw1-c2-eqiad BGP maintenance/ Tuesday 17th at 9:30 CDT - https://phabricator.wikimedia.org/T420158#11712621 (10cmooney) Can we hold off on any work related to this? I am planning to dr... [09:49:26] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Eqiad: lsw1-c7-eqiad BGP maintenance/ Thursday 19th at 10:00 am CDT - https://phabricator.wikimedia.org/T420159#11712623 (10cmooney) Can we hold off on any work related to this? I am planning... [09:53:04] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Nokia SR-Linux DHCP Relay Bug - https://phabricator.wikimedia.org/T411054#11712668 (10cmooney) [10:09:36] 07Puppet, 06collaboration-services, 10Gerrit: Edit puppet-merge to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org? - https://phabricator.wikimedia.org/T420184 (10ABran-WMF) 03NEW [11:23:41] 10netbox, 06Infrastructure-Foundations, 07Regression: after logging into Netbox, NDAs see an empty dashboard - https://phabricator.wikimedia.org/T404494#11713040 (10SLyngshede-WMF) 05In progress→03Resolved [11:35:13] o/ still looking for reviews of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1212097 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1212097 [12:08:40] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Nokia SR-Linux DHCP Relay Bug - https://phabricator.wikimedia.org/T411054#11713232 (10cmooney) >>! In T411054#11696665, @BTullis wrote: > Will all of the switches in rows C & D be getting this configuration change? Yes we need to fix it on a... [14:05:38] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11713777 (10ssingh) @Papaul / @ayounsi : Patches should be ready for this. Any preference for the day of (this) week for when we should do this? [14:07:29] 10Mail, 06Infrastructure-Foundations: Allow IT Services to view inbound email logs - https://phabricator.wikimedia.org/T419906#11713781 (10jhathaway) >>! In T419906#11710005, @taavi wrote: > If this data is too sensitive for our normal Logstash instance, why are we fine shipping them to a yet another third par... [14:08:25] 10Mail, 06Infrastructure-Foundations: Allow IT Services to view inbound email logs - https://phabricator.wikimedia.org/T419906#11713789 (10jhathaway) [14:19:36] 10Mail, 06Infrastructure-Foundations: Allow IT Services to view inbound email logs - https://phabricator.wikimedia.org/T419906#11713872 (10jhathaway) p:05Triage→03Medium [14:22:39] 10SRE-tools, 06Infrastructure-Foundations, 06serviceops-radar: Add --min-uptime to cookbooks - https://phabricator.wikimedia.org/T419967#11713899 (10elukey) p:05Triage→03Low [14:27:53] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Eqiad: lsw1-c2-eqiad BGP maintenance/ Tuesday 17th at 9:30 CDT - https://phabricator.wikimedia.org/T420158#11713919 (10cmooney) p:05Triage→03Low [14:28:02] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Eqiad: lsw1-c7-eqiad BGP maintenance/ Thursday 19th at 10:00 am CDT - https://phabricator.wikimedia.org/T420159#11713920 (10cmooney) p:05Triage→03Low [14:55:43] Hi there, I was wondering what procedure I could follow for an informal new service deployment on baremetal security review? [14:56:29] well, more than security it is good practices, as I usually have outaded information on puppet version, firewall, etc. [15:07:18] just add me to reviwer in Gerrit or Phabricator and I can have a look [15:10:22] sure [15:10:27] thank you [15:48:38] 10Mail, 06Infrastructure-Foundations, 10Observability-Logging: Allow IT Services to view inbound email logs - https://phabricator.wikimedia.org/T419906#11714299 (10jhathaway) [18:07:25] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:23:58] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11715178 (10Papaul) @ssingh thank you let me get back with you tomorrow. I have to double check some things in Netbox. [22:07:40] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:30:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11716213 (10Papaul) Removed interface et-0/0/1.1221 from both routers and cleanup all reference for sandbox1-ulsfo in Netbox ` - unit 1221 { - descr... [22:33:53] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11716224 (10Papaul) @ssingh I double check all for the new prefix 198.35.26.224/27 in Netbox all looks good. You can make your changes any time. Plea... [22:44:19] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11716272 (10Papaul) [22:46:03] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Eqiad: lsw1-c2-eqiad BGP maintenance/ Tuesday 17th at 9:30 CDT - https://phabricator.wikimedia.org/T420158#11716280 (10colewhite) [22:52:24] 10netops, 06Infrastructure-Foundations, 06SRE: Update esams network pop diagrams - https://phabricator.wikimedia.org/T368084#11716296 (10Papaul) 05Open→03Resolved Both diagrams for esams are now up to date. Closing this.