[04:08:43] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647 (10Papaul) 03NEW [04:09:30] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11695826 (10Papaul) p:05Triage→03High a:05cmooney→03ayounsi [06:11:20] 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#11695955 (10ayounsi) From JTAC: > I hope you are doing well. Our engineering team has found a fix for this behavior. However, the release you are running, 22.2, is already EoL... [06:16:00] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru <-> asw1-b3-magru link down March 2026 - https://phabricator.wikimedia.org/T418978#11695964 (10ayounsi) Awesome, thx!! [07:28:53] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696068 (10ayounsi) Using this opportunity to test my WIP rack depool cookbook (only in "show" mode). More info in {T327300} That's the current status of what... [07:43:02] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696088 (10ops-monitoring-bot) Draining ganeti1033.eqiad.wmnet of running VMs [07:43:40] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696089 (10MoritzMuehlenhoff) [08:21:16] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696205 (10MatthewVernon) [08:22:24] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696207 (10MatthewVernon) Can I check this is 15:00 UTC (particularly given daylight confusion...), please? Once it's done I'll check ms-be1091 [the frontends c... [08:26:28] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696225 (10MoritzMuehlenhoff) [08:30:10] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696229 (10ayounsi) >>! In T419647#11696205, @MatthewVernon wrote: > Can I check this is 15:00 UTC (particularly given daylight confusion...), please? Once it's... [08:52:32] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696297 (10MatthewVernon) Ah, I just put `10:00 EST` into `date`. You're probably right, but a confirmation would be helpful :) [09:32:58] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696410 (10MoritzMuehlenhoff) [10:16:14] 10netops, 06Infrastructure-Foundations: Nokia: implement maintenance mode - https://phabricator.wikimedia.org/T419673 (10ayounsi) 03NEW p:05Triage→03Medium [10:17:48] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696558 (10taavi) [10:23:06] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11696564 (10ayounsi) 05Open→03Resolved All done. [10:30:55] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696597 (10BTullis) [10:31:35] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11696605 (10BTullis) [10:46:19] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Nokia SR-Linux DHCP Relay Bug - https://phabricator.wikimedia.org/T411054#11696665 (10BTullis) Will all of the switches in rows C & D be getting this configuration change? I'm asking because I've got another host that is exhibiting a reimage... [11:07:29] 10CAS-SSO, 06cloud-services-team, 10Striker, 13Patch-For-Review: Use IDP for authentication in Striker - https://phabricator.wikimedia.org/T359554#11696763 (10Arendpieter) @taavi [[https://gerrit.wikimedia.org/r/c/labs/striker/+/1250537 | This is the second attempt]], where I made several different choices... [11:52:44] 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Cookbook for rack depool - https://phabricator.wikimedia.org/T327300#11697057 (10taavi) Should the `policy: local_command` option have a separate setting for a command for re-pooling the node? [13:35:03] 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Cookbook for rack depool - https://phabricator.wikimedia.org/T327300#11697389 (10ayounsi) yeah it's planned with `profile::server_pool` (and the same keys), focusing on the depool for now, especially for the `show` command. [14:31:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11697857 (10ssingh) Hi folks. I confirmed with Valentin that we don't need the public IPs, `pybal-high-traffic1-ulsfo.wikimedia.org` and `pybal-high-... [14:39:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11697929 (10ssingh) Sorry, @ayounsi reminded me that the main purpose of this task is to figure out what to do about the other public IPs. We will ne... [15:51:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11698383 (10ssingh) @Jgreen / @Dwisehaupt: `donate-lb.ulsfo.wikimedia.org` is the same IP as `text-lb.ulsfo.wikimedia.org` and that will change as pa... [16:24:22] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11698537 (10RLazarus) Service Ops triage here: Agreed there's nothing for us to do, thanks @ayounsi - untagging us. [16:27:51] moritzm, vgutierrez, hey, better to chat here, should we try a ncredir4003 kernel downgrade? [16:28:03] I don't think is the kernel TBH [16:28:03] or just a host reboot? [16:28:17] we have realservers using IPIP running 6.12.57 [16:28:49] so unless something got backported to 6.1.160 I don't see how that could be the culprit [16:34:03] we have ncredir hosts running on 6.1.140 and 6.1.147, there could be some 6.1-specific regression after .147? but it's mostly a stab in the dark [16:43:17] hmm https://www.mail-archive.com/debian-bugs-rc%40lists.debian.org/msg746247.html [16:43:23] are we hitting this? :) [16:46:39] very likely! https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.164 fixes this [16:49:06] nice.. /o\ [16:49:31] I'll update ncredir4003/4004 when a bookworm updat with 164 is out, then we can re-test [19:34:25] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad: lsw1-d2-eqiad BGP maintenance - https://phabricator.wikimedia.org/T419647#11699470 (10MoritzMuehlenhoff)