[08:43:52] 10netops, 06Infrastructure-Foundations: eqsin-codfw HE packet loss/latency - oct 2026 - https://phabricator.wikimedia.org/T439947#12410152 (10ayounsi) 05Open→03Resolved a:03ayounsi Thanks Willy ! [09:18:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12410260 (10cmooney) [09:53:05] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12410428 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=59bad94b-4992-4c49-bf42-4f76ff78c6e3) set by cmooney@cumin1004 for 2:00:00 on 16 host(s) and their services with reason:... [11:36:25] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12410893 (10cmooney) I've upgraded cr1-codfw now too so we can close this. [11:36:34] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12410897 (10cmooney) 05Open→03Resolved [12:16:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12411083 (10Jclark-ctr) Finished cabling for the A/B upgrade. Everything is complete except for **1 cross-connect between the cages** Cables are pre ran thou... [12:18:30] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12411092 (10Jclark-ctr) Additionally cabled lsw1-C1 Adjusting rack positions so scs-c1 is above lsw and mgmt | Network | Device | Port | Con... [12:37:43] 10CFSSL-PKI, 06Infrastructure-Foundations: i/o timeout from cloudvirt when talking to pki.discovery.wmnet - https://phabricator.wikimedia.org/T440581 (10fgiunchedi) 03NEW [12:41:21] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12411175 (10Jclark-ctr) Finished cabling for the A/B upgrade. Everything is complete except for **1 cross-connect between the cages** Cables are pre ran... [12:42:33] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12411177 (10Jclark-ctr) A1 was not cabled I have updated netbox for cableid's for all the uplinks except the spine links since they are not setup. [12:53:39] hey folks, I need to update the pki_group in policies/cr-cloud-hosts.yaml with the new LVS IPs [12:53:47] but I am a bit lost in homer_public [12:53:59] where are defs like "pki_group" defined? [12:54:43] implicitly via the "pki" hostname [12:55:38] when these are changed, I usually run https://netbox.wikimedia.org/extras/scripts/1/ and then the generated change needs to be deployed via homer [12:56:03] but for a more authoratative answer, better have Arzhel or Cathal confirm the finer details :-) [13:00:31] oh okok, I may need something custom then, because I need to add the pki LVS Ips [13:02:11] 10CFSSL-PKI, 06Infrastructure-Foundations: i/o timeout from cloudvirt when talking to pki.discovery.wmnet - https://phabricator.wikimedia.org/T440581#12411306 (10elukey) The homer policy policies/cr-cloud-hosts.yaml implicitly lists the pki hostnames, not the LVS IPs :( Need to figure out a way to add them to... [13:19:44] elukey: as moritz says they are generated by the hostname prefix. so the "pki_group" will only mean the primary IPs of those hosts [13:19:49] that won't cover the new svc IPs [13:20:20] what we probably need to do instead is put something into definitions/static.net to cover them [13:21:55] ah right! [13:22:31] and then replace it in the pki_group section right? [13:22:33] do you have the IPs handy? [13:22:42] 10.2.{1,2}.96 [13:22:43] I can do it now and check it works first [13:22:45] ok thanks [13:22:50] super thanks [13:27:05] that should do the trick: https://phabricator.wikimedia.org/P96642 [13:29:57] I tested on cloudvirt1080 but it doesn't work, does it need to be applied on both cr eqiad? [13:30:33] sry, we need to merge the patch then run homer on the CRs yes [13:30:34] https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1353197 [13:36:22] elukey: I pushed the changes from my laptop to the CRs so it should work now [13:36:47] I need to step away for 30 mins, if you can +1 I'll merge that patch when I'm back to keep things in sync [13:45:44] 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12411509 (10cmooney) [14:42:48] 10CFSSL-PKI, 06Infrastructure-Foundations: i/o timeout from cloudvirt when talking to pki.discovery.wmnet - https://phabricator.wikimedia.org/T440581#12411939 (10elukey) Cathal pushed https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1353197 to the routers, everything seems working now! [14:51:14] I have some high level questions about how we do DHCP in reimage, in the context of moving the logic to the DHCP cookbook - https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1348725 [14:51:45] today I was testing the dhcp cookbook with Filippo and something that we realized is that the user would need to know stuff like --opt82, --uuid, --mac etc.. [14:52:12] so I thought - wouldn't it be better if we delegate this complexity of choosing the right settings to the dhcp cookbook? [14:52:27] namely removing the possibility to set opt82, uuid etc.. [14:52:35] the cookbook does it for you [14:52:49] from my test it seems working fine, but I am wondering if I am missing some use case [14:53:11] the idea is to reduce the tunables and the complexity of dhcp and reimage [14:53:14] lemme know :) [14:55:58] side note - I am reimaging sretest1006 [15:05:14] sounds reasonable, but we needed those flags in the past [15:12:19] 10CFSSL-PKI, 06Infrastructure-Foundations: i/o timeout from cloudvirt when talking to pki.discovery.wmnet - https://phabricator.wikimedia.org/T440581#12412196 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Confirmed, thank you all! ` cloudvirt1063:~$ telnet pki.discovery.wmnet 443 Trying 10.2.2.96.... [15:20:46] jhathaway: IIRC it was when we introduced Nokias, but with the uuid workflow etc.. I think we ended up coding a function in reimage that tried uuid first, and used opt82 as fallback. This is why I am wondering if those are needed :) [15:23:14] with sretest1006 (behind a Nokia) I end up with Oct 8 15:21:42 main-menu[747]: (process:1264): wget: unable to resolve host address 'apt.wikimedia.org' [15:23:17] that is weird mmm [15:23:25] hmm, that is strange [15:23:38] in d-i, so dhcp etc.. should have worked [15:23:54] we use uuid everywhere, the opt82 stuff can be removed [15:27:14] yeah, bye bye option 82 [15:27:38] ok maybe I can do the removal as second follow up step, just to avoid mixing too many things [15:27:50] elukey: or maybe do it first? [15:27:51] I am currently trying sretest2009 and testvm2001 [15:28:01] that would be less things to refactor [15:28:13] XioNoX: I already have two patches ready to go, please don't make me rebase all :D [15:28:50] haha, my suggestions was to make it easier :) [15:29:21] but as soon as eqiad rows A/B are upgraded to nokia, the network will be free from any dhcp option 82 config [15:42:22] there is something off in the new version of reimage, sretest2009 doesn't boot into d-i, but with the old version yes [15:42:33] so I'll have to compare the differences :) [15:57:37] the dhcp snippets that reimage creates in both version are the same for sretest2009 [15:57:39] mmmmm [16:43:14] ok I think I found the bug, will send an update tomorrow after some tests! [17:26:58] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631 (10cmooney) 03NEW p:05Triage→03Medium [17:27:20] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12412990 (10cmooney) [17:28:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12412998 (10cmooney)