[04:19:14] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:14:14] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:02:34] 07Puppet, 06Collaboration-Services, 10Gerrit, 06Infrastructure-Foundations, 13Patch-For-Review: Change puppet-merge git origin to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org - https://phabricator.wikimedia.org/T420184#12415017 (10ABran-WMF) [08:23:08] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12415061 (10cmooney) @Jclark-ctr moved the uplink to the adjacent rack and everything is now working fine. I'll keep this task open... [08:33:15] hey folks [08:33:39] I have a weird issue with sretest1006 while testing the new reimage change (move DHCP to a cookbook etc..) [08:34:12] the host boots in d-i, but then it seems that there is no network available in there to fetch the preseed file from apt.w.o [08:34:40] I am not 100% sure why, I am reimaging all the other sretest hosts and they work [08:35:12] DHCP should work otherwise d-i wouldn't boot [08:35:33] any idea? [08:35:51] elukey: any time we seen that before it was due to firmware issue on the NIC [08:36:07] basically the pattern was: PXEboot stage, the interface comes up and DHCP works [08:36:27] D-I stage: with the Linux driver trying to init the NIC port the link never comes up [08:36:45] ahhh okok [08:36:56] I think through serial console we can get a shell in the d-i stage and check, also look at the switch side [08:37:22] I've not seen that issue in a long while though, it used to be quite common but I think the newer firmwares have been less problematic (need to check with dc-ops the exact status) [08:37:40] I quickly checked logs/ping/wget in the d-i shell and I didn't make them work, but nothing more [08:38:09] did the link show as being "up" with an IP address or did you check? [08:39:17] yeah no I was a bit sleepy and didn't think about it :( [08:39:21] 10CAS-SSO, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 13Patch-For-Review: Enable the OAuth device authorization grant in CAS and register a wmf-s3-login client - https://phabricator.wikimedia.org/T435602#12415189 (10Gehel) [08:39:49] I can kick off another one if you want to check with me (if you have time) [08:40:03] elukey: wait, no don't do taht [08:40:10] you are stealing my test host!!! how dare you ! [08:40:11] lol [08:40:26] 111!!!!111 [08:40:33] sretest is connected to lswtest switch, I don't think it's expected you can reimage that it won't work [08:40:53] it's set up in a custom way with a bunch of connections and namespaces for some qos testing I'm doing [08:41:06] ohhh sorry! I should've asked :( [08:41:28] ah no it's fine, tbh I'd not been using it [08:41:30] since my changes are invasive I wanted to reimage all sretests :D [08:41:37] Did DHCP work, and is that the bit you are testing? [08:41:54] we don't do DHCP in d-i anymore so I think once you are there it means it worked [08:42:26] probably what might be wrong - looking at the setup - is that we pass the IP from the PXEboot stage to d-i but it adds it to the wrong interface, there are three working network ports on that host [08:42:34] https://netbox.wikimedia.org/dcim/devices/2245/interfaces/ [08:43:15] ens2f1np1 is connected to a network which could work - I expect your DHCP happened on that interface [08:43:29] the other two are on a test vlan which doesn't go beyond the test switch in D8 [08:43:51] the change is https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1348725 [08:44:11] I know - sorry I have been meaning to review for you [08:44:14] it is a bit more than dhcp since I am simplifying a little reimage: uuid-based dhcp for all bare metals, except VMs [08:44:37] removing the option-82 stuff? [08:44:54] no that is still a fallback in the dhcp cookbook afaics, but we can remove it later [08:45:32] there were some mixed-thing in the code that I have removed, they all work but I want to be sure :D [08:45:39] so far the tests look nice [08:46:02] basically after this sre.hosts.dhcp --os trixie $somehost will work, and the cookbook will decide what dhcp config etc.. [08:46:05] yeah I've no objection to keeping it as fallback for now [08:46:19] so reimage/provision will call it [08:46:30] but they won't know anything about dhcp anymore [08:46:34] that is good in my opinion :D [08:47:32] so far I have reimaged sretest2009, testvm2001, sretest1005, sretest2001 [08:48:04] yep it is definitely a good move - fair play for doing that! [08:48:27] I think maybe just skip sretest1006 - the problem is undoubtedly due to the unique network setup [08:48:43] basically when I reimaged before I would mangle the network into a way it would work - then set it back up how I need for my tests [08:48:51] super, I think it is now in a weird state because I removed the puppet cert, I can fix it manually in case [08:49:34] if that's easy then great, otherwise I can take a look later and see if I can get it to reimage [08:49:52] I'll try and report back! [08:52:19] elukey: actually it is probably not a huge deal if it stays broken - I was asking John if he could move it to another rack for me later today [08:52:44] and I'll need to reimage it once moved - so probably not a big deal for now [08:54:30] I am always scared by Moritz or Riccardo checking hosts with puppet broken [09:02:54] yeah I really don't know how to do any of that manually at all it's scary [09:03:12] but anyway we can probably leave this? John I think can probably move it today [09:06:25] topranks: fixed! [09:06:34] awesome, thanks <3 [09:06:50] that gives us some leeway if John doesn't have time to look at it :) [09:06:51] I'll keep going with the other sretests! [09:07:08] thanks! I'll try to look at your patch when I have a minute, may be next week [09:07:20] when is your last week? [09:07:38] I was glad to see we are juicing you for as much work as possible before you go - putting you on call and clinic duty hahaha [09:24:17] :D [09:24:24] Oct 30th will be my last day :D [09:24:35] now you reminded me that I have to check the access requests [13:20:38] sretest2003 is legacy bios but it is listed as UEFI in preseed.yaml, I am going to re-provision it [13:24:09] 10CFSSL-PKI, 06Infrastructure-Foundations, 13Patch-For-Review: Add a load balanced service in front of PKI - https://phabricator.wikimedia.org/T436809#12416231 (10elukey) Last step before closing: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1346630 [13:31:25] aaand provision is trying to add settings that are not there, sigh [13:31:35] I'll update the idrac/bios [13:45:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12416304 (10Jclark-ctr) Both switches have been Racked in C8 and D5 [15:28:11] 10netops, 06Infrastructure-Foundations: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12416665 (10dr0ptp4kt) The replication factor was restored to 2 during a call. In case we hit a situation like this again, the temporary workaround is now documented... [15:28:29] 10netops, 06Infrastructure-Foundations: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12416667 (10dr0ptp4kt) 05Open→03Resolved [17:03:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12417125 (10cmooney) >>! In T418012#12411083, @Jclark-ctr wrote: > Finished cabling for the A/B upgrade. Everything is complete except for **1 cross-connect... [17:22:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12417216 (10Jclark-ctr) @cmooney I moved the existing SCS cables from cloudsw1 to the cloudsw3 switches. I verified the SCS connections, u... [17:35:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12417289 (10cmooney) Cool thanks. If we can run the following links from the Juniper switches in those racks to the new ones that'd be gre... [19:19:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12417704 (10Jclark-ctr) These cables have been connected >>! In T440313#12417289, @cmooney wrote: > Cool thanks. If we can run the foll...