[00:06:25] FIRING: SystemdUnitFailed: sync-puppet-volatile.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:21:25] RESOLVED: SystemdUnitFailed: sync-puppet-volatile.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:33:55] FIRING: [2x] SystemdUnitFailed: rsyslog-imfile-remedy.service on aux-k8s-worker1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:28:55] RESOLVED: SystemdUnitFailed: rsyslog-imfile-remedy.service on aux-k8s-worker1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:03:36] 10netops, 06Infrastructure-Foundations: magru HE down (oct 2026) - https://phabricator.wikimedia.org/T441591#12476057 (10ayounsi) 05Open→03Resolved That was quick ! thanks! [09:17:16] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12476562 (10cmooney) [10:00:07] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 13Patch-For-Review: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12476699 (10Fabfur) 05Open→03Resolved This should be ok now [10:41:28] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12476836 (10CWilliams-WMF) @Volans having cookbooks fail at random points due to 5xx or Phabricator down issues mak... [10:45:51] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12476842 (10CWilliams-WMF) [11:34:25] FIRING: SystemdUnitFailed: netbox_ganeti_ulsfo02_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:49:25] RESOLVED: SystemdUnitFailed: netbox_ganeti_ulsfo02_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:51:02] 10SRE-tools, 06Infrastructure-Foundations, 06SRE Observability (FY2025/2026-Q1): More frequent Puppet runs on the alert hosts? - https://phabricator.wikimedia.org/T398444#12477149 (10hnowlan) 05Open→03Resolved a:03hnowlan Puppet runs on alert hosts every 5 minutes - additional concerns around reima... [13:22:13] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12477707 (10Volans) I think that in general is the responsibility of the cookbook authors decide which dependencies... [13:29:31] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12477766 (10Volans) Of course that can also be improved to either do a more specific check on the failure or perfor... [14:06:18] 10netops, 06Infrastructure-Foundations, 06SRE, 07SecTeam-Processed, and 2 others: Routinator: CVE-2026-72752 CVE-2026-72753 CVE-2026-72754 CVE-2026-72755 CVE-2026-72756 CVE-2026-72757 CVE-2026-72758 - https://phabricator.wikimedia.org/T442502#12477982 (10sbassett) [14:06:26] 10netops, 06Infrastructure-Foundations, 06SRE, 07SecTeam-Processed, and 2 others: Routinator: CVE-2026-72752 CVE-2026-72753 CVE-2026-72754 CVE-2026-72755 CVE-2026-72756 CVE-2026-72757 CVE-2026-72758 - https://phabricator.wikimedia.org/T442502#12477984 (10sbassett) [14:18:21] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12478065 (10CWilliams-WMF) > I think that in general is the responsibility of the cookbook authors decide which dep... [14:21:29] I'm having some issues trying to reimage lvs5006 as trixie, cookbook seems to have some issues using ipmi [14:22:13] Running IPMI command: ipmitool -I lanplus -H lvs5006.mgmt.eqsin.wmnet -U wmfroot -E chassis power cycle [14:22:14] > Error: no response from RAKP 3 message [14:34:48] qq regarding Puppet utils/run_ci_locally.sh , is there a reason why "bundle exec rspec modules/xxx" always seems to error for me? https://phabricator.wikimedia.org/P97007 [14:38:21] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12478226 (10Volans) >>! In T442374#12478065, @CWilliams-WMF wrote: > The healthcheck would need to be done every ca... [14:50:40] moritzm: any idea what's up with https://netbox.wikimedia.org/dcim/devices/6319/ ? It's one of the server that is alerting with my new check: https://netbox.wikimedia.org/extras/scripts/results/411839/ [14:50:50] same for the sretests if you have any idea [14:51:24] looking [14:51:48] that's the ARM host, which still needs to be installed [14:52:56] so it's expected to have a "planned" status for more than 6 months? [14:58:31] well, I'm sure Alex planned to work on it, but never managed to get to it [15:08:06] :) [15:08:28] elukey: about T438161 do you have the reference of the host's network card? [15:08:29] T438161: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161 [15:10:54] PXE boot should happen against the local install instance? local as in the same DC [15:11:04] vgutierrez: yeah [15:12:36] so from cumin2003 reimage cookbook seems to be able to configure lvs5006 to boot from pxe but it's failing to boot from the installer [15:20:04] XioNoX: console log says it's fetching stuff from apt.wm.o.. and that's apt1002.wm.o.. so that explains why I can't get lvs5006 to boot at the moment [15:22:16] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12478688 (10CWilliams-WMF) Ah, I see, makes sense +1 [15:22:56] vgutierrez: it does get dhcp from the local install server then fetch D-I from apt iirc [15:23:00] (in a meeting) [15:23:31] yup.. that seems to be the case [15:31:57] yep I can confirm! [15:32:51] yeah.. so lvs5006 pxe boot timeouts fetching the installer from apt.wm.o and goes back to booting from disk [15:32:51] XioNoX: the only thing that I added to the task was https://phabricator.wikimedia.org/T438161#12350717, and John IIRC told me that those ports were on the rear end of the server [15:33:29] vgutierrez: I guess legacy right, not UEFI [15:33:45] elukey: dunno TBH [15:33:53] lemme check [15:34:29] yeah Legacy [15:34:37] (checked via spicerack shell and redfish) [15:35:55] vgutierrez: were you watching the serial console when the timeout happened? Otherwise we can kick off another reimage and check what's happening, maybe there is a clear message etc.. I recently refactored the reimage's DHCP section, shouldn't be the problem but better-to-be-sure [15:36:36] elukey: yes.. no clear message at all... it was around ~75% and next thing I saw was the grub menu from the previous installation [15:37:14] retrying.. but I'm assuming that fetches will be slow/crash given https://grafana.wikimedia.org/goto/scgb7m?orgId=default [15:41:00] vgutierrez: I am available to give it a check if you are willing to kick it off again, I usually start from there when debugging these things, lemme know how you prefer proceeding [15:42:02] elukey: it's running since 15:37 [15:48:37] vgutierrez: yeah ok but it already passed the pxe boot step, I'd need to see it :D If the cookbook is still running I can connect to the console, powercycle and force pxe [15:48:50] elukey: please do :) [15:49:14] yeah.. cookbook is waiting for a reboot into the debian installer [15:49:22] I can cancel that if needed [15:55:04] nono [15:55:45] vgutierrez: so lvs3008 right? [15:55:48] no I am kidding [15:55:49] :D [15:55:53] :D [16:05:46] I can see http://208.80.154.10/efiboot/autoexec.ipxe... ok [16:05:58] Configuring (net0 84:16:0c:5d:95:d0)...... ok ok [16:06:25] and then it tries to fetch apt.wikmedia.org [16:07:13] yeah.. and response size on nginx access log looks smaller than the file on disk [16:07:27] so it looks like the client is going away [16:07:50] I am wondering if we should use --use-http-for-dhcp in reimage [16:08:31] because afaics it is maybe trying with tftp [16:08:53] vgutierrez: is it ok to restart the reimage with that option? So I can check [16:09:08] it's possible that mr1-eqsin is having a bad day causing those issues, papaul is looking into it [16:09:13] I'll keep watching the serial [16:09:25] XioNoX: we are seeing a 8% percent of timeouts between eqiad<->eqsin [16:09:37] vgutierrez: prod to prod or mgmt only? [16:10:17] XioNoX: hmm that comes from https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?from=now-3h&to=now&timezone=utc&var-site=eqsin&refresh=1m [16:10:29] yeah looks like HE is having a bad day too again... https://grafana.wikimedia.org/goto/s92ks4?orgId=default [16:10:36] elukey: yes.. go ahead, I'll cancel the cookbook to release the lock [16:10:46] elukey: done [16:11:42] vgutierrez: no I meant if you could kick off the same cookbook again with the extra parameter, it may be quicker [16:11:47] will do [16:12:12] running `sudo -i cookbook sre.hosts.reimage -t T401832 --os trixie --new --use-http-for-dhcp lvs5006` [16:12:13] T401832: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832 [16:12:14] it would be great if APT was active/active and eqsin would hit codfw [16:13:10] elukey: I'm using cumin2003 BTW to avoid the issues between eqiad and eqsin in that part of the workflow [16:13:23] topranks: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1359323 (or anyone else who want to review it) [16:16:30] XioNoX: hmm yeah that doesn't look good [16:16:38] +1 [16:18:52] sadly same thing [16:19:13] well no, after a long time I see [16:19:16] http://apt.wikimedia.org/tftpboot/trixie-installer/debian-installer/amd64/linux. [16:19:16] .. ok [16:19:34] it is downloading the initrd now [16:20:49] vgutierrez: change pushed, let me know if it's better [16:20:54] ack, thx [16:23:36] elukey: I have a log entry from 15:07 with a failed fetch of initrd.gz [16:23:54] it managed to get 1046640 of 537044528 bytes [16:24:00] so no great :) [16:24:04] *not [16:24:11] let's see if now it's able to fetch it [16:28:21] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591 (10ayounsi) 03NEW [16:29:46] vgutierrez: still fetching initrd.gz afaics, do you know how the failed fetch is logged? I am wondering if we hit a server timeout on apt, and the poor iPXE script is left in a weird state [16:30:23] elukey: hmm I'm just grepping nginx access.log on apt1002 [16:30:45] this is the failed fetch of initrd.gz [16:30:46] 10.132.0.7 - - [27/Oct/2026:15:07:14 +0000] "GET /tftpboot/trixie-installer/debian-installer/amd64/initrd.gz HTTP/1.1" 200 1046640 "-" "iPXE/1.21.1+git-20240627.b66e27d-wmf1" [16:32:51] elukey: traffic is definitely flowing (just checked with tcpdump) [16:34:30] 10.132.0.7 - - [27/Oct/2026:16:34:09 +0000] "GET /tftpboot/trixie-installer/debian-installer/amd64/initrd.gz HTTP/1.1" 200 537044528 "-" "iPXE/1.21.1+git-20240627.b66e27d-wmf1" [16:34:33] vgutierrez: we are in d-i! [16:34:34] it finished \o/ [16:34:45] elukey: :D [16:36:36] cookbook already detected the debian installer [16:36:47] already looking better [16:37:35] I was about to suggest that the host is from 2022 so maybe an idrac/bios upgrade may have helped, but really risky considering the host [16:39:19] elukey: it looks like a networking issue more than anything wrong with lvs5006 [16:40:00] sure sure [16:41:55] elukey: are you still staring at the console? [16:42:00] is it progressing? [16:42:14] I just quick but d-i was working yes [16:42:33] ok... cause I've stopped seeing requests to apt.wm.o from lvs5006 [16:43:52] Host up (new fresh trixie OS) [16:44:09] hmm I guess I was wrong to expect packages being fetch from apt.wm.o [16:47:38] you are never happy :D [16:48:43] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12479581 (10ayounsi) I asked Claude to have a look at this task, as the issue was alerting on AlertManager and the problem well defined. Its investiga... [16:57:53] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12479708 (10cmooney) Are we sure the issue is DAD? If it is I don't see an issue with the sysctl changes, as you say we statically configure the addr... [17:22:39] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12480117 (10ayounsi) Not sure, but I think that's the workaround worth testing, and if it solves that issue, worth rolling out infra wide to shave som... [18:04:00] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591#12480487 (10cmooney) Seems it didn't last long but still {F111022877 width=600} [23:21:49] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Tab completion for cookbook names - https://phabricator.wikimedia.org/T367230#12482226 (10Lizimedia) Just as an experiment, I Rewrote It In Rust™ using [[https://docs.rs/ruff_python_parser|ruff_python_parser]] and the difference is...