[03:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [07:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [08:38:19] elukey: hello! iirc you had a chat with Supermicro some time in the past about DHCP option 97, is that correct ? do you remember the DL;DR; ? [08:38:23] tl;dr [08:41:08] XioNoX: hello! Just to be on the same page - 97 is the client id that dell sends but supermicro doesn't right? [08:41:57] elukey: yeah, but looks like it does send it for some hosts [08:42:20] I thought it didn't send it at all [08:42:37] I think I remember something about server generation, one couldn't send it but the next generation maybe could [08:42:46] but it's all very vague [08:42:47] anyway, they said a long time ago that maybe they'd have added it on new firmwares, but only for bigger host types (X13, we have X12 IIRC) [08:42:55] yes yes [08:43:19] so my understanding is that they are not going to add it for the server targets that we use [08:43:30] ok, jhathaway managed to get it working it looks like https://gerrit.wikimedia.org/r/q/topic:%22pxe-client-id%22 [08:43:40] so that's what I'm trying to figure out :) [08:48:29] just get a spicerack-shell, and iterate over the supermicros to get that uuid and see if you can get it from all [08:48:55] volans: nah it's there in redfish, the issue is is it there in the DHCP request [08:49:13] and that's more difficult to test on all the hosts :) [08:50:21] reboot them all! :D [09:14:48] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster2001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [09:29:48] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [10:09:13] can I reimage sretest2007 ? it's in codfw E1 so maybe it was used by topranks for some tests? [10:09:59] XioNoX: yeah fire away, you are correct I was using it in E1 to validate DHCP relay worked with the new network setup there [10:10:19] though if I remember right it's a pretty old recycled box, think the idrac could be way out of date [10:10:37] topranks: the opposite Purchase date 2025-03-26 [10:10:42] I only cared about relay so I tested from the installed system with dhclient tool, not sure if I reimaged it [10:10:45] oh right ok [10:10:50] er, that's the switch [10:10:56] server is Purchase date 2025-05-02 [10:11:00] I'm traumatised by some other ancient server obviously :P [10:11:16] let's see if that one have option 97 in its dhcp request [10:11:33] speaking of trauma.... it could be a supermicro ;D [10:11:44] we are looking at option 97 again? [10:16:13] hahaha, there is some hope with https://gerrit.wikimedia.org/r/q/topic:%22pxe-client-id%22 but I'm wondering if the hard blocker of "host doesn't expose it" is still there or not [10:16:30] basically I thought that it reqired a dhcp server upgrade, but looks like not [10:17:00] ah ok cool, you found a way to process it properly in isc-dhcpd ? [10:17:06] Jesse did :) [10:17:20] awesome [10:17:49] it could be used for dell though... anyway... more testing needed [10:18:10] but the MAC based system is working well, so at least there are no more blockers for Nokia [10:18:11] yeah if it doesn't work for supermicro probably not worth the effort [10:18:17] real shame if they don't send it [10:18:21] that's great [10:19:10] Nokia are adding the support later this year too so I think we are good on multiple fronts [10:20:16] eh, belt and suspenders, but ideally we would drop one of them [10:22:23] yeah be good to get rid of the option 82 stuff either way it's been a pain [10:34:18] hmm sretest2009 idrac seems fully offline [11:23:25] FIRING: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:28:25] RESOLVED: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [13:30:03] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [14:02:18] 10SRE-tools, 06cloud-services-team, 06Infrastructure-Foundations: sre.hosts.decommission often leaves dangling things in netbox - https://phabricator.wikimedia.org/T398052 (10Andrew) 03NEW [14:02:40] 10SRE-tools, 06cloud-services-team, 06Infrastructure-Foundations: sre.hosts.decommission often leaves dangling things in netbox - https://phabricator.wikimedia.org/T398052#10954206 (10Andrew) [15:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [16:31:39] 07Puppet, 06cloud-services-team, 10Cloud-VPS, 13Patch-Needs-Improvement: role::puppetmaster::standalone clones Git repositories as gitpuppet, git-sync-upstream overwrites them as root - https://phabricator.wikimedia.org/T152059#10954854 (10taavi) 05Open→03Resolved I believe this was fixed with the... [17:30:04] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [19:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [21:30:04] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [23:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts