[09:50:10] somebody from I/F could take a look to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1051307? thanks! [09:52:21] oh.. we got somebody from I/F on-call this week, how convenient, slyngs ^^ [10:06:05] slyngs: thx for the review [11:08:26] elukey: hi! https://gerrit.wikimedia.org/r/c/operations/puppet/+/1050601 seems to be breaking cloud puppet runs because `profile::puppetserver::git::exclude_servers` is missing from hiera [11:08:32] what value should it have there? [11:09:01] ahhhh I forgot clouuuud [11:09:20] it is fine [], the change is just to exclude servers if needed [11:09:26] lemme file a patch sorry [11:11:54] dcaro: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1051329 [11:12:32] awesome thanks, I was about to send the same patch :) [11:12:46] merged! [11:13:20] * elukey writes a note on top of his monitor - "remember cloud.yaml" [11:19:20] xd, iirc at some point there was a test in ci [11:19:24] not sure what happened to it [11:20:10] oh, it's there, but not failing the tests [11:20:12] https://usercontent.irccloud-cdn.com/file/b6HvTLC3/image.png [11:20:23] only shows in the logs [11:24:25] this is why https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/b8264b34cb04a4dbf33a252eb20377da9f68104b [11:27:28] this should enable it again https://gerrit.wikimedia.org/r/c/operations/puppet/+/1051332, but will have to do a more thorough check to see if it messes up anything else [11:40:05] topranks: the reimaging of coludcephosd1008 got stuck too, I'll open a task to investigate, should I add someone else but us for now? [11:42:32] dcaro: all I can recommend really is working with dc-ops on it [11:43:08] topranks: ack, I'll add them then 👍 [11:43:41] This host also has firmware 21.60.22.11 for the NIC, I think as unlikely as firmware would be as a cause it's probably worth installing 21.85.21.92 (using the sre.hardware.upgrade-firmware cookbook) on it and seeing if that might fix [11:44:20] sure, let me try that first [11:47:06] hmpf... the install removed the host from puppetdb, so now it does not show up for cumin xd [11:47:20] (so the upgrade-firmware cookbook does not find it) [12:09:09] vgutierrez: orchestrator is looking fine [12:09:41] marostegui: did you trigger a puppet run? [12:11:50] vgutierrez: yes, apache restarted [12:11:56] marostegui: cool, thx <3 [12:12:00] vgutierrez: <3 [12:31:53] dcaro: you can merge my puppet change if you want [12:31:56] XioNoX: I have a puppet patch from you pending merge, can I? [12:31:59] okok :) [12:32:05] :) [12:51:06] dcaro: there's a flag for the upgrade firmware cookbook that deals with the host not being in puppet (if still relevant) [12:51:34] kamila_: oh, that'd be useful, it will be relevant in a bit :) [12:51:38] thanks! [12:52:33] also, if it's getting stuck on netboot and is a 10G NIC, it is possible you hit some of the fun I hit a few weeks ago, ping me if that's the case [12:52:41] I'll write up more detail about that Real Soon Now [16:26:45] kamila_: it's 10G yep :), let me know when you have the writeup ready [16:26:47] https://www.irccloud.com/pastebin/qsQhV6xP/ [16:33:47] dcaro: will do; also, before going down that rabbit hole, have you checked the NIC BIOS PXE setting according to https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Dell_Documentation#Troubleshooting ? it might just be that [16:34:49] kamila_: it's worth checking yep [16:34:55] (you need that on the 10G NIC, I don't know which port is which, so I just make sure that it's enabled on both) [16:34:57] will do on the next reimage [16:35:37] sounds reasonable :)