[01:33:08] and it's merged \o/ I will deploy and run the script early next week. It'll take a while to clean up the mess [09:55:57] my best guess with HP servers, based on documentation, is that a BIOS update reenabled the sd card reader [09:56:22] but those hosts had a BIOS upgrade recently? [09:56:59] I cannot otherwise explain why remote IPMI was disabled and drive changed letter [09:57:00] is that something you can disabled remotely or do we need DCOps? [09:57:09] I can do it from bios [09:57:29] I have been fighting those servers enough time already :-( [09:58:25] :( [09:58:43] and it would make sense HP bios did it, because there was a task about reflasing that on HP only [09:59:34] but cannot find the task quickly [10:29:26] after disabling the sd card reader, the installer worked nicely :-), almost sure it was that [10:30:32] sweeet [10:30:58] my belive is that this: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/HP_Documentation#Initial_System_Setup was half done -e.g. the password, network, etc. was setup but remote ilo, sd card, etc no [10:31:25] and because it wasn't fully reset or fully setup, it was very confusing [10:31:32] where these hosts installed directly with stretch? I mean, first time they get a reimage? [10:31:52] probably a bios update or hw maintenance reseted partially the options [10:32:31] marostegui, sorry, I am not fully getting your question. Do you mean when it was bought or by me now? [10:33:59] Now I am only reimaging to buster- but it was not an os issue (I think I am not understanding what you really mean) [10:34:16] jynus: yeah, I mean, why did this problem showed up earlier? [10:34:45] why it didn't? probably the issue was there before, but after first reimage [10:34:58] it broke at some point but really doesn't affect until it is reimaged [10:35:00] That was my question ,if this was the first time those hosts got reimaged [10:35:17] probably, if my theory is true, since a bios update or a hw maintenance [10:35:32] so won't be shown on simple reboot [10:35:45] because linus most likely will use uuids for booting [10:35:46] yeah, that makes sense [10:35:48] *linux [10:36:10] but on reimage, we use /dev/sda as we expect the raid to be the first drive [10:36:40] I even tried to install to /dev/sdb once, but then grup tried to install to the first drive anyway [10:36:45] *grub [10:37:11] probably doesn't help that partman is not like the best software in the world :-) [10:37:47] ah, db1139 got a board change [10:38:10] and so did db2100 [10:38:18] Do we use FAI? You can do horrible hacks with that to tweak which drives get presented to the partitioner [10:38:29] so technically not a bios update, but same effect [10:38:59] so what we need to do is to tell dc-ops to double check bios settings after hw replacement like board change [10:39:44] Emperor, sorry, do not know what FAI is [10:42:48] I wonder if there is an easy way to alert based on bios settings or something? [10:44:53] also, I can see how useful a usb drive can be on a server, but an SD card??? :-D [10:45:02] jynus: we probably don't use it then :) (FAI = fully-automatic installation - https://fai-project.org/ ) [11:40:51] I am thinking of reloading a couple of redundant backup dbs from logical backups [11:41:45] so don't worry if you see a couple of backup source instances not fully in service [14:21:08] we should move to serverless, that would solve all our installation issues!!!! [14:21:47] I might have to go down the rabbit hole of djvu in beta cluster [14:26:32] 🐟 [16:58:51] I am disconnecting, have a nice weekend! [17:16:59] ~~~