[15:41:21] I have 15 hosts that need to be reimaged to bullseye (they're not in production yet). Is there a way to reimage them all at once, maybe by flipping the 'insetup' role on and off in site.pp or something? [16:55:19] Are they physical machines or VMs? [16:57:17] https://wikitech.wikimedia.org/wiki/Server_Lifecycle/Reimage anyway, this may help. I am not sure whether it allows for specifying multiple machines. At worst, you could c&p the command for every host in one big screen/tmux session [17:02:36] assuming physical hosts: the last time I looked at this, there wasn't a clean way to reimage a lot of hosts simultaneously, because occasionally it goes wrong and the process needs to be manually unstuck [17:03:22] you wouldn't want to reimage fifteen hosts at once, have two or three of them fail, and have to go dig through logs to figure out which ones and what state they're in [17:07:30] but you wouldn't necessarily have to do it all by hand, maybe something like `for i in hosts... do cookbook sre.hosts.reimage... || break; done` [17:07:42] in $hosts, you know what I mean [17:11:07] or you can just do it all at once and if it goes wrong ping v.o.l.a.n.s ;) [17:23:48] https://phabricator.wikimedia.org/T305589 tracks this and there is an upcoming c-ookbook for it [17:23:56] the usecase was Wikidough but yeah, not limited to it [17:24:06] (I am sure vol-ans has an alert for c-ookbook :P) [17:24:23] this is just for VMs though [17:25:55] sorry, https://phabricator.wikimedia.org/T306661 is a better task [17:26:06] > decommission + makvevm with the same hostname (and we could make adjustments to keep the same IP too). That will be logically equivalent to a physical hardware reimage, including the downtime of the host. [18:11:23] btw legoktm rzl -- if one or both of you could review my five klaxon patches starting at https://gerrit.wikimedia.org/r/c/operations/software/klaxon/+/813938 I'd appreciate it (and btw legoktm that one should unblock your other patch) [18:11:58] I should be able to get to it tomorrow if I'm not beaten [18:12:03] sure, I can look today [18:12:12] <3 [18:13:21] * bd808 hopes nobody ever beats legoktm [18:13:24] https://i.imgur.com/y8ZkZFP.png [18:15:31] bd808: heh :) we have a NYC meetup tonight so that's where my wiki time is going today [18:21:06] cool [19:21:01] thanks for the reimage advice, might do a for loop like r-zl suggested. I've done a lot of c-ookbook reimages lately and the process has been quite reliable so far, barring stuff like outdated firmware and BIOS misconfiguration [19:35:44] If you're a tmux user, you might give tmuxp a try, it makes it easier to set up a whole complex tmux session with repeated commands [19:36:20] But now I must go back to listening to vinyl from my fave Greek shop :) [19:41:25] thanks klausman , I'm trying get better w/tmux. switched over from screen [19:44:44] * bd808 is a long time tmux user but still uses screen key bindings :) [19:45:41] I'm deffo still on ^A as a leader. But most of the rest is hybrid