[09:33:39] moritzm: ok to merge yours? [09:35:45] oh, sorry. yes [09:36:03] done [09:36:32] thx [19:17:44] I'm getting an error reimaging elastic2107 , looks like it's trying to look up the puppet 7 key in hiera? Has anyone seen this before? https://phabricator.wikimedia.org/P58397 . FWiW this role has already migrated to puppet 7 [19:21:10] inflatador: is elastic2107 really in site.pp and has a role assigned though? [19:21:27] and the hiera setting for puppet7 is on that role level? [19:21:44] yea, I have seen it and once I could get the console with that "trick" [19:21:58] I could see puppet didn't run because of the cert issue [19:22:19] mutante it was a bookworm host that needed to be rolled back to bullseye, could that be the issue? [19:22:58] inflatador: step 1: do you know how to get the ganeti console despite that bug? [19:22:58] as far as the role, should be covered by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1007969/6/manifests/site.pp#1048 and hieradata/role/common/elasticsearch/cirrus.yaml for the role [19:23:41] mutante you mean `install-console elastic2107.codfw.wmnet` ? [19:24:06] this is a physical machine FWiW...the host is up and running bookworm ATM, but needs to be reimaged [19:24:29] inflatador: oh right, those are physical machines, not VMs, so ignore the ganeti comment.. and yes.. can you see the console ? [19:25:02] also, I think the reimage cookbook asked me which puppet version I want, 5 or 7 [19:25:15] I can check, but the host is already up at SSH on its previous OS. Seems to me that the cookbook is bailing before that [19:25:35] yeah, I've seen it ask before, it doesn't seem to be asking this time [19:25:41] inflatador: so when you run puppet there is it currently broken with a puppet cert issue by any chance? [19:26:04] good call...checking now [19:26:30] ah, it's failing to compile for a different reason...bah [19:27:14] I often set the role to the "insetup" role during reimage [19:27:20] and then change it to prod role after I am done [19:27:36] then the reimage is more likely to just work [19:27:54] and you can see afterwards more directly what puppet does [19:28:06] yeah, that was my original intention but I guess I missed this one [19:28:57] I guess I'll make a CR then ;( [19:30:37] but back to your original error... I dont get that [19:30:46] it fails when trying to check Hiera for profile::puppet::agent::force_puppet7 ? [19:30:57] but you have that for the role.. I see that [19:31:21] regardless I would try what the above next [19:31:32] the puppet error is related to java packages...known issue, which is why we're rolling back to bullseye [19:32:04] ack, I see. explains why the puppet run wouldnt finish [19:32:09] still a bit confounded that reimaging is dependent on puppet working on the target host, but we can work with that. Thanks for the suggestion! [19:32:13] yet the cookbook didnt say it was waiting for puppet [19:32:24] it said it failed to lookup the force_puppet7 thing for some reason [19:32:36] yea, np [20:16:48] mutante got the insetup patch if you have time to look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1008528 [20:45:29] inflatador: the "experimental" fails but then when you click on details it's "no difference" ? [20:45:56] there is a trailing | in elastic(210[0-6]|2109|) [20:46:55] but other than that it lgtm [20:47:47] mutante ACK, fixing now [20:50:14] +1ed [20:50:45] awesome, thanks for your help! [20:54:33] mutante OK if I puppet-merge your change? [20:54:46] inflatador: yes, please do multiple option [20:54:54] ACK, done [20:54:57] thx