[00:35:17] <jinxer-wm>	 (PuppetFailure) firing: (3) Puppet has failed on sessionstore2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[04:35:17] <jinxer-wm>	 (PuppetFailure) firing: (3) Puppet has failed on sessionstore2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[05:09:23] <marostegui>	 I'm going to start switching over pc1 eqiad 
[06:04:23] <marostegui>	 urandom: Not sure what is going on with sessionstore2004 as it seems to be in setup and The last Puppet run was at Sat Dec  9 00:54:08 UTC 2023 (3189 minutes ago), I have acked the above alert which has been going on for the weekend
[07:13:16] <marostegui>	 I am failing over phabricator proxy, it should be smooth and I won't force connections to move, but if you see something weird please ping me
[11:18:18] <Amir1>	 running the pl_target_id index change on master of s6 in eqiad. cc arnaudb 
[11:19:04] <marostegui>	 Amir1: Remember that this is the last week for us to run maintenance stuff
[11:19:26] * Amir1 screams
[11:35:30] <arnaudb>	 ack
[12:59:22] <marostegui>	 arnaudb: why is db1138 depooled? 
[13:02:30] <arnaudb>	 https://phabricator.wikimedia.org/T350458 it's supposed to be decomissionned, after https://phabricator.wikimedia.org/T344036 but it takes me a little more time than expected to finish https://phabricator.wikimedia.org/T343674 
[13:02:51] <arnaudb>	 I'll decomission it this afternoon so it's not confusing 
[13:03:55] <marostegui>	 yeah, we need to double check all the hosts that are depooled
[13:04:01] <marostegui>	 if they are ready to be decommissioned, let's do that
[13:04:10] <arnaudb>	 ack
[13:10:18] <marostegui>	 btullis: How's dbstore1003 replacement going?
[13:10:38] <marostegui>	 it's at 91% already
[13:12:27] <godog>	 Emperor: hi, I have this review out of your eyes, PTAL https://gerrit.wikimedia.org/r/c/operations/puppet/+/981298
[13:37:20] <Emperor>	 ack
[13:49:12] <btullis>	 marostegui: It's on the board, moving towards being started.
[14:25:59] <urandom>	 marostegui: thanks; that's a new host that dcops (presumably) just brought online (and I hadn't seen those alerts)
[14:26:51] <marostegui>	 urandom: No problem!. We should probably avoid having hosts online with puppet stopped for a lot longer
[14:46:03] <Emperor>	 Amir.1 is now to be known as OKRs Georg ;p 
[15:29:21] <godog>	 Emperor: I'm speedrunning https://gerrit.wikimedia.org/r/c/operations/puppet/+/982112 since my testing didn't uncover that condition
[15:37:52] <Amir1>	 :D
[16:16:38] <urandom>	 marostegui: ugh, that puppet failure comes from having an insetup role that uses puppet 7, and sessionstore is still puppet 5
[16:16:47] * urandom quietly weeps
[16:17:39] <urandom>	 oh wait, it's not even that... it must be worse
[16:17:42] <Emperor>	 godog: thanks
[16:17:57] <urandom>	 it hasn't had the sessionstore role applied, it's insetup::data_persistence
[16:18:16] <urandom>	 so I guess insetup::data_persistence is puppet 5, but the machine was imaged as 7?
[16:19:19] <Emperor>	 no, I think insetup::data_persistence is (meant to be) puppet 7 no; moritzm did the work
[16:21:24] <urandom>	 Emperor: actually, I just looked, it's puppet 5
[16:21:43] <urandom>	 hieradata/role/common/insetup/data_engineering.yaml lacks the bits to make it 7
[16:21:50] <urandom>	 like acmechief_host
[16:22:06] <Emperor>	 do you mean data_engineering there?
[16:22:18] <urandom>	 sorry, mispaste
[16:22:30] <urandom>	 data_engineering *is* puppet 7 in fact
[16:22:34] <urandom>	 data_persistence isn't
[16:23:49] <Emperor>	 Ah, maybe I'm confused because I reviewed a CR on 17 Nov that said "insetup::data_persistence should get moved next week."
[16:25:26] <urandom>	 https://www.irccloud.com/pastebin/n1TfLbJH/
[16:25:57] <urandom>	 those last two lines there are the "magic", as far as I understand it
[16:26:44] <Emperor>	 yeah, you're right, I think I'd just internalised incorrectly that insetup::data_persistence was going to be migrated in mid-Nov
[16:27:13] <urandom>	 I think I'm just going to reimage it
[16:31:36] <urandom>	 or...or maybe jclark is already reimaging it, score one for the addition of locking to cookbooks :)
[16:32:00] <volans>	 really? :)
[16:32:14] <urandom>	 volans: ima buy you beer
[16:34:49] <volans>	 lol, thx
[16:35:16] <volans>	 which host was this one?
[16:35:26] <urandom>	 sessionstore2004
[16:36:06] <volans>	 yep, were you able to undestand the owner of the lock from the output?
[16:36:12] <volans>	 was it clear enough?
[16:36:13] <urandom>	 I was!
[16:36:17] <volans>	 great :D
[16:37:50] <Emperor>	 [ T349619 is the tracking task ]
[16:37:50] <stashbot>	 T349619: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619