[00:05:38] <Krinkle>	 _joe_I've filed T347366 about the rename and puppet change proposal (\cc cwhite about `isTest`)
[00:05:39] <stashbot>	 T347366: Follow-up on wmf-config "ClusterConfig::isTest" method - https://phabricator.wikimedia.org/T347366
[00:05:56] <Krinkle>	 I've tagged it as onboarding so please don't write the patch yet :)
[07:30:25] <_joe_>	 Krinkle: ack. The traditional mwdebugs are a bit of a mess OTOH, that will need some expert to write the patch
[09:16:47] <jbond>	 _joe_: currently threre is no puppetdb ci tool.  i genrally just create a quick script on one the pu8ppetdb server (https://wikitech.wikimedia.org/wiki/Puppet/PQL)
[09:17:07] <_joe_>	 jbond: yeah I remembered incorrectly
[09:17:09] <jbond>	 with the new puppt db's we should be able to install puppetdb cli https://www.puppet.com/docs/puppetdb/7/pdb_client_tools
[09:17:14] <_joe_>	 also sorry for the ping in absentia
[09:17:18] <jbond>	 however i have not played with that yet
[09:17:20] <jbond>	 no worries
[09:17:28] <_joe_>	 another puppetlabs creation? 
[09:17:38] <jbond>	 possibly you where thingkming opf ./utils/puppet-debugger.sh but that has no puppetdb
[09:17:41] <jbond>	 yes
[09:17:41] <_joe_>	 did they learn cli UX in the last 10 years?
[09:17:46] <jbond>	 lo;l
[09:17:49] <jbond>	 no
[09:17:55] <_joe_>	 ok, gr8
[09:18:03] <_joe_>	 we want our tools slightly adversarial
[09:18:59] <jbond>	 :)
[09:23:18] <Emperor>	 #include <jokes/ancient/git.h>
[09:26:30] <Emperor>	 (a recent PR to fix ceph's build system involved way too much git rebase -i)
[13:23:51] <arnaudb>	 schema change for s6 is running
[15:25:12] <effie>	 swift has been returning more errors since ~15:00 UTC
[15:26:01] <effie>	 and the thumbor errors could be a sideeffect of that 
[15:28:48] <effie>	 Emperor: are you around?
[15:29:39] <jynus>	 effie: look security, it is not swift
[15:30:08] <effie>	 sigh, I was not in security due to a netsplit 
[15:30:28] <jynus>	 np
[15:30:37] <jynus>	 sorry about not noticing you asking earlier
[15:31:09] <effie>	 ok thanks
[16:11:57] <urandom>	 arnoldokoth: you ack'd https://portal.victorops.com/ui/wikimedia/incident/4077/details yesterday, and it just expired.
[16:12:08] <urandom>	 is it expected to be down still?
[16:13:33] <urandom>	 nevermind, I guess that's: https://phabricator.wikimedia.org/T347318
[16:14:05] <arnoldokoth>	 urandom: Yeah, I was just about to share that.
[16:14:40] <arnoldokoth>	 Looks like it's online.
[16:15:02] <arnoldokoth>	 So maybe we can resolve?
[16:15:31] <urandom>	 I just did, yeah
[19:01:11] <andrewbogott>	 jhathaway: Have you actually seen openldap run on bookworm? I'm seeing several issues with our slapd.conf, wondering if that's just me or if y'all just haven't got that far yet.
[19:05:29] <jhathaway>	 moritz.m: has been doing all the hard work, I've just been reviewing patches, but I am pretty sure he has been testing everything on bookworm
[19:06:22] <jhathaway>	 here is the task andrewbogott https://phabricator.wikimedia.org/T331699
[19:06:37] <jhathaway>	 and there are still outstanding patches, https://gerrit.wikimedia.org/r/c/operations/puppet/+/961066
[19:06:47] <jhathaway>	 so it looks like the puppetry is not yet complete
[19:06:57] <andrewbogott>	 ok, it was seeing that https://gerrit.wikimedia.org/r/c/operations/puppet/+/961066 won't work that caused me to ask
[19:07:11] <andrewbogott>	 so maybe I'm a bit further along, I'll have to figure out some time to talk to moritz about it
[19:09:57] <jhathaway>	 andrewbogott: I'm sure he would appreciate any assistance on his work
[20:33:43] <urandom>	 has anyone had a reimage fail where after a reboot, you get 'error: symbol `grub_file_filters' not found.`, and then it drops you to a grub-rescue prompt?
[21:11:23] <inflatador>	 urandom has it happened multiple times? A little paranoid just 'cause I'm working on a new partman recipe, hopefully haven't messed anything up for others
[21:12:10] <urandom>	 inflatador: it's happened to one host, I retried and failed in the same way the second time 'round
[21:15:26] <inflatador>	 urandom interesting. Checking DC Ops' channel to see if there's been a successful reimage since I merged the last patches. Working out of https://phabricator.wikimedia.org/T342463
[21:20:32] <inflatador>	 jclark-ctr ^^ I see you had a reimage failure for lists1004, did you get a similar error to urandom ?
[21:22:19] <jclark-ctr>	 I started imaging while at data center and started driving home I have not looked at it yet
[21:23:13] <inflatador>	 np, I'm getting a different error but I'll get a revert patch up just in case
[21:25:51] <urandom>	 inflatador
[21:25:54] <urandom>	 inflatador: oh!
[21:26:15] <urandom>	 https://usercontent.irccloud-cdn.com/file/Nu4LQnCa/image.png
[21:26:23] <urandom>	 that's lists1004
[21:26:49] <inflatador>	 urandom interesting. It seems impossible that this could happen just from these changes but I'll go ahead and wipe 'em out just incase
[21:27:09] <urandom>	 jclark-ctr: I didn't do anything to it btw, just connected to the console to have a look
[21:31:04] <inflatador>	 urandom OK, patch is here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/961245
[21:33:07] <urandom>	 this looks pretty self-contained, I can't see how it would cause problems
[21:33:18] <inflatador>	 yeah, I agree too
[21:33:32] <inflatador>	 But I'l go ahead and merge just to see what happens
[21:34:39] <urandom>	 inflatador: did you run puppet on the install server?
[21:34:54] <inflatador>	 urandom it's in flight now
[21:35:14] <urandom>	 codfw too?
[21:35:37] <inflatador>	 Y, used cumin alias to target all install hosts
[21:35:46] <urandom>	 👍
[21:37:24] <inflatador>	 urandom puppet is finished, let us know how it goes
[21:55:06] <urandom>	 inflatador: um...it worked.
[21:55:40] <urandom>	 sooo....why is that?
[22:10:10] <inflatador>	 urandom good question...hopefully a coincidence but I'm going to get a ticket started, maybe we can all put our heads together. jclark-ctr keep us posted on your reimage as well
[22:11:51] <jclark-ctr>	 mine failed for another reason. running it again but it is stuck on debian installer    i cant stop it have to wait for it to fail. 
[22:16:38] <inflatador>	 no worries
[22:23:16] <inflatador>	 OK, created https://phabricator.wikimedia.org/T347434 , feel free to add any relevant info