[07:24:21] <jelto>	 etherpad will be down for some minutes, reboot needed
[07:34:33] <jelto>	 done :)
[12:38:32] <klausman>	 I am running into a somewhat bizarre problem with the docker registry. I am trying to update the kserve images, and when trying to build it with --debug (locally, on my workstation), I get: 2023-10-20 14:36:57,837 [docker-pkg-build] ERROR - Unexpected error building image docker-registry.wikimedia.org/kserve-build:0.11.1: Image docker-registry.wikimedia.org/golang1.21 not found
[12:38:35] <klausman>	 (image.py:208)
[12:38:42] <klausman>	 and yet, I can docker pull that image just fine.
[12:39:14] <klausman>	 The first line of my temp,late is `FROM {{ 'golang1.21' | image_tag }} as build`
[12:39:40] <klausman>	 I have no idea what's going on. I earlier did the work to include golang1.21, and all of that worked just fine.
[12:40:01] <klausman>	 (include=make 1.21 and publish it from the buildhost
[12:53:03] <klausman>	 ah, somehoew the image_tag machinery breaks
[18:16:30] <sukhe>	 jhathaway: hi! is it correct to assume that lists1003 is not in production yet?
[18:16:44] <sukhe>	 I am asking because we got an alert about failed puppet runs: https://puppetboard.wikimedia.org/node/lists1003.wikimedia.org
[18:16:46] <jhathaway>	 sukhe: correct
[18:16:48] <sukhe>	 aaah ok!
[18:16:50] <sukhe>	 denisse: ^ 
[18:16:56] <jhathaway>	 I'll take a look at the puppet run however
[18:17:21] <denisse>	 Thanks Jesse!
[18:17:31] <sukhe>	 jhathaway: it can't connect to m5-master, which makes sense since the /etc/ferm/conf.d/10_mailman3 on dbproxy1021 (failover) doesn't have lists1003 
[18:17:47] <sukhe>	 which is why on list1003: Oct 20 17:54:23 lists1003 mailman3[155002]: sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to >
[18:17:55] <sukhe>	 Can't connect to MySQL server on 'm5-master.eqiad.wmnet
[18:18:25] <jhathaway>	 ah, makes sense, thanks for digging into it
[18:18:44] <sukhe>	 np, denisse and I were looking into it becaues we are on-call but if not in prod, we can skip it :) 
[18:19:25] <denisse>	 I checked the ferm rules and there's no grant for lists1003 to access m5.
[18:19:35] <jhathaway>	 for sure, apologies for leaving that alert around
[18:19:51] <denisse>	 No worries. :)
[18:19:59] <sukhe>	 denisse: yep, which matches the error above too, so all adds up
[19:45:59] <inflatador>	 jbond OK if I merge these idp test-related changed?
[19:46:47] <inflatador>	 jbond nm, that's for labs-private. I skipped over it and merged my prod puppet change