[09:04:53] <elukey>	 Krinkle: o/ I think that you need to run /usr/local/bin/puppetserver-deploy-code when you cherry pick, there is a new post-merge step now on puppetserver nodes
[09:23:45] <_joe_>	 Krinkle: wrong channel for beta questions.
[12:50:35] <Krinkle>	 elukey: thx, looks like one of the git commit hooks didn't run after my cherry pick. I figured a rebase would do it, but had to do an amend instead.
[14:39:38] <elukey>	 hello on-callers, I am playing a bit with kartotherian, if you see anything related to maps.wikimedia.org it is probably my fault
[14:42:30] <_joe_>	 always playing with fire smh
[14:42:54] <sukhe>	 and as usual being too considerate :P
[14:43:58] <_joe_>	 sukhe: moving stuff around kartotherian is like moving plutonium bars, tbh I think caution is deserved in this case
[14:44:15] <_joe_>	 not because of elukey's action, but because of,  well, that thing.
[14:45:32] <elukey>	 I have another thing that I want to do, namely change the k8s pods weights so that more requests will flow to them (instead of bare metals0
[14:45:43] <elukey>	 but I have an interview in 15 mins so not the right moment :D
[14:45:56] <_joe_>	 lmk if you need help
[14:45:59] <elukey>	 atm I just pooled the remaining two k8s workers in both eqiad and codfw
[14:46:04] <elukey>	 ack thanks!
[15:51:35] <Amir1>	 If you have some extra plutonium. I desperately need some
[15:51:45] <Amir1>	 for peaceful purposes only 
[15:56:53] <vgutierrez>	 Amir1: compiling MariaDB got that expensive? ;P
[15:57:35] <Amir1>	 :D
[15:59:53] <elukey>	 speaking of plutonium, I am going to change some weights for kartotherian in codfw (to push more traffic to k8s pods)
[16:09:20] <elukey>	 same in eqiad
[16:17:53] <elukey>	 something weird that I noticed is this https://logstash.wikimedia.org/goto/81bd58d747f495bfd803c404d3e481ee
[16:18:31] <elukey>	 the test probe seems to fail sometimes like
[16:18:32] <elukey>	 $ curl -k -i https://kartotherian.svc.eqiad.wmnet:6543/osm-intl/6/23/24.png
[16:18:35] <elukey>	 curl: (7) Failed to connect to kartotherian.svc.eqiad.wmnet port 6543 after 3 ms: Couldn't connect to server
[16:18:44] <elukey>	 only when I hit the svc endpoint though, nothing on specific backend nodes
[16:24:17] <elukey>	 seems to be happening only for k8s pods
[16:30:23] <elukey>	 I restored the previous lvs weights just as precaution, but this thing is very weird
[17:08:09] <elukey>	 there is something weird going on, so I precautionary depooled from the kartotherian-k8s-ssl lvs service all k8s nodes (eqiad and codfw)
[17:08:13] <elukey>	 I'll restart tomorrow
[17:08:19] <elukey>	 (pooled=inactive)
[17:15:47] <kamila_>	 thanks elukey <3
[20:03:32] <inflatador>	 is anyone else getting a 503 from https://apt-browser.toolforge.org/ ?
[20:04:18] <inflatador>	 err...502 that is
[20:20:14] <kamila_>	 inflatador: same here
[20:21:14] <inflatador>	 legoktm ^^ heads-up, LMK if I can do anything to help get 'er going again
[20:45:00] <inflatador>	 (and thanks k-amila_ for checking)