[10:38:49] <Emperor>	 do we have existing puppetry to mask/disable a package-provided systemd timer?
[10:41:57] <volans>	 we do have systemd::mask IIRC
[10:42:16] <Emperor>	 Ah, yes, that's the badger. Thanks :)
[10:44:12] <claime>	 dcaro: can I merge your toolforge::bastion change?
[10:45:21] <dcaro>	 claime: oh yes please
[10:45:34] <claime>	 dcaro: in progress
[10:46:23] <dcaro>	 thanks!
[10:48:03] <claime>	 done
[11:01:12] <volans>	 soo we're having a bunch of Wikimedia\RequestTimeout\RequestTimeoutException that started around 9am UTC this morning
[11:01:18] <volans>	 for mw-api-ext
[11:02:01] <volans>	 I don't see any related SAL
[11:04:22] <volans>	 reportin glogs link here for simplicity: https://logstash.wikimedia.org/goto/9c7086e0ce2a253a2ad35eb088a89960
[11:05:02] <volans>	 current status: we've around  5req/s of "The maximum execution time of 60 seconds was exceeded" errors
[11:06:53] <effie>	 volans: let me read p 
[11:07:55] <volans>	 allf or commons
[11:08:03] <effie>	 just noticed 
[11:10:19] <volans>	 I think all for 'allimages'
[11:10:25] <urbanecm>	 and all of them are `/w/api.php?action=query&aicontinue=*&ailimit=500&aiprop=canonicaltitle%7Csize%7Curl%7Csha1%7Cmediatype%7Cextmetadata%7Cmetadata&assign_time=*&format=json&list=allimages&seed_id=*` afaics, and from a single IP range AFAICS
[11:10:45] <volans>	 yeah it looks some crawler
[11:14:27] <effie>	 urbanecm: could you paste me the IP in private since you have already found it ?
[11:14:35] <urbanecm>	 effie: sure
[11:43:15] <volans>	 apparently a single pod was misbehaving
[11:43:29] <volans>	 deleteing the pod resolved the issue. PAge recovery arrived
[11:43:41] <volans>	 urbanecm: I think you're free to go with your deployment, sorry for the delay
[11:43:47] <volans>	 effie: do you agree? ^^6
[11:43:49] <urbanecm>	 volans: yay! thanks, proceeding then
[11:43:51] <urbanecm>	 oh
[11:44:15] * urbanecm waits for effie to respond just in case
[11:45:26] <volans>	 thanks for the help silent j.o.e
[11:50:39] <effie>	 sorry I had to step out for 5' as my kids came back 
[12:28:26] <_joe_>	 oncallers: I'm disabling puppet on the cdn, and merging changes one by one, I should not cause any issues, but I'll let you know when I'm done
[12:29:08] <volans>	 ack, thx
[12:31:11] <fabfur>	 ok
[12:39:31] <sukhe>	 _joe_: thanks and gl
[13:51:54] <_joe_>	 https://wikitech.wikimedia.org/w/index.php?title=Requestctl&diff=2207181&oldid=2204692
[13:52:12] <_joe_>	 should I send an email to sre-at-large maybe
[13:52:17] <sukhe>	 +1
[14:06:54] <volans>	 +1
[14:11:20] <inflatador>	 akosiaris just wanted to make sure we are still OK for creating kafka topics on kafka-main, re: https://phabricator.wikimedia.org/T367510#9902381
[14:23:03] <_joe_>	 inflatador: akosiaris is on vacation, but I think the kafka-main refresh is making disk space a scarce commodity
[14:23:18] <_joe_>	 jayme: you maybe remember?
[14:23:29] <inflatador>	 _joe_ ACK, had that in mind when I asked
[14:24:07] <_joe_>	 or claime 
[14:25:02] <inflatador>	 dcausse ^^
[14:25:38] <claime>	 yep, https://phabricator.wikimedia.org/T368714 
[14:25:55] <claime>	 a.kosiaris created procurement requests for more disks
[14:26:19] <jayme>	 ah, here
[14:26:26] <jayme>	 sorry. was in a meeting
[14:27:05] <inflatador>	 np...we were looking at doing this today if that is still OK
[14:27:07] <jayme>	 yeah. d.causse already asked me about this a couple of hours ago
[14:28:17] <jayme>	 we'll be holding back the refresh until we have the new disks. So go ahead with what was agreed on already
[14:29:07] <inflatador>	 ACK, thanks again jayme ... will get it started then
[14:29:11] <dcausse>	 jayme: thanks!