[05:02:43] <wikibugs>	 10serviceops, 10MediaWiki-extensions-Linter, 10Platform Engineering, 10WMF-JobQueue: Linter jobs are running slowly - https://phabricator.wikimedia.org/T299302 (10Legoktm) >>! In T299302#7624762, @Joe wrote: > The backlog has been recovered; tomorrow I'll lower the concurrency for such jobs.  Given that th...
[07:44:26] <wikibugs>	 10serviceops, 10MediaWiki-extensions-Linter, 10Platform Engineering, 10WMF-JobQueue: Linter jobs are running slowly - https://phabricator.wikimedia.org/T299302 (10Joe) >>! In T299302#7625149, @Legoktm wrote: >>>! In T299302#7624762, @Joe wrote: >> The backlog has been recovered; tomorrow I'll lower the con...
[09:42:58] <moritzm>	 headsup: I'm switching kubetcd1004-kubetcd1006 to DRBD-backed storage in a bit, they'll need to remain that way until the Ganeti Buster update in eqiad is complete (~ four weeks). latencies will rise a bit and if it causes any morning alerts we'll need to downtime them
[09:46:04] <jayme>	 ack
[10:11:08] <_joe_>	 akosiaris, sobanski should we meet on tuesday or another day? today people in the US are off
[10:11:20] <_joe_>	 so no Reuven and no daniel at the very least
[10:37:46] <akosiaris>	 moritzm: ok. Thanks for the heads up. 
[10:39:07] <akosiaris>	 Reuven is on leave anyway. He 'll be back on Wednesday
[10:39:40] <akosiaris>	 so moving it to tomorrow won't help with regard to him but it will help with Daniel indeed. 
[10:39:43] <sobanski>	 I can do tomorrow, same time or 30 minutes later
[10:39:59] <sobanski>	 From a glance at everyone's calendars, it should be ok to move
[10:40:14] <akosiaris>	 if it wasn't ORK planning season I would be ok with cancelling but I 'd rather we did it tomorrow
[10:40:38] <akosiaris>	 sobanski: ok, I 'll move it to tomorrow then. 
[10:40:44] <sobanski>	 Pointless trivia: ORK is the IATA code for the Cork airport
[10:42:36] <jayme>	 does cyrusone have a datacenter there? :p
[11:22:03] <moritzm>	 JFTR, kubetcd1004-kubetcd1006 are done now
[15:24:32] <wikibugs>	 10serviceops, 10MediaWiki-extensions-Linter, 10Platform Engineering, 10WMF-JobQueue: Linter jobs are running slowly - https://phabricator.wikimedia.org/T299302 (10RhinosF1) As train runners for this week, please be aware of ^
[15:56:07] <mutante>	 oooh.. it's a holiday today!
[15:56:17] <mutante>	 why am I working:) and no meeting per calendar, heh
[15:56:51] <mutante>	 _joe_: I did just remove all the font* and xfont* remnants from anything called a "canary" of any type
[15:56:57] <mutante>	 but stopped there
[15:57:42] <_joe_>	 mutante: ack, let's look at mediawiki errors from those servers tomorrow, then proceed
[15:58:19] <mutante>	 ACK, yea
[16:24:59] <wikibugs>	 10serviceops, 10MediaWiki-extensions-Linter, 10Platform Engineering, 10WMF-JobQueue: Linter jobs are running slowly - https://phabricator.wikimedia.org/T299302 (10ssastry) >>! In T299302#7625232, @Joe wrote: >>>! In T299302#7625149, @Legoktm wrote:  >> I'd also suggest that the above Parsoid patch be backp...
[23:25:07] <wikibugs>	 10serviceops, 10Observability-Logging, 10SRE, 10Wikimedia-Logstash, 10observability: ensure httpd error logs from "misc apps" (krypton) end up in logstash - https://phabricator.wikimedia.org/T216090 (10lmata)
[23:49:54] <wikibugs>	 10serviceops, 10Observability-Logging, 10SRE, 10Wikimedia-Logstash, 10observability: ensure httpd error logs from "misc apps" (krypton) end up in logstash - https://phabricator.wikimedia.org/T216090 (10Dzahn) A few things have changed since I created this ticket.  scholarships, the app mentioned, has bee...