[11:10:56] <dcausse>	 !log stopped image_suggestions_weekly while some issue with cassandra is being worked out
[11:10:56] <stashbot>	 dcausse: Not expecting to hear !log here
[11:12:41] <dcausse>	 lunch
[14:13:30] <inflatador>	 <o/
[14:28:29] <dcausse>	 o/
[14:51:10] <inflatador>	 Yet another CR for changing TLS on cloudelastic if anyone has time to look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/995041 . Low risk change as we'll roll out on one host and revert if necessary
[15:50:15] <inflatador>	 cleaning up the thanos swift containers now...so far the savepoints have not made much of a difference
[16:05:17] <ebernhardson>	 \o
[16:06:31] <inflatador>	 <o/
[16:06:56] <inflatador>	 Leaving soon to do volunteering. I'll be driving someone to a medical appointment, so I should be able to hop back on once I get there
[16:10:05] <dcausse>	 o/
[17:01:46] <inflatador>	 back
[17:41:17] <dcausse>	 dropping $wgCirrusSearchAllFields and all related code, not sure that's wise but I'm sure we're now making plenty of assumptions that the "all" fields exist and almost sure that cirrus might not work well with something other ["use"=>"yes", "build"=>"yes"]
[17:41:32] <ebernhardson>	 dcausse: makes sense
[18:27:18] * ebernhardson finds it curious there don't seem to be LogOnlyRemediator logs in logstash
[18:28:15] <ebernhardson>	 they should be info logged to CirrusSearch channel, and i find other messages of those kinds....curious
[18:33:53] <dcausse>	 weird... I see 'CirrusSearch' => 'debug' in wmgMonologChannels...
[18:35:17] <ebernhardson>	 hmm, found them on mwlog2002. I suspect they are being rejected by logstash, we provided a Title object instead of a title string, which serialized into {"MediaWiki\\Title\\Title": "My page name"} instead of just "my page name"
[18:37:45] <ebernhardson>	 can get all the page ids from mwlog at least.  Although now i'm thinking it would be nice to have the matching logs for eqiad, to diff against
[18:37:57] <ebernhardson>	 but we don't log those :P
[18:41:16] <ebernhardson>	 fix rate might be up a bit, but hard to say yet. typical delta between cloudelastic and eqiad is ~30 fixes/day. yesterday it was ~100 fixes delta.  I suppose if it is a problem it will be more obvious by monday :)
[19:02:34] <dr0ptp4kt>	 oh ha, there's a caffeinate command line utility right there in macos.
[19:18:48] <inflatador>	 back
[19:58:15] <inflatador>	 appointment, back in ~90,
[21:02:34] * ebernhardson mutters at kafkacat taking ms timestamps to start and end at..instead of something simple like an iso8601 timestamp :P
[21:31:36] <inflatador>	 back
[21:45:47] <inflatador>	 I'm going to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/995041 (cloudelastic TLS changes) momentarily...no disruption expected, but just an FYI
[21:51:10] <inflatador>	 Puppet run was successful...rolling out to the rest of the cloudelastic fleet 1 by 1
[21:54:21] <ryankemper>	 inflatador: awesome!
[22:06:33] <inflatador>	 SSL alerts are resolving too
[22:36:55] <inflatador>	 ryankemper got a couple more CRs if you have time to look https://gerrit.wikimedia.org/r/c/operations/puppet/+/995107 https://gerrit.wikimedia.org/r/c/operations/puppet/+/995110
[22:38:02] <ryankemper>	 inflatador: +1'd the canary patch
[22:41:53] <inflatador>	 cool, merging shortly
[23:16:03] <inflatador>	 So we're merged, looks like traffic is flowing to cloudelastic from lvs. Looks like access logging is completely off for cloudelastic nginx (which is fine), but I missed the sweet satisfaction of seeing that file grow ;P