[01:57:32] <ebernhardson>	 hmm, network usage going out of codfw cluster dropped back down, but the snapshot shows everything still in progress with no complete shards :S
[08:36:17] <zpapierski>	 Nice, surprisingly, I actually have a use case for that https://usercontent.irccloud-cdn.com/file/GqE6ot0R/image.png
[09:01:54] <zpapierski>	 errand
[09:02:20] <gehel>	 ejoseph: meeting? https://meet.google.com/ukb-kgxq-gvq
[09:40:04] <ejoseph>	 gehel: i am ready
[09:40:24] <gehel>	 https://meet.google.com/kct-zqiy-mnz
[10:41:55] <ejoseph>	 zpapierski: can we continue plugin update?
[10:42:11] <zpapierski>	 need to grab a coffee first, but yeah
[10:42:22] <zpapierski>	 give me about 15min
[11:08:49] <gehel>	 Lunch
[11:10:25] <dcausse>	 lunch 2
[11:16:32] <ejoseph>	 https://gerrit.wikimedia.org/r/c/search/extra/+/711226
[11:17:01] <ejoseph>	 https://gerrit.wikimedia.org/r/c/search/extra/+/711226/3/extra/src/test/java/org/wikimedia/search/extra/regex/expression/ExpressionTest.java
[11:49:24] <zpapierski>	 lunch 3.14
[12:05:13] <kostajh>	 \o is search traffic still routed through codfw? or is it back on eqiad?
[12:07:56] <kostajh>	 For context, I'm looking at T296376 and suspect the switch to codfw had something to do with what I see on the graphs there
[12:07:56] <stashbot>	 T296376: Investigate rendering speed variations starting around 10 November - https://phabricator.wikimedia.org/T296376
[13:09:47] <gehel>	 kostajh: we're still routing search traffic to codfw
[13:10:01] <kostajh>	 gehel: ack, thanks
[16:04:00] <gehel>	 Wednesday meeting anyone? Am I in the wrong Meet?
[16:04:34] <zpapierski>	 dcausse: lets talk on  wednesday meet 
[16:46:34] <ebernhardson>	 kostajh: i looked over it, i would be hard pressed to attribute an increase from ~2s to ~10s due to us moving traffic,  the latency we added is 40ms per request
[16:48:19] <kostajh>	 right, must be something else going on. seemed suspicious with the timing, though.
[18:56:22] <cbogen_>	 ebernhardson: not sure if you saw the request for an update from Marshall on T295316 in Slack, I know you're not on Slack much
[18:56:23] <stashbot>	 T295316: Add an image: pre-deployment model refresh - https://phabricator.wikimedia.org/T295316
[19:00:02] <ebernhardson>	 cbogen_: i updated it yesterday with all relevant details, the only thing to add would be the thing that was supposed to be done in an hour is done now
[19:00:26] <cbogen_>	 ebernhardson: okay I'll let Marshall know, thanks! so it's all complete?
[19:00:32] <ebernhardson>	 cbogen_: ya
[19:00:42] <cbogen_>	 cool, thanks!
[19:04:47] <ebernhardson>	 turns out i don't get pings in threads unless i've joined the thread. Overall i'm not a fan of slacks threading model, it seems to hide everything i usually see in irc :P
[19:05:01] * ebernhardson will probably end up writing something to auto-join all threads or some such
[20:07:36] <ebernhardson>	 so unexpected, but when restoring the index to eqiad it restore the index alias as well
[20:07:46] <ebernhardson>	 i suppose i should have tested that
[20:09:49] <ebernhardson>	 general cirrus logging quiets down a ton at that point, from ~20k logs/10 minutes down to ~50
[20:15:17] <ebernhardson>	 started up the catchup routine, it's going to drop a bunch of jobs into the queue's and probably backlog other writes for a little bit
[20:25:23] <ebernhardson>	 doesn't seem too bad, it's pushing ~100 pages/s to be indexed but the queues are all running fine. I suppose saneitizer might usually use more capacity than that
[20:54:56] <ebernhardson>	 everything should be restored. Could move traffic back to eqiad