[00:19:16] * ebernhardson now wonders if the reindexing script should live in puppet, or maybe just in the deployment-charts service directory
[10:45:49] <dcausse>	 lunch
[15:12:06] <inflatador>	 <o/
[15:13:45] <dcausse>	 o/
[15:24:07] <inflatador>	 Flink buckets are cleaned up, but we're still over the 50G alert threshold...going to bump that up to 100G shortly
[16:06:10] <ebernhardson>	 \o
[16:07:56] <dcausse>	 o/
[16:09:56] <topranks>	 inflatador: thanks for banning/depooling those hosts ahead of our maintenance in B2 (T355868)
[16:09:58] <stashbot>	 T355868: Migrate servers in codfw rack B2 from asw-b2-codfw to lsw1-b2-codfw - https://phabricator.wikimedia.org/T355868
[16:10:06] <topranks>	 we're all done now so feel free to bring them back into service
[16:10:07] <topranks>	 thanks!
[16:35:04] <inflatador>	 topranks ACK, thanks for the heads-up...glad to hear everything worked OK
[17:22:07] <inflatador>	 lunch, back in ~60
[17:46:02] <jfolv>	 ebernhardson: Just wanted to thank you again. My jobrunner worked like a charm, and search is updating correctly now. I'm going to keep that running for the future.
[17:46:16] <ebernhardson>	 jfolv: awesome! glad to here it's fixed up
[18:36:19] <inflatador>	 sorry, been back
[18:57:05] <inflatador>	 small CR for raising the object storage alert threshold: https://gerrit.wikimedia.org/r/c/operations/alerts/+/1005791
[19:08:16] <inflatador>	 getting some "elastic not indexing" alerts from the hosts I just unbanned...looks like  false positives but still checking
[20:26:02] * ebernhardson wonders while updating the wikitech:Search page what exactly we should do for re-generating an index from the database
[20:26:25] <ebernhardson>	 we wouldn't really do that on a big wiki, but we've certainly used that for example when a new wiki is created but the index isn't, and we create the index days later
[20:28:08] <ebernhardson>	 i suppose we might want a way to make ForceSearchIndex.php work even when normal writes aren't flowing there
[20:52:59] <inflatador>	 Maybe it's better to recommend snap/restore index from the other DC?
[20:53:59] <ebernhardson>	 it's a good thing to have available, but i would be very uncomfortable if there is no way to regenerate the search index from the database :)
[20:55:41] <inflatador>	 Yeah, I guess you're already saying restoring from SQL is a last resort...maybe call out snap/restore above that in the docs, as long as it's actually appropriate for the use case
[20:57:55] <inflatador>	 I guess the main use case for regenerating would be what you just said, new wikis?
[20:59:51] <ebernhardson>	 well, thats the most recent time i can remember using that :) I can't think of other times we recreated indices from scratch, we always keep them going and fix up with backfills and saneitizer
[21:00:11] <ebernhardson>	 although i noticed today the docs for backfilling a small wiki say to blow it away and rebuild from sql :)
[21:00:27] <ebernhardson>	 we've kinda unified and do the same thing everywhere instead
[21:05:04] <inflatador>	 We are in kind of a weird spot...we almost never do this, and it would be impractical/impossible to do on a wiki of any size. But we still need it documented in case the need does come up
[21:05:34] <inflatador>	 here's where I throw up my hands and move on to the rest of the doc ;P
[21:46:44] <ebernhardson>	 :P
[21:50:52] <inflatador>	 quick break, back in ~20
[22:08:18] <inflatador>	 back