[09:21:14] errand, back in 20' [11:09:13] lunch [13:35:22] o/ [14:14:10] \o [15:02:07] ryankemper: DPE SRE meeting is starting: https://meet.google.com/rnb-jtio-dcy [15:45:14] some of our code somments are amusing.."We found a redirect. Great. ... Also, it returns null on self redirects. Great!" [15:47:28] also we seem to not change things all that much, that comment will be a decade old in a couple months [15:56:19] * ebernhardson randomly wonders if some sort of chronology protector magic (record master replication position after commiting the writes, then followups wait for replica to see that position) [15:56:48] if some sort of magic could resolve some of our annoyances around not knowing if we should read from master or slaves when generatinga doc [15:57:12] unfortunately that only works within the same DC though [16:03:15] workout, back in ~40 [16:40:40] back [17:09:28] lunch, back in ~1h [17:54:06] back [18:51:02] * ebernhardson couldn't exactly figure out how the upload bug was happening...but managed to convince someone else to look and figure it out :) T344285 [18:51:20] * ebernhardson also thought there was a bot that expanded Txxx numbers here [20:13:43] ebernhardson: I just merged https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/833861/. anything I should check as far as post-deploy steps? not sure if there's much to check [20:21:40] ryankemper: hmm, so the fun thing is that will only effect newly created indices [20:21:48] but! you're in luck. There is a ticket to reindex all wikis everywhere :) [20:22:27] I was pondering when exactly i should start that, I suppose the important question would be are we going to be restarting servers this week? [20:23:02] Last time i started a reindex on a friday to avoid restarts, but then some of the reindexes were bad and we broke search for a weekend. So I guess tuesday is as good a day as any to kick it off [20:24:04] i suppose for completeness, if we weren't going to be recreating the indices we would need to manually apply those changes via elasticsearch's index settings update api [20:24:28] we have a "just mapping" option for the index updater, but i dont think we have a "just settings" option. Maybe it would be something to add [20:27:18] ebernhardson: not aware of any plans to do es restarts so now seems like a fine time [20:27:53] ebernhardson: okay so basically there's already an in-flight ticket to reindex all wikis so we'll just let the new index changes get applied when that reindex kicks off? [20:28:15] ryankemper: yea, i can kick that off this afternoon. Last time around i wrote some bash/tmux script that does it all. Just gotta kick it off [20:30:52] I'm never quite sure where these scripts go...it could go in CirrusSearch's scripts/ directory, but it's a bit wmf specific (mwscript, dblist stuff, etc.). Puppet is the other place those things sometimes end up, but is often inconvenient [20:37:20] reindex started, it will take a little while to get to enwiki. It mostly goes alphabetical [21:37:40] ryankemper reimaging wdqs1012/13 in tmux window 'wdqs' on cumin1001. Should be done in ~30m, if you have time, can you run the deploy and data xfer cookbook? docs at https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service#I've_got_a_new_WDQS_host,_how_do_I_get_it_ready_for_production? [21:37:51] deploy/cookbook cmds: https://phabricator.wikimedia.org/P50588 [21:38:04] inflatador: sounds good, thanks for the links [21:38:23] np, thanks for finishing those off