[08:20:26] re: commons indices missing, hm I guess we should reroute traffic to codfw again? [08:22:34] these indices are most probably empty [08:44:50] manually recreated the geo field to save the switch [08:45:20] because I just realized that we can't route only commons traffic [09:21:16] Ahh, I forgot that check in is in the morning [09:21:24] Will miss it [11:40:53] lunch [12:48:08] lunch [13:19:16] ejoseph: let me know once you're back (I'll probably be out for lunch soon as well, though) [13:35:45] dcausse: are you actively working on this atm - https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/739916/5 ? [13:36:46] zpapierski: I'm working on the next part after this, it's -1 by jenkins because one test depends on https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/737429 [13:37:13] ah, so it's up for review, thx [13:37:48] this patch should not change much unless I discover something completely wrong with this approach in the coming days [13:39:37] that's cool, judging on a size, it will take me some time anyway :) [13:53:45] lunch break [14:38:26] ejoseph: I'm back (at least until 4PM, meetings armageddon ensues afterwards) [16:02:44] ryankemper, ebernhardson: triage meeting: https://meet.google.com/qho-jyqp-qos [17:38:35] snapshots into swift started from codfw, also moved cirrus traffic to codfw [17:38:50] thanks! [17:40:11] archive took 1.8 minutes, slightly faster :) [17:40:26] i guess thats cheating though, it's always tiny [17:58:18] annoying limitation, you can't have multiple snapshots running. You can shapshot 1 index with 30 shards. But you can't snapshot 2 indexes with 4 shards each because "it's too hard on lucene freezeing all those segments" [17:59:07] to be fair they built this with a different use case in mind, it supports incremental snapshots and you aren't intended to have them run all that long [18:04:59] strange... seems pretty limiting esp for a large cluster [18:52:48] dinner [20:20:04] * ebernhardson waffles between minimum viable changes to fit wcqs into the notebooks, and making things more coherant... [20:20:13] so many things have the name wdqs in them :P [21:55:09] Ugh. I just discovered that the Lucene Bengali normalizer uses a phonetic algorithm (like Soundex) to "normalize" Bengali tokens. That seems like a really aggressive way to find misspelled words. Thanks to tanny411 for her super helpful insight that made it a lot easier to figure out what's happening.