[10:28:52] lunch [12:33:07] break [13:37:58] relocating [15:01:17] \o [16:58:14] o/ [17:31:29] i was thinking of cscott, he gave the 'Wat!' talk about wikitext parsing [18:06:24] Hi all, I need to re-test the initial elasticsearch index creation and search index bootstrapping as described in the CirrusSearch README. Even though my wikis' indexes already exist in AWS ES, would it be sufficient to just re-run the same commands or would I need to first delete all of the indexes? [18:07:17] FWIW I need to re-test to determine the approximate times it would take for the initial indexing so I can estimate the maintenance downtime requirements for my live migration in a couple of weeks. [18:19:56] justinl: hmm, probably you just need to delete the indices manually. We have a script that we use for that in CI, called nukeAllIndexes, but it's not really for prod use :) [18:20:15] justinl: hmm, actually maybe we do have some options for that, there is a start over setting. lemme double check [18:20:48] ebernhardson cool, thank you [18:22:27] justinl: looks like you should be able to issue --startOver to UpdateSearchIndexConfig.php (probably --startOver --indexIdentifier now) and it will delete the index in place for you [18:23:44] doesn't look like that does anything else, it simply gates the choice between quit or delete if the index alias already exists [18:24:32] Great, thanks. I'll check it out. [18:27:57] The --help output says that --startOver will blow away the specified index and recreate it with no data. Do you think that recreation will make much difference in the time it will take for the subsequent UpdateSearchIndexConfig and ForceSearchIndex commands? [18:38:38] It's saying that I should run with --reindexAndRemoveOk [18:40:49] justinl: depends what you want to measure, if you are measuring a full reindex from the database, probably having an existing index wont make much of a difference [18:41:19] tbh full reindex from database scares me, we haven't run one in 5+ years :) [18:42:04] Yes, basically I'm trying to redetermine how long a clean index will take (this is on my dev wikis) for when I do my live migration. The live ones haven't had anything done yet other than I built the AWS ES cluster, no indexing though. [18:42:51] ahh, in that case it doesn't really matter what you do with the indexes, --reindexAndRemoveOk will copy the data from your old index into the new one, --startOver will skip that and just delete the old index [18:42:57] What about it scares you? The wikis will be down with our maintenance page up, so load won't be an issue. And these are brand new indexes. Anything you think I should be further aware of that I might not already be? [18:43:06] if anything, reindex will take longer as it needs to copy [18:43:31] justinl: not for you, but for us :) Estimates are that it would take more than a month to full-rebuild the wmf search indexes from the databases [18:43:40] When I ran the --startOver --indexIdentifier now is when it said I needed to use --reindexAndRemoveOk [18:43:56] Nice. That would royally suck, I'm sure. [18:44:19] Another team just recently had to move several TB of data, it took about 2.5 months. :P [18:44:20] justinl: hmm, thats odd. sec [18:45:04] ``` [18:45:13] ``` Validating es_wikidb_gw2_content alias...cannot correct! [18:45:14] The alias is held by another index which means it might be actively serving [18:45:14] queries. You can solve this problem by running this program again with [18:45:15] --reindexAndRemoveOk. Make sure you understand the consequences of either [18:45:15] choice.``` [18:45:32] Validating es_wikidb_gw2_content alias...cannot correct! [18:45:33] The alias is held by another index which means it might be actively serving [18:45:33] queries. You can solve this problem by running this program again with [18:45:34] --reindexAndRemoveOk. Make sure you understand the consequences of either [18:45:34] choice. [18:46:04] Argh, every freaking chat system has its own syntax for everything. So annoying moving from one to another.... [18:46:21] hmm, oddly that message is only gated by the reindexAndRemoveOk flag in latest, lemme check REL1_34 [18:49:12] To move forward, i suppose i would go with --reindexAndRemoveOk. It looks like --startOver no longer works as expected, perhaps not surprising because it's not often used functionality [18:49:30] Ok I'll give it a whirl and report back on my findings... [18:50:29] It's running, small db, only a few GB IIRC, so it shouldn't take too long. [18:52:28] I created a subtask for just our team's portion of the "migrating prometheus alerts to alertmanager" task: https://phabricator.wikimedia.org/T289077 [18:52:32] And then I moved the parent task (https://phabricator.wikimedia.org/T281454) to watching/waiting on our workboard since our work is now captured in the above subtask [18:53:51] cool. another new thing to learn :) [18:54:20] (not that i ever really learned icinga...that thing is a beast) [18:56:35] ebernhardson It took about 3.5 minutes and completed successfully. Now I'll run the README commands for generating the initial indexes and see how long that takes. IIRC it was around 10 minutes the first time. [19:02:48] TIL that “Wiki” is a Hawaiʻian word that means “quick” (per https://gist.github.com/Trey314159/a9c6332b5a3b0e55636c3c930fea6d66#file-wmf-programming-task-L30) [19:15:10] for some reason it stuck in my head...so i looked up that old talk. No video, but slides: https://upload.wikimedia.org/wikipedia/commons/0/08/Templates_are_dead%21_Long_live_templates%21.pdf [22:03:32] ryankemper: even more fun, Ward picked up the name from the airport shuttle at HNL. :) https://en.wikipedia.org/wiki/WikiWikiWeb#History [22:04:25] bd808: neat history! [22:36:34] random insights from looking over ~5 hours of user-perceived update latency metrics...it's often quite good, under a minute. But there were two time periods where it climbed up to ~5 minutes before coming back down