[07:03:43] hiya! [07:49:54] atsukoito: can't see any errors either and the index in both dc keeps growing [07:50:11] I guess that's good news, well done! [07:51:09] my next plans: 1) i'll try figure out patch for the config to split test and prod servers, so the testwiki will go to ttmserver-test and the prod to ttmserver, and 2) will test how prod index regeneration works on test wiki using the script [07:51:27] sounds good! [07:51:34] after all this we will be ready for switchover [07:52:04] the prod index will take for ever to populate via ttmserver-refresh and won't be doable during the scap test phase [07:53:12] we can probably do 2 steps switchover then: adding new servers and populating it, and then removing old servers [07:53:33] or i can populate the index on ttmserver-test, and then copy it maybe? [07:54:10] atsukoito: hm.. possibly if you have enough capacity but the copy may be tedious [07:54:29] i see, i'll research it [07:55:27] 1/ create an empty ttmserver index with proper settings/mappings in the prod dse-k8s opensearch cluster, 2/ add them to the config, 3/ populate, 4/ switchover [07:56:50] agree, it is the easiest, yes [07:58:01] is there any wiki for the plugin itself? i want to add some maintenance notes there afterwards [08:03:29] atsukoito: you mean a doc for the Translate extension? [08:03:47] more like a wikitech entry.. [08:04:38] hm looking [08:04:57] if no, i'll probably create a small page just to link to the somewhere here https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/OpenSearch-on-K8s and to keep notes about operations [08:05:03] the doc is at https://www.mediawiki.org/wiki/Extension:Translate but it's generic [08:05:28] yeah, i won't put the details of the ttmserver deployment here [08:05:54] we can ping niklas or abijeet they might know if something exists [08:12:17] gonna ping him on slack [08:17:41] gonna create something like https://wikitech.wikimedia.org/wiki/Search/TranslateExtension [08:19:01] atsukoito: unsure if it has to be under "/Search", it's not strictly owned by search [08:20:02] what if Search/TTMServer? [08:20:19] if not, i'll just put it under OpenSearch-on-K8s/ [08:21:21] atsukoito: if this doc is mainly targeted at operations on the opensearch cluster related to the translate extension I believe that putting that under "Data Platform/Systems/OpenSearch-on-K8s" might be accurate? [08:21:48] alright [08:31:27] https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/OpenSearch-on-K8s/TTMServer [08:32:36] atsukoito: awesome, thanks! [10:03:06] lunch [12:46:13] \o [12:56:41] o/ [13:26:16] o/ [13:52:25] dcausse, ebernhardson, Nikerabbit: I'll attempt to populate `ttmserver` index on eqiad to estimate how long it is going to be [13:52:45] nice! Hopefully not too long, it's not a massive index, but not sure how long the mw side takes [13:53:08] on `eqiad-test` to be precise [13:55:31] `mwscript-k8s --sal --attach -- extensions/Translate/scripts/ttmserver-export.php --wiki=default --ttmserver eqiad-test` [14:00:07] didn't work. `Fatal error: no version entry for `default`. in /srv/mediawiki/multiversion/MWMultiVersion.php on line 701` [14:00:28] atsukoito: i'm not sure the exact details, but i would probably try `testwiki` instead of `default` [14:00:31] atsukoito: I believe you have to run it for multiple wikis (wikidatawiki, metawiki and possibly others [14:00:33] ) [14:01:08] ahh, I see. lemme see what values are there then [14:01:38] I suspect it has to be all wikis that may have translation memories [14:02:06] yea makes sense, might just need a foreachwiki [14:02:37] there's perhaps a dblist for it? looking [14:03:28] yeah, i wanted to ask if we can select distinct from the database for all the wikis that are using translate.. [14:03:33] yes there's a "translate" dblist [14:04:14] atsukoito: on a deploy server: expanddblist translate, mwscript-k8s might have an option to use a dblist I think [14:18:26] doh, i guess i didn't update the patch. I do have a bunch more pulled into IndexablePageResolver now, i just hadn't gotten that far yet. Was trying to find parts i can pull out and test more directly [14:22:01] oh indeed found it surprising that it had only two ifs and started to read it before ForceSearchIndex and was a bit confused [14:22:50] yea 4 bools and 2 if conditions is probably not enough to justify a class, but hopefully a bit more and it'll make sense [14:23:50] yeah, adding `--dblist translate`and removing `--wiki...` should do the trick [14:23:54] take two [14:33:34] dcausse: somehow the `ttmserver` index on eqiad-test doesn't have one mapper, the diff between `eqiad-test` and `codfw-test` shows next [14:34:17] atsukoito: what do you mean by "mapper"? [14:34:32] https://www.irccloud.com/pastebin/weYDNJBw/ [14:34:46] sorry, `mapping` [14:35:31] hm... [14:35:34] the error I got from the script is next https://www.irccloud.com/pastebin/e4LULBEW/ [14:35:37] yea opensearch/elasticsearch have unexpected naming..i would call their mapping a schema, but it's what we get :) As to those changes, hmm [14:37:06] We only have this specific difference on a single index, `ttmserver@opensearch-ttmserver-test.svc.eqiad.wmnet`. all other indices on k8s has this field. do you think i can just drop it and re-create it again? [14:37:39] the index settings defined in ElasticSearchTtmServer appear to have the prefix analyzer defined, turn it off and turn it back on seems a reasonable first attempt [14:37:40] (or could it be that it happened because I run mwscript on it) [14:37:45] atsukoito: it's very possible that one of these was created by me manually and I just copied the minimal mapping to make the query not fail [14:38:01] cheking the date [14:38:25] it could be an order of operations, i see in ElasticSearchTtmServer they send the mapping and index settings separately (cirrus does too, i don't know why. Historical artifact?) [14:38:39] so that could happen if it sent the mappings before doing the index create with settings [14:38:39] - "creation_date": "1779093959583", + "creation_date": "1781004538560", [14:39:50] Mon 18 May 09:45:59 BST 2026 vs Tue 9 Jun 12:28:58 BST 2026 [14:39:56] looks like it all gets done in the same method though, shouldn't be able to send separately. Maybe david's theory that it was hand created holds [14:40:12] ah yes, if you look at the settings https://opensearch-ttmserver-test.svc.eqiad.wmnet:30443/ttmserver/_settings it has nothing, pretty sure it's me creating the index to quick fix T426467 [14:40:13] T426467: Translation memory is not working on multiple Wikimedia wikis - https://phabricator.wikimedia.org/T426467 [14:41:04] I think I did not bother creating the index the right but copied over the necessary fields and ignore the analysis condif [14:41:09] s/condif/config [14:41:32] so yes you should definitely recreate the one in eqiad using ttm-export [14:42:11] I'll proceed with dropping the index and creating it with `--clean` [14:42:31] it should have a refresh or recreate IIRC [14:42:36] option I mean [14:45:14] seems like `--reindex` [14:45:58] i'll do both to just re-create an empty index, and then run dblist [14:46:50] sure [14:46:56] no, it doesn't drop the index, will issue manual `curl -X DELETE` [14:47:15] `curl -X DELETE https://opensearch-ttmserver-test.svc.eqiad.wmnet:30443/ttmserver` [14:48:01] sounds good [15:28:01] yeah, and now it creates the indices with replica factor 2 :+1: [15:29:02] `collabwiki Error while creating TtmServer eqiad-test: No configuration for name 'eqiad-test'` [15:29:04] hmm! [15:33:23] ??? [15:34:14] atsukoito: possibly a mistake in the dblist [15:36:32] yeah, most probably. skipping this one and proceeding with `--local_dblist` [16:34:55] dinner [18:50:58] * ebernhardson wonders if archive contains redirects...probably?