[08:24:29] <atsukoito>	 good morning!
[09:30:14] <dcausse>	 hi!
[09:32:30] <atsukoito>	 dcausse: hiya!
[09:33:02] <atsukoito>	 i was thinking.. we cant use https://wikitech.wikimedia.org/wiki/Mw-experimental to debug in prod
[09:33:11] <atsukoito>	 *we can
[09:37:11] <dcausse>	 atsukoito: currently using shell.php with mwscript but struggling to find the cause, the query works... suspecting some data issues but still not 100%
[10:25:00] <atsukoito>	 dcausse: I seem to be able to reproduce the error in shell.php, https://phabricator.wikimedia.org/P94232, could you please review if what I did is correct?
[10:25:18] <atsukoito>	 I was walking over the trace https://performance.wikimedia.org/excimer/profile/3b0dcad2421b0a78
[10:26:28] <dcausse>	 atsukoito: yes sounds good, I can repro as well, the issue is around the two queries run, the first one fetches the matching translation, the second fetches by id on the target language
[10:27:03] <dcausse>	 the first one works in dse-k8s, I see matching translations but the ids query on the target language returns nothing
[10:27:15] <dcausse>	 seems like ids do not match
[10:27:45] <dcausse>	 id is like "collabwiki-Translations:BOTE2022/April Announcement/Email/2-87868/en"
[10:28:01] <dcausse>	 then it strips the '/en' to replace with '/de'
[10:28:18] <dcausse>	 that is where it's not found...
[10:29:09] <dcausse>	 it's why I suspect some "data" issues if the ids are not well grouped
[10:29:36] <atsukoito>	 could it be that the maintenance script missing something?
[10:30:25] <atsukoito>	 what if we drop the data from test server, then populate a single record via web ui, and then try to fetch it?
[10:31:24] <dcausse>	 I don't enough about translate sadly...
[10:31:35] <dcausse>	 the size of the index seems to match so that's surprising
[10:35:16] <dcausse>	 this is one missing for instance: curl -d '{"query": { "term": {"_id": "wikifunctionswiki-Translations:Wikifunctions:Status updates/4-86945/de"}}}' -HContent-Type:application/json localhost:6518/ttmserver/_search | jq .
[10:35:33] <dcausse>	 it's found on the prod cluster
[10:38:42] <dcausse>	 I mean it's there but the id is not what it should be: curl -d '{"query": {"bool": {"must":[ { "match": {"localid": "Translations:Wikifunctions:Status updates/4"}}, {"match": {"language": "de"}}]}}}' -HContent-Type:application/json localhost:6518/ttmserver/_search | jq .
[10:39:11] <dcausse>	 "wikifunctionswiki-Translations:Wikifunctions:Status updates/4-88357/de" vs "wikifunctionswiki-Translations:Wikifunctions:Status updates/4-86945/de"
[10:39:36] <dcausse>	 the "88357" is supposed to be the rev_id
[10:39:53] <dcausse>	 but it's getting into the weed of translate internals and I'm now lost
[10:41:22] <dcausse>	 will write that in phab and ask the translate team what this means
[10:48:01] <atsukoito>	 i added difference to the task https://phabricator.wikimedia.org/P94237
[10:51:36] <dcausse>	 thanks!
[10:54:28] <dcausse>	 added a quick comment too
[10:56:06] <dcausse>	 lunch
[11:17:19] <atsukoito>	 dcausse: funny, if I get the IDs we got from the first query from Elastica, they are present in K8S
[11:46:09] <atsukoito>	 i'm off for lunch, let's have a quick call maybe after?
[12:16:55] <dcausse>	 atsukoito: sure
[12:39:23] <ebernhardson>	 \o
[12:39:51] <dcausse>	 o/
[13:07:53] <inflatador>	 <o/
[13:08:12] <ebernhardson>	 \o>
[13:11:05] <dcausse>	 o/
[13:19:30] <inflatador>	 FYI I ended up configuring envoy TLS termination for cirrus in beta cluster. It was pretty painless, so I'll probably look at enabling it in relforge/cloudelastic soon
[13:48:41] <inflatador>	 dcausse I'm taking my son to summer camp, may be a little late to pairing
[13:48:52] <dcausse>	 inflatador: np!
[14:09:46] <inflatador>	 OK, back
[16:56:04] <dcausse>	 dinner
[17:14:44] <atsukoito>	 see you tomorrow