[08:24:06] good morning! [08:24:42] hi! [10:33:14] lunch [12:54:57] \o [12:55:56] o/ [13:04:53] o/ [14:16:49] o/ [14:17:53] .o/ [14:23:42] Trey314159: I am discussing our search quality metrics (zero results rate, click position) with Test Kitchen. Right now, they could give us up to 50% of all traffic. While this might be sufficient for enwiki, I assume, this will be insufficient for smaller wikis. Do you recall any notorious wikis we had trouble with gathering enough data in the past? TK would want to support us in setting up a first experiment, so we could as well [14:23:42] run this one of the problematic wikis. [14:38:21] claude can be amusing at times: Good news and bad news: the space is more crowded than you'd expect, but there's a real gap exactly where you're aiming, and the WMF is actively working in this area right now — which is worth knowing before you build [14:39:29] LOL, very comforting vapidity there [14:48:34] * ebernhardson idly wonders how you might take the cirrus AST, the opensearch query, and the per-doc explain, and somehow align them all to report on a per-AST-node basis what matched where [14:48:59] sadly we don't directly build the query from the AST, so it might be awkward :P [14:58:57] separately wonder how crazy a "post-analyzed" option for cirrusdoc would be. General idea would be to provide all the mapping sub-fields with varied analysis. [14:59:46] ouch, seems like you want to build an highlighter in php :) [15:01:37] lol, actually i was pondering a tool where it could tell you given a query and a page why the page does or does not match the query. But to do that reasonably we probably need the tokens [15:02:07] explain might be able to bridge the gap to explain why something matches, but it misses all the filter queries and doesn't explain a no-match [15:08:37] yes that seems tricky to get right, something we don't use but could possibly be usefull is named-queries where you can tag parts of the queries and get a hint in the response whether that part matched [15:09:04] does not solve the token/highlighting issue tho [15:09:10] so crazy idea: Take the AST, transform each node into an elasticsearch query with explain, and run them all in an msearch so we have separate explains for each AST node. Would also need an id filter to query a single page. [15:09:16] but still doesn't help explain not-match [15:09:38] yea named queries would at least make things alignable [15:10:22] i guess we would know exactly which AST nodes match and don't match [15:10:47] we just wouldn't know why that AST node doesn't match [15:11:34] how would you explain why something does not match? [15:11:52] i don't know, but we would know that the KeywordNode with intitle didn't match, and could surface that info [15:12:54] ah yes, the named-query could possibly help for these building blocks [15:13:03] but [15:13:29] but IIRC named-queries are rerun during the fetch-phase so it's not entirely free [15:13:57] also tricky to get it working if the tool is external to cirrus [15:14:10] for a debug only query i suspect that's probably ok. My AST idea would have to issue parallel queries for each AST node anyways. [15:14:37] ah ok, so that must run in cirrus or possibly in cloudelastic if you don't want that in cirrus [15:15:06] oh, i hadn't thought about cloudelastic. Maybe that would work too [15:15:31] it's not a perfect replica but could allow to iterate a lot quicker, I don't know... [15:16:20] huh, i didn't realize. OpenSearch has two different explain apis. The explain in a search query does what we have now, there is also a //_explain/ that accepts a query and does report negatives. Or so it claims, testing what that does [15:16:47] oh [15:21:22] yea, it includes "Failure to meet condition(s) of required/prohibited clause(s)" [15:21:34] still not entirely sure how to align, but that might be helpful [15:30:16] interesting [15:32:07] perhaps the AST could a unique ID that is tagged in the query via "_name" or some other custom tag if you use cirrusDumpQuery, but unsure what context //_explain/ is returning back to you [15:33:21] dcausse: it only returns two things, a matched: bool, and an explanation: which is the serialized Lucene explain [15:33:46] i'm thinking maybe we can skip the alignment by running each AST node separately, so we have an explain per-AST node [15:34:34] slower since there in no parallel, but for debug should be irrelevant [15:36:11] although, hmm, do we actually have the AST->OS query parts done? Been so long maybe not... [15:39:14] i guess the totally hacky way would be to use the offsets the AST reports and run the regular query builder on those substrings [15:40:41] oh right the AST->OS is not available :( [15:41:19] dcausse just saw you merged that labs patch, thanks! [15:41:26] it's partly in this patch chain https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/550891 but it's way too old [15:42:28] I need to get this project restarted... was lacking motivation to takle this again but perhaps with claude I could get back into it more quickly [15:42:34] pfischer: enwiki is so big that we might not need even 50% of the traffic to get meaningful values for metrics like ZRR, CTR, or click position (though we want to keep track of the sample rate for each wiki so we can extrapolate back to the full set for metrics that scale with the sample size, like qps). [15:42:43] There's nothing that rises to the level of "notorious", but wikis with medium-sized query volume suffer from being scaled unnecessarily. Really small volumes (~!00/day) are too small for fine-grained (e.g. daily or hourly) metrics and are often overrun by bots and spam. [15:42:45] In the dim and distant past, I think we had trouble with systems that could only sample at one rate for all wikis, making the stats for medium wikis less useful, and/or systems that didn't know the per-wiki sample rate, making all the scale-dependent metrics less useful. [15:43:35] inflatador: np! it's still populating the indices [15:45:41] dcausse ACK, I used an extremely minimal opensearch.yml ( https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/opensearch/files/opensearch-deployment-prep.yml ) , so if there are optimizations I should add from prod LMK [15:46:00] just remember the nodes only have 8 GB RAM ;) [15:46:21] yes... we'll see how it goes, hopefully that's enough :) [15:48:34] but if we had 2 replicas in the old cluster with 3 nodes that should be almost identical, old nodes had 8G as well [16:11:12] dcausse: release on friday? [16:25:16] atsukoito: you mean https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/1304058 ? we'd have to justify that's an important for ship for a friday backport [16:25:28] * dcausse can't write [16:25:50] I mean I'm not sure this fix qualifies for a friday (out of window) deploy [16:26:41] but if this gets merged in master today or tomorrow we can definitely do a cherry-pick and ship that monday morning [16:26:45] ah, I forgot we don't have deployment windows on friday [16:27:14] sure then! [16:27:30] ack, I'll monitor the patch and ping you once ready [16:28:25] good! if we backport it on Monday morning, we can surely switchover on Tuesday and unblock the upgrade [16:45:30] sure! [16:45:32] dinner