[07:22:43] <dcausse>	 o/
[10:41:09] <dcausse>	 lunch
[11:32:01] <pfischer>	 o/
[11:32:12] <pfischer>	 dcausse: welcome back!
[12:52:00] <dcausse>	 thanks! :)
[13:01:04] <inflatador>	 <o/
[13:01:11] <inflatador>	 Welcome back d-causse!
[13:05:24] <dcausse>	 thx! :)
[13:24:42] <ebernhardson>	 \o
[13:26:07] <dcausse>	 o/
[13:35:00] <ebernhardson>	 dcausse: welcome back!  Have a little demo to help ponder how howlighting should work: https://people.wikimedia.org/~ebernhardson/mdeberta-highlights/
[13:36:56] <dcausse>	 ebernhardson: thanks! I did have a brief look but there are still things I don't quite understand in the demo was planning to focus on this next after having done a couple reviews
[13:37:45] <dcausse>	 from what I got is that they no longer want the model to abstain
[13:38:05] <dcausse>	 which is something IIRC the model is explcitly trained for
[13:38:49] <ebernhardson>	 dcausse: well, we have to decide what to do when the model abstains. We could still show the mdeberta highlight even when it abstains, i guess my question was how wrong is that.  I suppose i kinda feel like a longer sentence based highlight seems more fuzzy than a super narrow but wrong highlight, so perhaps better
[13:39:16] <ebernhardson>	 but we could just expand the mdeberta highlight to sentence bounds, or use the jaccard-similarity sentence selector, or use the cirrus prod highlighter
[13:40:54] <dcausse>	 yes... but if we fallback to a broad sentence highlight, what do we do if the passage is a single sentence?
[13:41:11] <ebernhardson>	 highlight the whole thing :P  Or maybe nothing at all
[13:42:09] <dcausse>	 same for short passages, I think I set up somehting to skip the forward pass on short passages (<40 tokens), is this something we want to relax and always try to highlight?
[13:42:57] <ebernhardson>	 hmm, i suspect if it's short enough we don't really need to highlight what it was.  40 chars is already the basic size of a highlight
[13:44:58] <ebernhardson>	 we can also just run with the mdeberta highlights, one of the first bits talked to product about was just always using the mdeberta highlight and that seemed reasonable to them.  But i was looking in the demo at the kinds of passages that get very low confidence, and it's right there isn't an answer (or maybe the query isn't a question)
[13:45:16] <ebernhardson>	 but we agreed to not use abstention to remove results
[13:47:29] <dcausse>	 oh I think I missed this idea of completely skipping a result if it can't highlight anything, yes that'd seem odd to give the highlighter such power
[13:48:43] <ebernhardson>	 there is also a separate question of if/how we can ship that confidence back to cirrussearch.  It seems like a potentially useful signal, but not sure what we would do with it.  If we could at least record it might be useful
[13:49:08] <ebernhardson>	 one option talked to peter about was maybe it could influence the ui, give some sort of low-cofidence marker
[13:50:16] <ebernhardson>	 but i suspect we can't thread that through opensearch
[13:50:25] <dcausse>	 I think it should definitely be recorded somehow, esp. if they attempt to infer no-click satisfaction
[13:50:47] <dcausse>	 yes I think it's more on the UI side to record that, or possibly in cirrus for the backend logs
[13:51:11] <ebernhardson>	 right, but how to get that back to cirrus if it goes through the opensearch highlighting api?
[13:52:01] <dcausse>	 cirrus can still read the highlights and detect when it transforms start/end markers to html spans
[13:53:04] <ebernhardson>	 sure, but can we carry the payload that this highlight had a confidence of 0.123?
[13:54:09] <dcausse>	 oh... sorry misunderstood this... yes I suspect that would need a dedicated response format... not trivial at all
[13:54:43] <ebernhardson>	 yea i'm not sure what to do with that...pondering
[13:55:37] <dcausse>	 well... we can run the inference request directly from cirrus, gives us way more liberty but at the cost of more moving pieces in the cirrus logic
[13:58:00] <dcausse>	 for post-analysis I guess we can always replay the logged data-set but if it's for making decision at search-time (from the client) this seems hard
[14:02:10] <dcausse>	 ebernhardson: for the demo you used a custom python script to run mdeberta or something on top of https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1330567 ?
[14:07:13] <dcausse>	 "jaccard sentence" is the idea that Santosh raised at T433872#12235709 ?
[14:07:14] <stashbot>	 T433872: Create a liftwing service to extract semantic search query highlights - https://phabricator.wikimedia.org/T433872
[14:10:51] <ebernhardson>	 dcausse: yea
[14:11:09] <ebernhardson>	 dcausse: for mdeberta i ran it through a custom runner, so different
[14:11:15] <dcausse>	 ok
[14:12:07] <dcausse>	 technically speaking "jaccard sentence" could still highlight nothing but surprisingly I can't find a single example in the dataset...
[14:12:21] <ebernhardson>	 oh and i added a followup to the liftwing one, ozge had a test run i couldn't reproduce, but it took centerpiece and started the highlight for piece, likely it was tokenized there.  The followup pulls it out to the next word boundary (and skips a set of langs that probably dont have spaces)
[14:13:58] <dcausse>	 yes saw it, thanks! completely missed that obviously multi-token words would not be properly highlighted
[14:14:34] <dcausse>	 kind of surprised that cirrus fragment is not at 100% but "jaccard sentence" is...
[14:16:35] <dcausse>	 or does it highlight the first sentence when no words match?
[14:19:02] <dcausse>	 one example is query: "forsaken roblox" result: Roboquest
[14:19:58] <dcausse>	 can't really see how it can match, unless it's mathing tokens and somehow it got "robo" out of roblox and roboquest
[14:24:08] <dcausse>	 sorry meant the token "rob" not "robo"
[14:35:03] <inflatador>	 might be 10-15 m late to search standup
[15:00:11] <ebernhardson>	 dcausse: hmm, i'm not sure about jaccard (claude implemented it).  will have to check
[16:06:53] <ebernhardson1>	 looks like the cirrus-toolbox report for lagging indices was alredy reviewd.  merged and moved ticket to done
[17:16:32] <dcausse>	 dinner
[18:56:10] * ebernhardson1 is copying repos back from the loaner laptop to this one...for some reason i thought it would be fast. should have used ssh and tar instead of scp...