[13:13:26] <ebernhardson>	 \o
[14:51:46] <pfischer>	 o/
[14:54:21] <pfischer>	 ebernhardson: There’s a meeting concerned with reranking semantic search results in a couple of minutes (forwarded the invite). Would you be around to join?
[15:24:00] <ebernhardson>	 pfischer: sorry! i hadn't looked at my schedule this morning and ran an errand after the school run
[15:54:45] <pfischer>	 ebernhardson: no worries, Sucheta will post a summary to the semantic search channel later.
[16:35:29] <ebernhardson>	 hmm, feel like i'm perhaps getting lost in the weeds trying to finish off the model evaluation.  The thing is i've noticed that mdeberta does better on questions (who, what, etc.) but falls off on the llm rewrite of questions into keyword form.  Given the prevalence of keyword form that seems worth investigating, but then i've found the difference is perhaps in over-commit rate. 
[16:35:30] <ebernhardson>	 Essentially mdeberta fails to commit (answerable goes from 92.6% to 58.8%, non-answerable stays at 18%), while xlmr-distilled over-commits (answerable drops 85.5% to 77.2%, but non-answerable increases from 22% to 47%)
[16:36:20] <ebernhardson>	 since the plan is to fallback to lexical anyways, we aren't rejecting passages based on abstension, seeing if i can convince mdeberta to commit even when it thinks abstain is better
[16:37:19] <ebernhardson>	 i guess i'm wondering if perhaps xlmr-distilled is better since it at least keeps answering, while we see mdeberta decline to answer far more often under the keyword regime
[18:02:06] <ebernhardson>	 hmm, started to look into the *_titlesuggest age problem.  Realizing now that whatever was broken...it fixed itself.  Age increased without resetting between 8-14 and 8-18, but they all reset back to < 1 today without doing anything
[18:02:13] <ebernhardson>	 unsatisfying to say it's fixed, and i have no clue why :P
[18:29:16] <ebernhardson>	 Hmm, so it turns out we can ignore abstention for mdeberta,  if we take top_k=2 abstention as top result will still have a span as second-best.  Under this regime mdeberta and xlmr achieve the same score on keyword queries. I guess my summary is still go with mdeberta, but for the prod test we might consider the top_k=2 bit