[06:26:19] running a nfa engine on a user provided regex sounds dangerous to me [07:33:14] actually it's not really a nfa engine, it determinizes lazily, if the failure mode is similar for pathological regexes why no? but not sure that we should run both (run the new nfa approach only when failing the full determinization) [07:38:46] but really hard to know, in the highlighter case we run the regex on every matching docs, while in the regex term query it's applied once per segment on the term dict [07:40:04] so reusing the dfa actually makes a lot of sense for insource and highlighting [07:42:25] sigh... might have missed the initial context from Erik: "it checks for a leading wildcard and uses an optimization if found.", seems like it's totally unrelated to insource & highlighing regex :) [09:40:12] dcausse: I have rechecked both ebernhardson patch to trigger the Java 21 jobs which are running at https://integration.wikimedia.org/ci/job/search-extra-maven-java21/1/console and https://integration.wikimedia.org/ci/job/search-extra-analysis-maven-java21/1/console [09:40:21] thanks! [09:40:47] search/highlighter does not have an open change, maybe it is pending on Erik machine or he did not have a chance to start looking at it [09:40:52] or maybe it just passes fine on Java 21 [09:41:03] hashar: yes Erik is on it [09:41:04] hopefully the builds will not timeout (there is no maven cache yet) [09:41:19] and of course the Maven output does not have color grr [09:41:38] seems to run fine so far [09:42:45] extra-analysis is V+2 \o/ [09:42:54] extra is bigger so slower [09:43:38] will +2 extra-analysis so see post-merge runs [09:52:48] postmerge build success [09:53:10] hashar: all seems fine, thanks! [09:59:17] and because maven does not emit colors I went to file a task to enable that https://phabricator.wikimedia.org/T436915 [09:59:30] which really is all about passing `-Dstyle.color=always` [09:59:35] https://gerrit.wikimedia.org/r/c/integration/config/+/1334776 [09:59:43] I did test it locally and going to build those images + switch jobs to it [10:29:48] done [10:29:55] maven now outputs color https://integration.wikimedia.org/ci/job/search-extra-analysis-maven-java21/4/console [13:06:55] o/ [13:38:55] \o [13:49:00] dcausse: regex bit is kinda two things, one is that the highlighter has a fix for .*foo that reverses and uses a two-phase forward/backward check to make it faster (verified on 2kb inputs). The initial problem was the the NFA changes mean the automaton is no longer minimized, our detection of the leading wildcard started breaking [13:49:17] fixed that by swapping it to a check on the AST instead of inspecting the automaton [13:49:58] o/ [13:50:02] but it leaves the question of if we should be trying to fix too_complex_to_determinize by swapping to the NFA-ish variant that doesn't need that. But my read is they don't have anything to prevent memory explosion, we would essentially need to check the size of the automaton from the CharRunAutomaton [13:50:14] ebernhardson: oh completely forgot about this optim... [13:50:47] you mean for the second pass recheck and the highighting? [13:51:19] i'm initially looking at highlighting, but i assume it would have to be matched with the query-time and highlight-time using the same [13:51:33] otherwise one works and the other fails determinization [13:53:00] probably simplest is to ignore the NFA's for the initial migration, but i wonder if it's something we should consider implementing to reduce errors [13:53:25] i'm trying to remember...i feel like i've seen totally normal regexes fail determinization (not just the crazy stuff in the test suite), but didn't turn anything up yet [13:53:38] if there's no system to prevent explosion that seems dangerous... also seems like it was mainly designed for regex term queries [13:53:40] also haven't dug through logs in hadoop yet either though [13:54:19] the regex term query will apply the regex once on the term dict IIRC, as opposed to the rechecker and the highlighter [13:54:35] that will run for every hit [13:54:47] it could definitely be tested [13:54:48] it's not clear to me why it isn't baked in, but the system to prevent explosion is to basically check how big the automaton is once in awhile and bail if it gets too big. Claude was able to come up with a test pattern that ended up generating a 1MB+ NFA [13:55:35] ahh, i didn't realize there was a term varient. hmm. [14:10:31] another random new thing in regex i wasn't sure if we should support, it now has https://github.com/apache/lucene/pull/14381 CASE_INSENSITIVE and CASE_INSENSITIVE RANGE. I couldn't think of what that really buys us over the lowercasing today, except the ability to less-awkwardly support \D and such [14:13:49] seems cleaner? today we lower-case the expression and thus can mess-up character classes no? [14:15:14] yea it could be cleaner with less edge cases. I guess messed-up char classes would be for unicode? I suppose i don't know how awkward ordering is in there, but i can imagine across languages it does something weird [14:15:32] my read is the case_insensitive_range expands the ranges use icu somehow [14:36:39] I assume no retro today because of the staff meeting. [14:36:42] o/ I would suggest we skip our retro everyone can join the Monthly Staff Meeting. [14:36:49] jinx! [14:37:20] That’s how I imagine quantum effects… [14:41:24] ebernhardson/dcausse, if you have some time left today: I iterated the query latency [metrics CR] (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/1330612) it no longer uses `SearchContext` and turns `getPoolCounterType` into a dimension. [14:46:00] pfischer: sure i'll check it out [15:01:44] ebernhardson: thanks! [15:08:29] Just a heads up from yesterdays Semantic Search Sync meeting: Selena (and Chris) see high potential in Semantic Search and asked for rolling out Semantic Search to web ASAP. They are fine with serving only a fraction of users and a selection of wikis. Since there’s extra budget for hardware, I’ll create a ticket where we can gather estimated hardware requirements together with SRE. [15:09:31] ACK, keep us posted [17:03:54] dinner [18:13:38] read up and i agree, the NFA optimization seems most useful when you have very small text like is typical in a term dictionary. against oversized source text [18:14:07] against oversized source text it's not much help