[14:37:30] \o [14:54:59] .o/ [15:35:33] meh, glent ran through but failed the part where it generates suggestions via all-pairs-levenshtein [15:35:51] at least it's failing somewhere else now [15:47:22] hmm, we have a comment about "24g ooms as of may 2020", i guess just bump the memory another 8g. [15:47:48] maybe someday rethink if all-pairs-levenshtein is the right approach, maybe something better exists today [15:48:37] i guess to be fair, the comments for the same thing call 800 cores "half the available cores", but yarn.wikimedia.org says there are 8.5k cores [16:34:56] random idea, we want levenshtein with k=2, in theory that means if you split all queries into three strings and then group on those strings, you should now have much smaller buckets to perform all-pairs on. maybe :P [16:59:15] i dunno, it generates the FST in ~10-20m, and then needs 30m to generate and run the automatons, for 180M queries it's probably fine [16:59:48] (naively it would be n^2, or 3.2e16) [17:50:33] * ebernhardson waffles between sticking some caching code in DeepcatFeature, or making a thin CachedSparqlClient that wraps the existing SparqlClient. [18:12:55] * ebernhardson kinda wishes we had a stand-in service that could fake sparql query responses [18:13:07] even something simple like hash the query, and return canned response [19:00:32] ebernhardson: there are a few mock services out there: jest, wiremock - would any of them suffice? [19:04:29] pfischer: for now i stood up a php service that always returns the same response no matter what it's asked, but that isn't part of the end-to-end integration testing at all. Would have to ponder if it's worthwhile to add something to the docker env we use [19:04:56] the php thing basically lets me do manual testing, maybe ok-ish [19:06:29] we have mocks in the phpunit side of things which is good enough for that [22:05:18] ryankemper I'm in pairing + CR for new opensearch image https://gitlab.wikimedia.org/repos/data-engineering/opensearch/-/merge_requests/12 [22:05:42] brt [22:08:16] inflatador: slo patch for ya as well https://gerrit.wikimedia.org/r/c/operations/puppet/+/1230399 [23:13:24] we have a working OpenSearch 3 docker image, `opensearch:2026-01-22-222021-cee6a9491ea54122ff6df5fd31ca2345f25f6e26-production3` [23:56:16] inflatador: patch to provision `opensearch-semantic-search` ns: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1230512 [23:57:04] inflatador: i copied the `opensearch-ipoid-test` files and just did `s/opensearch-ipoid-test/opensearch-semantic-search/g` so there may be errors if there's something specific to that implementation [23:57:40] oh, I think I still need admin-ng changes