[14:05:42] \o [14:18:33] .o/ [14:50:15] hmm, i was sure we had some ticket about load testing / evaluating perf...but not finding it [14:55:12] for Semantic Search? [14:55:16] ya [15:03:25] I'm not finding anything either. d-causse repo for load testing is at https://gitlab.wikimedia.org/dcausse/semanticsearch_demo [15:04:22] pondering what we need to test still, i guess the two main ones would be full index queries (i've been running single-shard queries) with the reduced readahead, and then the same test while running the indexing notebook to see if it changes much when doing reads and writes [15:05:10] It sounds like we have a way to set readahead, but it'll require running a privileged initContainer [15:05:33] ya i saw, i don't know though, how quickly does k8s-sig make decisions? [15:05:44] is it like a monthly thing? [15:07:12] ahha, this is the one i was thinking of: https://phabricator.wikimedia.org/T414623 [15:07:18] will try and update it with the findings from friday [15:09:50] pfischer: one minor problem, i know we were aiming to get api access on prod going this week, but the patches still need to be merged to get on the deployment train: https://gerrit.wikimedia.org/r/1229645 [15:37:25] ebernhardson: I’ll do my best to review the patch. [15:37:52] Anything specific we need from k8s sig? I can ping jayme. [15:38:40] pfischer: we need to set readaheads as part of an init container, but init containers are not currently allowed root access which is needed [15:39:32] Okay, I’ll reach out. [15:39:56] Is there a ticket I can refer to? [15:40:16] it defaults to 8mb readahead, which results in only getting decent performance if we have enough memory to fully cache the indexes (1:1 memory/disk ratio) and avoid ceph at runtime. With readahead reduced to ~64kb we can get decent performance with a 2:1 ratio of index:mem, haven't tested lower yet [15:40:51] pfischer: there is https://phabricator.wikimedia.org/T414623 but it doesn't have content yet, i'm writing up a response in there currently detailing some of fridays findings. [15:41:29] https://wikimedia.slack.com/archives/C055QGPTC69/p1772224159557889 has some slack discussion in dpe-sre [15:42:04] I'm not sure how long it would take to make a decision, I'm guessing less than a month. [16:33:16] ebernhardson, inflatador according to jayme SIG will most likely deny this request. As an alternative he suggested puppet system settings, to roll out such config to all workers [16:52:15] :( ok we are pondering some other options, will discuss a bit at dpe-sre planning meeting tomorrow [17:39:26] one-liner to get the OpenSearch vols from the K8s worker: `for n in $(crictl ps | grep opensearch | awk '{print $1}' | sort | uniq); do crictl inspect ${n} | jq -r '.info.config.mounts[] | select(.container_path == "/usr/share/opensearch/data")'; done` [17:59:11] Created T418776 for the OpenSearch readahead discussion [17:59:12] T418776: Consider options for automating the setting of read_ahead_kb for OpenSearch pod volumes - https://phabricator.wikimedia.org/T418776 [19:16:15] fun bug from commons village pump: https://commons.wikimedia.org/wiki/Special:MediaSearch?type=image&search=Kazakhstan+maps+logos [19:18:17] returns a bunch of burger king stuff because it gets synonyms from a wikidata search, and https://www.wikidata.org/w/index.php?search=maps+logos&language=en&title=Special%3ASearch&ns0=1 returns starbucks/mcdonalds/burger king. Not sure why it selects it out of all the options [19:23:34] * inflatador wonders if there are Burger Kings in Kazakhstan [20:29:46] * dr0ptp4kt ebernhardson i forwarded an airflow to the internal discovery list for your look (or any other search airflow-er) [20:30:30] airflow email is what i forwarded, that is. words! [20:36:44] :) thanks! [21:20:40] kicked it off again with higher limits, memory usage was higher than typical [21:28:07] wow, the opensearch deb package is 1200 MB all by itself [21:32:39] thats well more than i would have ever guessed. I mean i know they bundle a jvm, but still [21:56:24] * inflatador wonders why the MacPorts version of wget has a dependency on X11 pkgs [22:01:47] inflatador: 6’ late [22:06:30] ryankemper np, I'm around