[08:39:44] <atsukoito>	 good morning!
[08:40:18] <dcausse>	 hi!
[08:55:40] <dcausse>	 re-indexed ptwiki with jina in eqiad, updating codfw now
[10:43:10] <dcausse>	 lunch
[13:33:09] <inflatador>	 <o/
[13:33:51] <dcausse>	 o/
[13:34:08] <inflatador>	 Before I forget, I have an old action item from DPE SRE retro we could discuss at standup if everyone is OK w/it: "Decide whether to further shards commons_wiki file (currently shards are 50-70GB, we usually target ~30GB as our shard weight)"
[13:36:37] <dcausse>	 inflatador: sure, but there's a DPE all-staff and not yet sure who's going to attend search standup vs this one
[13:45:13] <ebernhardson>	 \o
[13:46:20] <dcausse>	 o/
[13:47:35] <inflatador>	 dcausse ACK, good call. We can talk about it async instead if that's better. I think that came out of the cluster quorum issues, since those have been fixed maybe we don't care as much anymore
[13:48:51] <dcausse>	 inflatador: sure, we probably need to do some math first, commons might start to be tricky, reducing shard size by hald requires doubling the number of shards but IIRC we're already quite high in that dimension too :(
[13:49:04] <dcausse>	 s/hald/half
[14:00:35] <inflatador>	 Yeah, it's not urgent AFAIK. I'll make a task if one doesn't exist already
[14:04:04] <dcausse>	 thanks!
[14:39:03] <inflatador>	 Oh yeah, I guess I won't be making stand-up anyway due to the all-staff 😳
[14:51:34] <Trey314159>	 Yeah, I'm presenting our second-try work at the DPE meeting so I definitely won't be at the triage meeting
[14:57:37] <pfischer>	 Trey314159: Great stuff, I’ll be there for the presentation too.
[14:57:41] <inflatador>	 semantic search is still planned to run on the main Cirrus clusters, right? Just looking at T437877
[14:57:41] <stashbot>	 T437877: Temporarily repurpose relforge machine for semantic search load test - https://phabricator.wikimedia.org/T437877
[14:58:00] <pfischer>	 inflatador: not decided yet
[14:59:14] <pfischer>	 inflatador: https://wikimedia.slack.com/archives/C055QGPTC69/p1789396200950929 we want to run an experiment first, then run semantic search on k8s (as for the first phase) and only then evaluate where we want to land.
[15:00:17] <pfischer>	 ebernhardson: did you see that slack convo? Do you think Ben’s proposal would work too to get local SSD storage insights?
[15:01:24] <inflatador>	 pfischer ACK, I'll follow up on that thread
[15:02:17] <ebernhardson>	 pfischer: hmm, looking
[15:03:36] <ebernhardson>	 pfischer: seems plausible, i worry a little about "We can flip a coin to see if we give it back" from balthazar, but it's probably tongue in cheek
[15:03:49] <ebernhardson>	 it sounds like they may not need the host initially anyways
[15:16:01] <ebernhardson>	 could use code review on the 3.8 debian package: https://gitlab.wikimedia.org/repos/search-platform/opensearch-plugins-deb/-/merge_requests/23
[15:16:10] <dcausse>	 looking
[15:16:58] <dcausse>	 lgtm
[15:17:08] <ebernhardson>	 thanks!
[15:18:54] <inflatador>	 👍
[15:19:22] <ebernhardson>	 sigh...CI failed: : Release file for http://snapshot.debian.org/archive/debian/20260906T040200Z/dists/bookworm-updates/InRelease is expired (invalid since 1d 13h 6min 7s). Updates for this repository will not be applied.
[15:19:32] <dcausse>	 translatewiki.net is finally running opensearch 2.19.5 apparently (https://translatewiki.net/wiki/Special:Version)
[15:19:41] <inflatador>	 {◕ ◡ ◕}
[15:19:54] <dcausse>	 we can drop elasticsearch, and possibly drop os 1.3 support
[15:20:20] <dcausse>	 ebernhardson: yes they made new images today
[15:20:30] <ebernhardson>	 oh, needs a new base image? k
[15:20:34] <dcausse>	 yes
[15:21:22] <ebernhardson>	 hmm, we don't specify a version. i guess it's just cached.  Can specify the version i suppose
[15:21:42] <ebernhardson>	 the latest is docker-registry.wikimedia.org/wmf-debci-bullseye:0.0.3-20260830 though which is a few weeks ago
[15:21:58] <ebernhardson>	 oh, bullseye...we want newer
[15:27:01] <ebernhardson>	 oh that's annoying...ci doesn't offer the build option unless i change debian/changelog.  but the top patch changed the CI image :P
[15:38:36] <ebernhardson>	 sigh, it failed with the -trixie image as well :S
[15:39:09] <dcausse>	 :/
[15:39:20] <ebernhardson>	 err, hmm.  actually i guess i failed to override it? i should read the included repo...I updated our specs to trixie, but apparently it still used the -bookworm image specified in repos/sre/wmf-debci. hmm
[15:59:53] <ebernhardson>	 got referencd to https://phabricator.wikimedia.org/T437069 so i guess we are just out of luck for now.  I suppose we can build the .deb locally instead of letting CI do it
[16:01:39] <pfischer>	 Trey314159: Thank you for the demo! I am glad that this got visibility.
[16:11:10] <dcausse>	 frwiki should be served by jina now, started enwiki, might take a while
[16:13:38] <ebernhardson>	 nice!
[16:23:09] <dcausse>	 is there a dashboard that shows space usage of the semantic-search dse-k8s cluster from the ceph perspective? I barely remember that something like this exist but can't find it...
[16:23:37] <inflatador>	 let me check
[16:31:10] <dcausse>	 using sum(opensearch_fs_path_total_bytes{cluster="opensearch-semantic-search"}) - sum(opensearch_fs_path_available_bytes{cluster="opensearch-semantic-search"}) (this is from opensearch POV but should be good enough for me)
[16:33:40] <dcausse>	 thinking about it ceph pov might not help because I think it might see the total volume space and not know what's actually used in the volume
[16:34:59] <inflatador>	 yeah, it gets a little confusing...plus the ceph usage would be 3x due to replication
[16:44:38] <dcausse>	 unrelated but noted last friday that now our fulltext qps is almost similar to comp_suggest... https://w.wiki/UWsD (fulltext) vs https://w.wiki/UWsJ (auto-complete)
[16:45:05] <dcausse>	 big bumps in the last 3 or 4 months...
[16:46:21] <dcausse>	 pool-counter rejections obviously increased... not sure if we should worry/do something about it, but this is clearly not traffic from legit users
[16:57:34] <dcausse>	 dinner
[17:04:07] <ebernhardson>	 yea that sawtooth looks very unnatural...not sure what we should do about it.  The failure rate is at least quite low
[17:29:46] <inflatador>	 Yikes