[00:51:37] i'm going to have to look more tomorrow...but my hax did do one thing for sure: It made index recoveries crazy slow :P [00:52:46] * ebernhardson separately wonders if the madvise is also slowing the recoveries in the primary cluster...probably but maybe less effect as disks are local [14:06:52] Not feeling great this morning. I'm going to rest a bit and I should be back for the 2nd half of my shift. [14:07:27] inflatador: sure, hope you feel better soon! [14:10:52] \o [14:11:26] pfischer: btw policy in the gerrit side of things is opposite to gitlab, in gitlab we usually self-merge after approval, but in gerrit the expectation is someone else merges your code (although there are exceptions, but for mw stuff particularly) [14:12:05] just responded to comments on the patches, and dropped that comment that was left over from the class i copied to start implementation [14:12:30] ebernhardson: Sure, I’ll go ahead and +2 then. [14:17:27] thanks, i'll get those into this afternoons deploy window [14:25:51] pfischer: something else i wonder if we should look into, this evaluation would have been a bit easier if i had root on dse-k8s. Might not need it again for awhile, but it might be convenient to have, and i suspect dpe-sre would be amicable [14:26:07] (this == readahead evaluations on k8s opensearch) [14:30:35] ebernhardson: sure, I’ll reach out for those permissions. [14:32:52] it also looks like i do have a viable way to turn off readaheads strictly from the helm chart (via LD_PRELOAD of a custom lib), but it's perhaps a bit too-brute force. Could iterate on it, but i suspect we will get a decent enough result from something less hacky. [14:34:09] but data transfer graphs and load testing confirm it at least works, 35rps w/10 users is seeing ~250mb/s, when we were seeing 4GB/s without it. [14:39:21] Wow, that’s a strong indicator indeed. [14:45:11] i think it also scaled slightly better with the 16kb readahead that was set yesterday, vs the no-readahead i've hacked into it. It basically just stays more consistent with the small readahead, p95 and p99 stay more in check. with no readahead at all it seems like there is some more variation to the stats coming out [14:49:09] ebernhardson: https://wikimedia.slack.com/archives/C055QGPTC69/p1772635623432339?thread_ts=1772635338.340509&cid=C055QGPTC69 - does that sound like it would help you already? What (else) permissions do you need? [14:50:53] pfischer: hmm, some suid script might be plausible, but narrow. Maybe the right question is to access the wider SRE group for SRE-level access to the cluster? I know that's not unprecedented, there are a variety of non-sre in the admin.yaml ops group [14:51:00] s/access/ask/ [14:51:33] not a ton of non-sre's in that list, but not unprecedented [14:52:19] Sure, let’s ask for that, but I assume that needs some time to be approved, as suggested by Ben? [14:52:38] yea it would have to go through the normal access policy, i'll have to look how we do that (it's been a few years :P) [14:54:12] https://wikitech.wikimedia.org/wiki/SRE/LDAP/Groups [14:56:33] ebernhardson: here’s a link to create an Access-Request task: https://wikitech.wikimedia.org/wiki/SRE/LDAP/Groups/Request_access [15:19:01] random guess at a purpose: Debugging/tuning search infrastructure as we migrate from fully bare metal (where search currently has root access) to having some search infrastructure in k8s. Most recent example would be evaluating/mitigating over-aggressive default readahead settings in dse-k8s. Other examples include working with the k8s opensearch-operator, and work in puppet to support [15:19:03] search infrastructure. While many things can be done pairing with DPE-SRE members, multi-day debugging efforts are difficult to pull off in a timely manner when it requires regularly interrupting others to apply changes to be evaluated. [15:28:08] hmm, patch failed jenkins due to typed class constants, added in php 8.3. And indeed the compat table says 8.2.0+....sigh, fixing [15:28:19] i was sure we had used those somewhere already, but maybe not for const's [18:50:57] o/ [19:20:40] just catching up on Slack/IRC scrollback. Sounds like we are moving fwd with a puppet-based solution for readahead? [19:27:39] inflatador: i believe so, yes [19:29:48] I don't love it, but it should do the job I suppose [19:30:20] I guess the "right" place would be a Ceph CSI plugin? I dunno [19:30:30] err...in the Ceph CSI plugin that is [19:31:22] inflatador: there might be some way of setting mount options? Although i haven't looked deeply, but the ceph docs mention an `rasize` option [19:33:31] ebernhardson good find, I'll add that to the Slack thread [19:33:59] inflatador: it's from https://docs.ceph.com/en/octopus/man/8/mount.ceph/ [19:35:42] Looks like that's for cephfs, I wonder if there are similar hooks for OSD (block-based). If not, I wonder if you can just set that as a generic mount option for ext4 or whatever [19:40:06] hmm, not seeing one for ext4 [19:40:31] according to random search results, it's a block device based setting ( https://groups.google.com/g/mongodb-user/c/N2E6oQ6Ect4 ) [19:40:43] might be different for CephFS since that's ceph under the hood anyway [19:41:30] at least google (even in an incognito window to avoid recent search history pollution) `mount rasize` finds pages from ceph, suggests it's specific [19:41:43] but there might be a related option with different name [19:52:54] lol, completely random, but the qwen3 embedding model when loaded into llama-server has an http interface, but no matter what you type into it it returns "exampleText example text example example example ...." [19:53:50] oh actually if you start a new convo and put in something else, it's basically repeating variations of the last word [21:27:58] inflatador: having the same problem as yesterday, masters-2 came up on dse-k8s-worker1028.eqiad.wmnet and is refusing network connections [21:29:55] ebernhardson damn, I wonder what happened? I'll take a look [21:45:15] ebernhardson I cordoned again, it looks like `masters-2` is on 1003 now. I don't see anything in IRC that explains why it 1028 is being scheduled. The cookbook said it was already cordoned, so IDK. I'm going to shut it off and suppress alerts until we have time to look [21:46:06] inflatador: thanks!