[10:21:01] lunch [13:18:23] o/ [13:45:10] \o [13:46:41] o/ [14:27:22] o/ [14:31:34] .o/ [14:37:57] ebernhardson: Seddon was asking about the mdeberta model decision (https://wikimedia.slack.com/archives/C0975D4NLQY/p1790081154851239). Do you have a ticket in progress for the write-up, or would you like me to throw together a quick stub so we can start centralizing the discussion? [14:52:58] heads-up that I'm gonna do an IP change on cirrussearch1120 per T436571. I've verified that it's not a master and set a downtime, so no user impact is expected [14:52:59] T436571: Re-IP Search Platform-owned EQIAD hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T436571 [14:59:49] pfischer: This is the main writeup, but only the first 1/3-1/2 is edited by me. The second half is claude explorations: https://phabricator.wikimedia.org/P96020 [15:00:16] the main thing i need to do is go through and clear out all the extra uninteresting data and think about what parts actually make the argument [15:01:04] i suppose i also get distracted because when i look into it the distinction between mdeberta and xlmr is not so clear, i'm leaning towards mdeberta because it seems to do better on questions, but xlmr is potentially equal or better at keywords [15:01:12] or keyword-style search i should say [15:18:38] We're getting a morelike P95 alert for CODFW, checking now. [15:20:41] :S [15:21:15] https://grafana.wikimedia.org/goto/s9spsh?orgId=default [15:21:36] hmm, all load in on a single cluster because of the switchover [15:22:46] 325 req/s is a lot of morelike, load is pretty high across the board. might simply be that throughput has increased too much since the last switchover? [15:23:26] Maybe so, possibly related to what pfischer mentioned about increased traffic on the cluster [15:23:26] morelike cache also cratered on the swtichover, hopefully it comes back. went from 45% to 25% [15:23:49] iirc it's a 3 day cache, so might take some time to fill [15:24:19] actually it's 1 day [15:25:09] Apologies for completely missing the DC switchover ;( [15:26:06] Maybe we should add some cache warming steps to https://wikitech.wikimedia.org/wiki/Switch_Datacenter#ElasticSearch ? [15:27:00] inflatador: i'm not entirely sure how, it's millions of random pages. Something we've done before is re-direct all morelike traffic back at the idle cluster until the cache fills [15:27:20] we have a flag in the cirrus config for routing classes of queries to specific clusters, skipping dnsdisc [15:27:46] ebernhardson ACK, do you think it would be appropriate to do that here? [15:28:10] inflatador: hmm, probably. i can prep that [15:30:40] similar patch which eems to hvave the same reason from last septembers switchover: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1190737 [15:30:57] they switched mw as well, probably why the cache got messed [15:31:24] perhaps we can wait for them to repool mw@eqiad and see? [15:32:31] i don't know the plan this week, how long will eqiad be depooled? [15:34:19] from memory I think that's a two steps procedure, first services switch to codfw only (mw stays active/active), then mw, wait 1 week running everything on a single DC [15:34:51] hmm, in theory over the next 24h the more_like hit rate should improve and we should get back to healthy. [15:35:00] they're repooling eqiad for mw-web [15:35:22] just found https://phabricator.wikimedia.org/T433363 [15:35:50] can't remember if the RelatedArticle extension is hitting web or api [15:36:45] should be visible soon [15:37:01] i also wonder if perhaps our more_like poolcounter is too big? we are getting rejections in other areas, but not morelike [15:37:23] yes... very possible [15:38:51] looking in #operations it sounds like they are rolling back the switchover [15:39:13] we currently allow 150 concurrents. Commented as "increase from typical sizing to handle the expected empty more_like cache on switchover" [15:39:13] must be hitting api, fulltext is going up in eqiad but not morelike [15:39:33] so, we upsized it to allow this to happen. But the cluster isn't happy :P [15:39:42] yes... :/ [15:41:26] mw-api-ext-ro is going to be switched back, that should solve the issue for now [15:41:26] poking at cluster overview graphs, i would guess our limiting factor is CPU. [15:42:17] i suppose a different idea we never looked at...could make more_like cheaper when needed? Maybe less words or some such. Not sure if appropriate [15:42:38] very possible... with the increased qps of the fast few month on fulltext + cache miss on morelike there's probably not enough threads to handle all that in time [15:42:48] re: CPU, that's interesting since I recently turned off performance governors for cirrus hosts. Maybe we look at flipping those back on [15:43:10] inflatador: mostly it's that IO looks minimal, network doesn't look crazy, and CPU is at 80% in much of the cluster [15:43:29] yes more_like will run significantly faster on opening_text IIRC but I have no clue how that fallback could pollute the cache somehow [15:44:40] hmm, the cirrus side cache will replace itself in 24h, should be fine, checking what related articles uses [15:44:41] ebernhardson that's supported by the PSI metrics, apologies in advance for the half-baked dashboard: https://grafana.wikimedia.org/goto/sgng27?orgId=default [15:45:24] they send smaxage=86400, so also one day [15:46:21] inflatador: nice! [15:46:24] but how do we prepare for that? seems difficult from a mw-worker POV to decide if it has to degrade morelike [15:46:44] i agree...it doesn't really have any insight there [15:56:23] sigh... this morelike cache is very slow to populate [15:56:35] lol, yea. It has moved, but not much :P [15:58:05] other random crazy ideas..some sort of cross-dc cache population. If i had to guess i would say the wikis visited probably differ between eqiad and codfw based on geography, so codfw simply doesn't have a well populated cache for the kinds of traffic (europe, africa?) eqiad sees [15:58:15] but i don't really know how that would work [15:58:25] still waiting for a decision on the switch-over but yes if it takes too long before they repool mw-api-ext we might have to split morelike out of the discovery endpoint [15:58:32] but i also feel bad thats it's all an in-memory cache [15:58:57] i have a patch prepped: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1344014 [15:59:03] thanks [16:07:46] lol, refreshed and suddenly the line jumped...it's because the old hit rate fell off and the scale is 24-28% :P [16:07:58] it's improving at least [16:22:21] still unsure what's going to happen, sre seemed to agree to repool api@eqiad but not seeing any effect and suspecting this had happened [16:22:33] *not happened [16:23:19] yea same [16:23:34] fulltext did partially come back to eqiad [16:23:41] ~200qps [16:24:00] they repooled mw-web@eqiad, so probably only Special:Search [16:29:30] perhaps we need yet another endpoint for morelike & possibly completion to rapidly switch things to manually split search traffic in these circumtances [16:31:09] hmm, yea i suppose if we had separate dns for specific high-impact use cases it could be useful. Routing traffic with mw configs isn't great [16:39:15] I guess we'll wait for the cache to slowly populate, not totally sure that mw-api-ext-ro@eqiad being still depooled is expected... [16:42:31] yea at this point seems reasonable [16:42:49] p95 looks like it should get back under 1s soon, maybe [16:47:33] wondering if cirrus logs have enough data to verify this cache hypothesis that we serve quite a different set of pages from both DC [16:54:00] hmm, i suspect yes? I think it could be an outer join of (dc, wiki, page_id) and then comparing the number of pages that come from 1 or 2 dcs? [16:54:29] maybe the search query could replace page_id, not sure how much we would have to dig into the request logging [16:55:14] looks like they got mw-api-* repooled now, eqiad traffic is coming up, hit rates as well [16:55:31] search logs have have all_cached:true on the happy path could possibly be useful [16:55:51] oh right, that can [17:01:20] looking at load, dnsdisc must be all pointed at codfw [17:01:49] so it's currently mw app servers in both dc's all talking to codfw, but that lets the eqiad caches serve morelike requests [17:03:36] yes, internal services switched I guess [17:03:42] kind of hard to see in the graphs [17:04:05] the cache is back but a single search cluster is still serving [17:04:57] how could this be worse with the cache back? [17:05:25] i'm was wondering the same :S [17:06:08] CPU is completely burning down in CODFW, I'm inclined to turn on the governor and beg forgiveness later [17:06:44] i suppose one potential problem is we have separate eqiad and codfw pool counters, so if that was doing any queueing it would stop when we split the app servers between dc's [17:06:55] not sure if that's significant enough though [17:07:22] oh indeed the pool-counter is less aggrissive now (waits less) [17:07:58] so even with the cache back and unless I'm missing something obvious it's clear that a single DC cannot sustain all traffic [17:08:22] seems potentially yes [17:09:08] 1k/s fulltext is probably not cheap [17:09:46] yes they take a slot in the thread pools... [17:13:21] https://w.wiki/V3t9 [17:13:48] CPU still looking pegged out in CODFW, talking it over with DC Ops now [17:14:46] i mean it's looking better than an hour ago, but yea it's not happy [17:28:12] I've got tentative DC Ops approval to turn on some rows, but apparently power is already peaking from the switchover [17:37:25] was curious, apparently our codfw cluster increased from 12 to 18kW on switchover from the load [18:01:31] CR up for enabling performance governor on row D if anyone has time to review: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344041 [18:06:33] restarting opensearch-semantic-search in dse-k8s-eqiad for 3.8.0 upgrade [18:43:25] Thanks for the +1! I just enabled the governor for the row D hosts, let's see if it changes anything https://grafana.wikimedia.org/goto/ss9vpt?orgId=default [19:03:11] not clear if that had any effect :( [19:04:36] yeah, I was hoping to see an immediate change like we did with wdqs back in the day [19:07:18] currently prepping patch to add opensearch-semantic-search-ssd to admin_ng and dse-k8s-services, i imagine those should be two patches for separate deployment? [19:08:05] yeah, a chained patch set would be nice. I can help review when they're ready [19:31:23] this is the admin_ng patch, new cluster is also up as second patch: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1344062 [19:34:08] ebernhardson do we still need the more_like patch? https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1344014 [19:37:35] inflatador: hmm, maybe although the reasoning would be different. more_like is back to reasonable cache hit rates (after eqiad mw app servers were repooled), but latencies are still too much [19:38:47] but the general idea of move some expensive traffic to another cluster makes sense and should fix the latency issue [19:39:02] ebernhardson ACK, I added +1 to both patches, if you need me to apply the admin_ng patch LMK [19:44:02] i suppose the alternate to the mw side patch is to repool eqiad to dnsdisc. Not sure which is better. -security suggests dc switchover is plausibly going to be canceled anyways [19:44:18] inflatador: sure, if you could apply admin_ng when you have a cahcne [19:50:31] ebernhardson ah, since there's a new namespace we need a Puppet patch too. I'll work on that, are you OK with rolling out the WM patch? It's been a long time since I've done one of those [19:53:59] yea i can [20:00:11] cool, rolling out the puppet change now [20:10:02] more_like traffic shift is also deploying now [20:14:52] i randomly wonder if we should attach some header to search responses so cirrussearch can record metrics with a metric for the cluster that responded. right now it's only the mw app server cluster in the metrics [20:17:27] we used to know, but now it's mostly mw@{dc} -> dnsdisc [20:18:59] yeah, that's an interesting idea [20:20:12] https://chatgpt.com/s/cx_6ab2e24bdbb08191b4284bc9507a9d22 here's the clanker's idea on why the governor didn't change. "It was already at max CPU" seems plausible [20:20:22] governor didn't change CPU performance, that is [20:20:45] deployed. cpu pressure stall numbers are way down (but not 0 yet). latencies improving across the board [20:23:34] curious thing is delta between eqiad and codfw latencies, both talking to codfw, it about 60ms on fulltext, and 30ms on comp suggest. I forget what, but that implies fulltext gets an extra round trip that comp suggest doesn't [20:23:40] maybe the near match (go) lookup? [20:27:30] seems plausible [20:29:13] latencies are back to last week numbers, approximately. So without the more_like load it's capable. [20:29:58] I do suspect we have opportunity for improved caching in more_like, but would probably require some research to determine what would be effective [20:31:41] what i would almost like is a cache that doesn't TTL values, more like a database of recommendations. Then we could directly tune the recommendation refresh frequency, although it would still need backpressure from search somehow [20:34:47] randomly guessing at graphs, the full morelike traffic costs ~10-15% of one clusters cpus (and codfw was at capacity with 80%, so its more like 15/80=20%) [20:35:35] I know we talked about maybe moving the compsuggest indices to their own clusters, would that have bought us any CPU? [20:36:49] hmm, probably not a ton. it is memory resident and cpu-bound, but i think it's just not expensive enough. comp suggest is peaked at ~1.1k req/s, but the requests take like 10ms on the backend so thats only maybe 10-20 cores [20:37:30] i suspect what we would buy with moving compsuggest into it's own cluster is more consistency to response profiles, no noisy neighbors, etc. [20:37:35] well, different noisy neighbors [20:37:50] ACK. I assume almost all of the action was happening on the main clusters anyway, so moving psi/omega to different hosts wouldn't have done much either? [20:38:18] curiously...i don't know that breakdown. I have a vague idea that traffic is ~50% enwiki and 50% everything else, but i forget where that comes from [20:38:37] could be pulled out of our logs in the analytics cluster [20:39:32] i think i wrote in a presentation a long time ago that 90% of traffic is the top-30 wikis [20:39:56] Even without moving them, we could use cgroups to guarantee a certain % of CPU cycles for the main cluster. Just not sure if that would move the needle enough to matter [20:39:57] but thats old enough info it's probably at least somewhat outdated with recent traffic shifts