[15:08:11] what's the latest on the switchover attempt? [15:09:20] cdanis: I depooled eqiad this morning, p50 latency went up to 180/200ms [15:09:29] at 20k rps right? [15:09:31] not great not terrible [15:09:37] yeah but 80% sat [15:09:42] which sucks [15:10:05] I have a couple tuning patches for memcache extstore which may help that I'll test tomorrow [15:10:40] We *may* have to do the read traffic migration slower so that memcache warms up a little more gently [15:11:00] I'll try to gather graphs and thoughts etc. either later tonight or tomorrow morning [15:18:55] > We *may* have to do the read traffic migration slower so that memcache warms up a little more gently - cdanis mentioned an idea for how to do this [15:19:27] sorry the formatting went quite wrong [15:22:59] we can not change the TTL, or even lengthen it, and wait longer between services [15:23:37] claime: there are more devious options [15:23:42] speaking of, I should probably restore the TTL i reduced to 10s this morning to its prod value of 300 [15:23:46] ATS supports SRV records including priority and weight [15:24:26] cdanis: you think we can have that in and tested before the 23rd though? [15:24:41] enough to use it once in a switchover? yes [15:53:35] cdanis: my, naive and rusty, take is that, we have to give it some time to normalise [15:54:35] with the amount of traffic and the traffic pattern itself, we need to give memcached some time to get itself sorted to [15:55:33] if there are ways to do this more gradually, it is even better. We cant expect the same performance as eqiad, but lets not forget that, our memcached capacity in eqiad was doubled in may, so that is not long ago [15:59:21] cdanis were you able to capture it in a task? [16:00:34] working on that today :) [16:03:59] cdanis: looking at the DB during the time you did teh switch [16:04:27] codfw connecttions went from 5k -> 20k [16:04:47] the time on sept 23rd ? [16:06:05] no, today [16:06:51] https://grafana-rw.wikimedia.org/goto/shvznb?orgId=default [16:07:12] https://usercontent.irccloud-cdn.com/file/bHQc1vmw/image.png [16:09:22] In may, mw-web was getting around 10k rps total [16:09:57] It's around 20k now [16:12:05] sure sure, we should work out an acceptable latency number, or accept that we will be ok with whatever we land on [16:16:45] it's not any one db section :/ [16:18:30] what's the deal with all the s3 writes during? [16:18:44] https://grafana.wikimedia.org/d/000000278/mysql-aggregated?from=2026-10-14T08:10:08.515Z&to=2026-10-14T13:12:30.723Z&timezone=utc&var-site=codfw&var-group=core&var-shard=$__all&var-role=$__all&viewPanel=panel-19 [16:19:23] (and before) [16:25:14] network saturation looks fine on codfw memcached for the whole duration, but wikikube workers have a bit of an rx saturation spike https://grafana-rw.wikimedia.org/goto/smhzq5?orgId=default not anything that should cause many problems though [16:56:04] traffic had increased anwee bit before the switch this morning, so it could be a scraper [17:24:45] yeah i delt with one and removed about 4krps from mw-web [17:25:10] but i'm struggling to explain how we doubled uncached traffic between may and now tbh [17:30:07] we may need to go through changes [19:01:23] I'm going to deploy a change to changeprop-jobqueue, just moving a job out of low_traffic_jobs. I have been assured that this is just a standard helmfile apply, but I haven't deployed this specific service before, so if anyone notices an alert about job queues start screaming, it's probably because I missed something atypical. [19:23:32] I had momentary worries that this was going to get complicated to monitor because my first diff showed that several prior patches hadn't gone out to staging, but fortunately they *had* all gone out to production, so the only production-change should be my just-merged patch. Which is deployed, and as far as I can see all the pods are back up and running. [22:44:12] kemayo: apologies was in a debug rabbit hole and just seeing these)) is [0] expected with regards to the change in [1]? - not super familiar with editSuggestionCountsPrecomputes nature but bringing to attention just incase it's unexpected [22:44:12] [0] - https://grafana.wikimedia.org/goto/s27697?orgId=default [22:44:12] [1] - https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1354914 [22:58:02] https://usercontent.irccloud-cdn.com/file/pnXEYVKA/editSuggestionCountsPrecompute%20-%20JobQueue%20-%20Service%20-%20Dashboards%20-%20Grafana.png [23:51:56] jasmine_: yeah, all looks fine. The spike is actually unrelated to the change. Or rather, the change was done because I knew the spike was going to happen when we turned up the load of that job.