[01:07:27] TimStarling: ack [01:07:40] TimStarling: I'm catching up with https://phabricator.wikimedia.org/T267270#6927518 to refresh my recollection [01:08:18] I'm also writing a quick script to see how much, if anything, we can generally expect to generally last for 1 min. e.g. setting 10K keys with ttl of 60s and checking them after 50s [01:09:34] My expectation is, that without empirical data, that we're not supposed to store such tokens in memc and relialby expect them to remain, esp with TKOs and gutter pool and general eviction under pressure. It seems a poor failure scenario. Afaik aaron was thinking of asking to stand up a separate cluster for these rather than use the main. [01:11:12] we also have the open question for centralauth non-token data, and depending on what we do with that, perhaps this can go along with that in the same object or or a similar key, not sure it needs a separate cluster all just for that. If we do a cross-dc write with memc from the get token API request, to then re-use in the POST req, then maybe we could just as well write to the primary redis_local and keeping using that for non-LRU data. [01:13:05] seems worth talking about anyway, and closing the loop with SRE in terms of making sure we're on the same page with regards to expectations and failure scenaris, tolerances, what degrades graceful and what doesn't. I imagine they'd prefer we not treat the main memc cluster in a way that when partially unavailable starts failing edit attempts. [01:13:56] I'm noticing that when I use getLocalClusterCache, I can easily get 10,000 key SET in under 5 seconds from a maint script. When I use WANObjectCache interface, it gets only to about 6000 and something. [01:14:45] https://phabricator.wikimedia.org/P29911 [01:22:59] you mean 6000 in 5 seconds or 6000 before failure? [01:23:24] oh I see TIME_SETTING [01:25:07] I assume your script is showing no check() failures? [01:25:21] Indeed. Ran it three times, seems to hold up. [01:26:12] you know my view on multi-DC in general: constrain scope, try to get things done [01:26:22] That doesn't mean SRE won't object or that it'll always be reliable, but it certainly contradicts what I expected based my general impression of Memc being under eviction by default/by design to not keep things unless there's hits to it. [01:26:57] I'm guessing that sense, if true, is applicable to a larger time scale only. E.g. I would expect if I set the ttl to 24h, that they won't last that long by default if there's no hits in between [01:27:10] but Memc also stores thigns separately internally and tries to avoid evicting new data soon [01:27:39] I studied the memcache source when we first deployed it, around 2005, so my memory might be a bit rusty [01:27:53] and it has presumably changed somewhat [01:28:30] but I understand it has "slabs", i.e. size classes, and an LRU list in each slab [01:28:49] actually IIRC a size class may have multiple slabs [01:29:12] so when it needs to evict, it chooses a slab somehow and evicts entries from the head of its LRU list [01:30:04] https://github.com/memcached/memcached/blob/master/doc/new_lru.txt#L10-L24 / https://github.com/memcached/memcached/wiki/UserInternals#when-are-items-evicted [01:32:34] TimStarling: okay, so if I understand correctly, the current patches and your current thinking is: we use the main memc cluster, and use the mcrouter prefix such that it always reads/writes these in the primary dc; and then not need to route any get-token or token-using requests differently from normal, is that correct? [01:33:24] noting that the proposed routes at https://phabricator.wikimedia.org/T267270#6907588 did not make it into cdn routing task plan (yet). [01:34:05] yes [01:34:52] the routes already exist in mcrouter's config, you can already ask for /eqiad/mw/$key when you're in codfw [01:36:30] ack. one thing to double check is how that interacts with the gutter pool. I don't recall if we inject that at the level above or the level below. ideally we'd make it naturally follow the same gutter pool as for traffic within the same DC, so that we get automatic failover. [01:38:13] I gues in terms of comparing it to using redis, the downside is that 1) afaik that's not on tls so not great for cross-dc reads or writes and would mean we need to try to route those requests and 2) lacks failover and so not great for stuff that hard-fails. [01:38:21] the gutter is always local to the DC in the current config [01:39:33] also in the current config cross-DC traffic is "ssl" [01:42:45] regarding the CDN routing task, I will have to make that update. Note that I copied everything in the CDN routing task into actual Lua code ready for deployment [01:53:28] TimStarling: hm.. you mean as optimisation try to route both obtaining and anything using token to the primary dc? [01:53:53] that was my suggestion in 2021 [01:54:09] with mcrouter it is not totally necessary, but it may have better latency [01:54:29] just for foreign API tokens [01:54:52] but for CentralAuth login tokens this is already implemented in my lua patch [01:54:56] I feel like we should confirm with serviceops that using this route cross-dc is fine and w.r.t gutter pool, but something we could do without dependency is just point it to 'mcrouter' as-is, since we're not multi-dc yet, that'll at least give us most of the experience and data we want to feel confident in the direction for a little while, and then add mcrouter-primarydc and switch to that once svcops clear it. [01:55:06] but overall I'm on-board with this. [02:01:50] I recall when we talked with Effie about the mcrouter set up last year, there was quite a lot of complexity around how the mcrouter configs are built, and still is: https://github.com/wikimedia/puppet/blob/26e2343da3b89f55e999fdb70958f056fe52a8dd/modules/profile/manifests/mediawiki/mcrouter_wancache.pp#L81 - from a quick look I don't see any obvious difference between the implicit default route for the local dc and explicitly specifying [02:01:50] /eqiad/mw/, but afaik there was some non-obvious reason for avoiding to use /eqiad/mw directly from MW. [04:01:09] > I feel like we should confirm with serviceops that using this route cross-dc is fine and w.r.t gutter pool [04:04:41] serviceops are going to ask me about the latency, and the best way to measure the latency is to deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/683022 and then test it with mctest.php [04:08:41] so I'll do that, then that can be used to decide whether we want to deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/683465