[11:22:38] zabe: thanks! [11:22:52] yw [16:51:37] apergos: the only other cause of finding new module source code, that I can think of, would be on-wiki mechanisms that effectively deploy code, ie editing MediaWiki namespace pages such as Common.js and gadgets. Those are loaded as the "site" and "ext.gadget.*" modules respectively. That'd generally invalidate just one module on one wiki though so not a spike [16:53:02] From the other end, I guess a mass restart or outage of memcached could cause stuff that hasn't changed to act as new. [16:55:37] but we've not had such incidents in the recent past I take it? [17:13:54] apergos: we have, but when we do, RL latency is not the the most important problem. [17:16:15] For a good year or so we have regular incidents during which the physicial links between appservers and memc servers became saturated, leading the mcrouter to think memc hosts are down and thus proactively fails-fast (return false) for all key access for a minute or so. [17:16:24] (mcrouter is the memcached proxy on each appserver for persistent connections and key hashing distribution) [17:16:49] some details at T244852 [17:17:56] the main two things we did to improve that was: upgrade the links from 1G to 10G, and add a "gutter pool", where the mcrouter proxy instead of stubbing out several hosts for a significant portion of keys for a minute, instead fallback to a dedicated alternate mc host that's standing by for that purpose. [17:19:46] closed to home, in bagostuff, what we (perf team, Aaron) did is implement support for "hash stops" in WANObjectCache so that meta keys and value keys for memcached are stored on the same physical memc server so that reduces the impact of 1 host going down [17:20:39] i.e. instead of routing enwiki:something:Main_Page:v and enwiki:something:Main_Page:t (notice hte last char) to different memc hosts, they are now guruanteed to hash to the same host by putting ":v" after the hash stop. [17:20:45] ref T252564 [17:26:28] the hash stop bit is really nice [17:26:47] (reading one of the patchsets from T252564)