[11:58:44] Krinkle: cdanis and I were looking at T439239 before the power issues on Wednesday. seemingly not recent but probably worth fixing regardless, I've sent an untested patch [11:58:44] T439239: CentralNotice should batch campaign mixin lookup - https://phabricator.wikimedia.org/T439239 [12:17:51] taavi: nice. That's all within load.php right? That's on mw-web so will help with the problems there. It's outside pageview latency and should be heavily cached in cdn and localstorage but nice to fix for sure. I'll have a look [12:23:29] Krinkle: AIUI the code paths outside load.php are all related to campaign management and not to displaying banners on page views, but not 100% sure [12:24:45] taavi: yeah, I worked on moving it all out of ourpurpage and api and bundled/cached as part of the JS payload to reduce round trips and delays [12:25:31] That should make this path rare. But as I like to say: design for the cache miss, so no excuse :) [12:26:33] https://grafana.wikimedia.org/d/000000067/resourceloader-module-builds?from=now-7d&to=now&timezone=utc&var-module=ext_centralNotice_choiceData [12:26:39] This one, I think? [12:28:05] yep [12:28:45] That's faster than I'd have thought. Under 5ms, if we are to believe the quantile estimate [12:29:02] Which is at a bucket boundary so yeah fair [12:30:38] Might have some more caching internally. Maybe there's another miss case within this that's slower. Around 100 req/s across wikis * skins. And p99 within 5ms too. Anyway lots of queries is a cost barrier even if they're fast. [12:31:10] Your trace was 600ms [12:31:25] Although that's with a dozen other modules batched. [12:31:43] yeah, ChoiceDataProvider seems to do its own caching [12:32:00] Although although, just one of those queries in the trace says it took 18ms [12:32:18] So yeah I guess they're outside what we plot entirely. [12:34:12] we found the traces when looking at a replica that was struggling. it's possible that the query is a faster normally, although with network RTTs etc it's not going to be much faster [12:34:30] a bit faster* [22:34:31] Krinkle: is there anyone else to review https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1343308 ? [22:39:56] AaronSchulz: possibly Martin, he reviewed https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1326894 for me