[01:16:16] ori: https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red?orgId=1&viewPanel=57 [01:16:54] that's scaling_governor=performance rolling out [01:20:59] it's a little ridiculous [01:21:04] like it should not have been that easy [01:21:20] you'll have to find a way to dress it up for the blog post [07:00:58] That's... quite an improvement [09:25:35] you can see a power usage change if you look very closely at the exact right place, maybe 1kW in the appserver cluster, maybe another 1kW in the api_appserver cluster [09:25:55] but it's absolutely dwarfed by a 10kW jump in analytics power usage today [09:32:40] p50: 50ms to 36ms https://grafana.wikimedia.org/d/000000066/resourceloader?orgId=1&from=now-24h&to=now&viewPanel=45 [09:35:37] Sidebar p75 parse: 14ms to 9ms https://grafana.wikimedia.org/d/lqE4lcGWz/wanobjectcache-key-group?orgId=1&from=now-24h&to=now&viewPanel=52&var-kClass=sidebar [09:35:50] Many more via wan cache miss timing [15:54:12] Krinkle: I made this [15:54:15] https://usercontent.irccloud-cdn.com/file/GQxGk96s/image.png [15:54:36] since you made the webrequest ones, any edit on it would be really appreciated [15:54:46] https://www.irccloud.com/pastebin/ibjwdNog/wikimedia-relational-databases-2022.xml [15:55:35] specially the arrows are a mess [16:45:18] One thing I'm thinking would be useful to separate read and write (e.g. I don't want to imply memecache write to ES) [19:22:31] Krinkle: can you look at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/827858/ ? [19:29:08] ack, left 2 Qs [19:38:52] I think one important takeaway from the frequency scaling change is that the performance team should be involved in evaluating app server hardware [19:39:14] (apropos of nothing)