[12:19:18] Krinkle: you can easily see the hit rate of PC has bumped since two days ago https://grafana.wikimedia.org/d/000000106/parser-cache?orgId=1&from=now-7d&to=now&viewPanel=7 [12:19:53] while by the deployment we practically invalidated all mobile-only caches [12:23:15] Amir1: nice. i'm slightly surprised at first that the absolute counts (the row below this panel) has gone down. it didn't translate misses into hits very much [12:24:03] maybe that's memcached absorb it through mostly existing cache keys that now work for both, but I thought it counted those. [12:24:16] (given it's abstract to ParserCache that it has a memc layer) [12:37:01] * Krinkle adds annotation to Grafana [12:42:13] we'll have to see where it settles but... optimically there might be a ~40ms decrease on p75 mobile backend pageview timing . https://grafana.wikimedia.org/d/QLtC93rMz/backend-pageview-timing?orgId=1&viewPanel=63 (click on p75 only) [12:42:29] from ~250 to ~210ms [12:43:20] https://usercontent.irccloud-cdn.com/file/7RSCc1WS/Screenshot%202023-03-01%20at%2012.43.01.png [13:18:08] Amir1: btw, noticing a spike of mediawiki-errors from mwmaint under jsonTruncated, something about "role-primary: INSERT INTO `querycache` …" may be of interest. [13:18:59] thanks do you have a share link handy? [13:19:33] https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2023.03.01?id=E71NnYYBxYrKUyhzsJmc [13:19:43] https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2023.03.01?id=fUpNnYYBtuN2AbPYDNh3 [13:19:47] two samples there [13:19:56] there's about 1000 of them in the past hour [13:20:35] Expectation (maxAffected <= 1000) by MediaWiki\\Maintenance\\MaintenanceRunner::run not met (actual: 5000) [13:20:41] * Amir1 dies inside [13:21:12] that can choke replication [13:24:50] Krinkle: Are you happy with the general direction of https://phabricator.wikimedia.org/T330590? [13:25:53] having get(P/R)Connection('urlshortener') "just" work sounds good to me. [13:26:10] It's less clear to me how we should handle the case of per-wiki domains on x1 [13:27:21] It's also ambigious in the case of echo [13:27:26] which does both [13:27:28] there will be a new concept: virtual domain, e.g. for some echo tables we call them "echo_external" [13:27:42] and then in config, it sets the cluster [13:28:33] even if we can make it work, it seems too significant to hide entirely. "echo" is not a single database, and having it translate to differnet ones depending on wiki context feels a bit too magical for me. It doesnt seem unreasonable to want to connect another wiki either. [13:28:44] It's quite likely externalstore does this already given that revision works cross-wiki nowadays [13:29:17] yeah, for that they should probably stick to getConnection tbh [13:29:42] OK, in that case maybe we can do that for both current and cross-wiki if you need a per-wiki external db. [13:29:49] that's basically only echo and externalstore [13:29:54] most x1 use cases are single DBs [13:31:13] for local we do have growth experiments and the thing is that I want to encourage sharding out of core dbs for extensions [13:31:57] as we need space in core dbs [13:33:03] I can think of a few ways to make that more ergonomic, one way would be for $domain to take a value like $domain = VirtualCluster::from($cluster, $dbname = false) and then handle it that way, a signature like getExternal(P/R)Conneection($cluster, $domain = false); [13:33:25] or a signature like* [13:34:16] The part I'm worried about is where we use the current clean signature in a way where $domain points to something that doesn't exist anywhere as a database and defaults in a way you can't override in the same signature to the current wiki. I think that will lead to mistakes as it's not obvious that when you pass something other than false that you still get trapped into a default. [13:34:39] plus it's a kind of thing where we do two things that are observably different but look the same [13:34:56] core/external db I think is neatly abstractable and logically the same. [13:35:08] I like the first option (value object) [13:35:21] but these per-wiki one I think are useful to expose as different concepts to devs, help with awareness. its not something they should forget [13:35:34] +1 on encouraging this more for extensions over time [13:36:06] I want to also make it better testable [13:36:22] specially making sure they don't accidentally join with core tables [13:36:39] and this is also something to consider https://phabricator.wikimedia.org/T314908 [13:36:49] like wiring in updater [15:34:00] Krinkle: I clicked on Special:Random in enwiki in debug mode and I think CP is activated for that. I checked and I don't have CP cookie (from what I'm seeing) [15:34:02] https://usercontent.irccloud-cdn.com/file/EgAfWN1H/grafik.png [15:35:04] it is explicitly sets to false but CP decides to grab master position anyway? [15:36:16] https://logstash.wikimedia.org/goto/420740fcd9bc910913bed881bb00f0e6 [16:50:37] Amir1 this is https://phabricator.wikimedia.org/T314434 [16:51:57] The current overhad should be "just" a not found fetch to memc and a no-op rdbms call, not an actual wait [16:52:16] thanks [16:53:15] I want to avoid opening db connection super early, that's my biggest reason but also maybe do some clean ups, half of lb/lbf is there because CP needs them [16:54:18] if it's rare, it should be fine [16:54:23] Aaron moved the CP wait from eager to lazy last week I think. It used to try to apply it directly now it waits for you to make a connection and then apply the loaded pos. I think? [16:55:00] Might be more dead code left behind appearing to do it [16:55:11] Or we missed non-dead logic :) [16:55:16] also we should take a look why cp is enabled in the passive dc (codfw back then) [16:55:44] it makes an expensive db call there [17:29:15] Hmm... not sure about that. We don't replicate the cookie-key in memc, and if there's a cookie you'd also have a sticky CDN cookie along with it. Would be good to have a repro case for that [17:29:38] That is... we probably have a mismatch between theory and reality simewheee