[06:32:54] Krinkle: https://wikitech.wikimedia.org/wiki/Performance/Runbook/Grafana_best_practices#MediaWiki_deployments are the timestamps in UTC? I am trying to match up some entries I see in Grafana with https://sal.toolforge.org/production [06:48:35] kostajh: internally yes, data and annotations both UTC. Whether your Ui does depends on the dashboard as a whole. See general settings. [06:55:52] Krinkle: thanks for clarifying. The UI for my dashboard is on UTC [06:57:15] There's a MW deploy `scap` annotation for `2022-09-06 03:09:00`, but I don't see a corresponding log entry in https://sal.toolforge.org/production?p=7&q=&d=2022-09-06 (which is intermittently 500'ing at the moment, T296072) [06:57:15] T296072: SAL tool giving intermittent 500 Internal Server Errors - https://phabricator.wikimedia.org/T296072 [07:18:38] kostajh: https://wikitech.wikimedia.org/wiki/Server_Admin_Log#2022-09-06 [07:18:54] I see a scap there from 03:02 to 03:12 [07:19:48] 03:38* even [07:21:01] I don't know what time exactly the annotation is created. There's also possibility of rounding/resolution loss in graphite after 7 days, snaps to 15min windows, but we're still away from that. [07:21:42] Although depending on zoom level and date range in grafana it does affect you sooner as well [07:22:09] Anyway, if it's annotated as a full scap and not a file sync, I assume it's this [07:38:47] Krinkle: ok, thanks. That was my thought too. The discussion in T317187 has pretty much ruled out wmf.28 as a factor in the increased load times we see for Special:Homepage, but "Started scap: testwikis wikis to 1.39.0-wmf.28 refs" seems to pretty clearly show up just before the spikes [07:38:47] T317187: GrowthExperiments Special:Homepage: investigate performance regression since September 6 2022 - https://phabricator.wikimedia.org/T317187 [10:33:03] kostajh: ack, without a theory as to how to effects other wikis it does seem like a long shot. It's not impossible but yeah I agree it aligns a bit more than we previously thought [10:33:22] The sync is not even to group0 as a whole [21:51:19] AaronSchulz: TimStarling: https://make.wordpress.org/core/2022/09/12/lets-make-wordpress-officially-support-sqlite/ [21:51:26] Amir1: ^ [21:53:50] While I disagree on sqlite merits but yeah. We should support more dbmses [21:54:34] They current don't have an abstraction layer for neither queries nor schemas. [21:55:05] It's basically formatted strings with some escaping utils [21:56:25] And their stable interface is: keep forever. They still ship deprecated functions from 15y ago and support those [21:56:58] Which is honestly amazing since it's all implied global state functions in skin templates [21:58:36] It's somewhat simpler than us since their default is internal instead of public/stable like us. And I think they do deprecate and remove stuff that us plug-in facing rather than skin facing. Eg plug-ins regularly use internal stuff and those come/go. [21:59:15] But it's very useful that the heart of it can go on indefinitely once you upload a site with a theme you build.