[09:20:52] Just upgraded python3-conftool and spicerack on cumin hosts as FYI :) [09:20:57] lemme know if you see any issues [09:35:12] <_joe_> elukey: did you upgrade also dbctl? [09:35:21] <_joe_> that's the piece I'm slightly nervous about :) [09:37:37] good point, I didn't, do you prefer me to do it on cumin2002 and test? [09:40:46] _joe_ --^ [09:41:36] <_joe_> elukey: uhh yes dbctl needs to be at the same version as conftoo [09:41:46] <_joe_> yet another reason I hate packaging python as debs [09:44:27] no sorry I am stupid, installing python3-conftool upgraded all the dependent packages [09:44:30] already good [09:44:42] to be fair, the Debian packaging system can cope with "these two packages need to be the same version" [09:44:47] Ah, good. [09:45:04] yes the problem is definitely me, not debian packaging :D [15:30:37] does someone know where I can find a dashboard for the edit latency or the prom metric for calculating it [15:30:43] essentially I am trying to see how much is the edit latency in general and even better if I can filter it by site [15:30:57] I see various edit count related dashboards but not the latency ones [15:41:23] Someone more knowledgable, please correct me, but I *think* that metric is `mediawiki_action_executeTiming_seconds_sum{action="edit"}` [15:41:50] cwhite: thank you! it's a start so that helps [15:43:19] only seeing action view though hmm [15:43:48] Feels like something Krinkle may know [15:44:42] thanks! [15:44:57] (since you did the ping already) [15:46:03] sukhe: https://grafana.wikimedia.org/d/000000429/backend-save-timing?orgId=1&from=now-24h&to=now&timezone=utc I think [15:46:33] mediawiki_WikimediaEvents_editResponseTime_seconds_bucket [15:46:55] thanks! exploring this one [15:48:28] also I think for mediawiki_action_executeTiming_seconds_sum you actually want `{action="submit"}` [15:50:19] mediawiki_site_stats_total{engagement="edits"} new one to me [15:52:20] yeah I guess I have to play around with these a bit since I don't know which I should be looking at :] [15:57:23] 👋 is it possible to (retroactively) see how many resources a given job i started via mwscript-k8s consumed? Specifically, regarding RAM. Is https://grafana.wikimedia.org/goto/j5rpPozvR?orgId=1 a reasonable chart to use for that? [15:57:43] hm I'm not sure about the action submit vs edit now [15:58:07] urbanecm: yep! [15:58:14] Thanks! [15:58:30] cdanis: yeah no worries, I will wait for Krink.le. I think he is presenting at the meeting today will ping him later. [15:58:58] since I guess I will have to look what these means and perhaps a more direct answer is better [16:00:04] what I am trying to do here is essentially estimate the edit latency in general and then try to see the overhead of another service, from edges to the cores [16:02:41] there's a histogram recording rule le_kubernetes_namespace:mediawiki_WikimediaEvents_editResponseTime_seconds_bucket:rate5m{} [16:05:55] thanks, looking [16:22:19] sukhe: https://www.mediawiki.org/wiki/Manual:Parameters_to_index.php#Actions:~:text=Shows%20the%20editor%20for%20the%20page. [16:26:19] cdanis: you mean this as way to measure the edit time by virtue of the editor being loaded? [16:26:58] sukhe: no, but that documents the difference between action=edit (which is the editor) and action=submit (which is mostly just saves) in the action_executeTiming metrics [16:27:08] ahhh [16:27:26] yeah that explains it clearly [16:28:38] also I suspect the WikimediaEvents one is recorded client-side [16:34:38] yeah I am lost again so giving up now. I wil live with the vague estimations from the backend-save-time one :) [16:38:02] mediawiki_action_executeTiming_seconds_sum{action=edit} measure the time to render the wikitext edit form, usually GET requests. It doesn't cover lots of other ways to edit, and doesn't cover actually submitting that work. [16:38:29] action=submit does include some edits, but also lots of form opens, or previews, diffs, failed submissions, etc. [16:38:43] and only desktop wikitext ones, not anything else like VE or mobile. [16:39:01] in this case, the GET request does work for me since essentially all I am trying to estimate is the latency from edges to cores, for the edit workflow [16:39:16] do you think that is a fair estimation? I am not looking for exact numbers, just some rough ones [16:39:18] mediawiki_WikimediaEvents_editResponseTime is specifically latency for POST requests that succesfully create a new revision, both wikitext, VE, mobile, APP, API. [16:39:34] ok [16:40:24] it is a backend metric so time from PHP request start to response end [16:41:01] you mean mediawiki_WikimediaEvents_editResponseTime? [16:41:14] anything mediawiki_* is measured inside MediaWiki PHP. [16:41:26] ok thanks [16:41:43] well, a few come from statsv.js so some from JS, but I don't expect anyhthing to include or measure only time elsewhere in infra [16:43:54] I don't know of an obvious way to measure base latency between cdn nodes and mediawiki nodes. I mean you could measure it emperically with curl using some cheap-as-possible MW url. If you want the minimum, I suppose we might have latency data somewhere for response times from Varnish, looking at cache misses for /w/load.php or /w/rest.php and taking the lowest numbers. Although numbers under 5ms might be hard if there's no buckets for it. [16:44:24] * Krinkle mutes until after staff meeting :) [16:44:29] thanks and gl :) [16:45:17] for later, like I mentioned I am looking for rough estimates, to see how bringing up a new service in the edit workflow on the edges can help improve edit times [16:45:49] well not directly in the edit workflow but related to it (hcaptcha) [16:46:34] (I mean, it is in the edit workflow in that respect? but the request path is separate and hence) [19:01:58] ATTENTION: Do not run puppet for the next 10 minutes. We are migrating puppetserver1001's primary network connection from the old to new switch stacks. There is less than 3 seconds disruption to connectivity expected. [19:02:09] will update when done [19:02:48] ok! [19:03:57] mov ecomplete [19:04:05] puppetserver1001 move complete [19:06:36] puppet run works for me [19:18:04] I'm messing with autoinstall files on apt1002, please let me know if you need to reimage something and I'll back out my changes.