[00:09:33] Krinkle: my plan for today was X-Wikimedia-Debug testing of multi-DC requests, but ideally we would have DependencyStore in its final configuration first [00:09:55] so I was waiting to see if you would deploy your cherry pick [00:10:26] if you're not going to do it today, I can do it, or I can find something else to do [00:12:35] TimStarling: checking schedule, doing it now [00:13:07] thanks [00:27:40] in grafana is there a reason for using bar graphs with hundreds of bars for continuous time series data? [00:27:55] e.g. https://grafana-rw.wikimedia.org/d/000000580/apache-backend-timing?orgId=1&viewPanel=2 [00:28:07] to me that should be a line graph [00:42:05] confirmed via verbose log that e.g. `load.php?modules=skins.monobook.styles&only=styles` still does changeTTL on mwdebug1001 but not 1002, and that using a new lang/skin combo does still produce the appropiate INSERT. [00:42:22] rolling out now, and prepping config change [00:52:48] prod error noise, ref T249745, but going ahead [00:52:49] T249745: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745 [01:08:07] Scap being broken by default, ref T310835, let's assume that's normal too and go ahead [01:08:07] T310835: Scap pool counter error while backporting - https://phabricator.wikimedia.org/T310835 [01:08:15] did a second sync just in case [01:11:59] and as second time for hte config change as well, despite no scap warnings, because ref T311788 is also still unfixed [01:11:59] T311788: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 [01:12:14] so much breakage [01:12:25] I'll work on T311788 next, instead of the session stuff I promised you. [01:14:15] the disk read/write graph is helpful, thanks for adding that ot the Host overview dashboard [01:18:03] mysql aggr: 200->400/s rows written (yesterday: 4K/s row writes) [01:18:10] for the last hour I've been reviewing a lot of dashboards, making little tweaks [01:18:39] currently I'm migrating the "Application Servers RED" dashboard to thanos, allowing multiple selection of site eqiad+codfw [01:18:54] seems like we will need that for multi-DC [01:19:48] ack, it's an endless work, adding useful infos, fixing label consistencies, removing confusing labels in favour of more common terms we use elsewhere, adding units or verifying current units are correct. Enabling or disabling stacking, adding or removing zero binding, disabling confusing "connected" lines over nulls. [01:20:35] I tend to prefer bars for quantity rates like requests, esp when stacked, over e.g. many lines going through each other. It depends on whether the total is interesting or not I guess, and whether it is useful to have a visual distribution of that total or not. [01:21:30] you can always shade the area [01:21:41] Though I sometimes punt towards adding a pie chart instead over the total for the currenet dashboard range, as on https://grafana.wikimedia.org/d/000000002/api-backend-summary [01:21:58] seems like a bar graph with 1200 bars is pretty much the same as an area graph [01:22:24] if it's stacked, then yes, for sure, that'd be equally fine imho. [01:24:45] https://usercontent.irccloud-cdn.com/file/4umaheWI/grafana%202022-02-07%20ApiStash%20after.png https://usercontent.irccloud-cdn.com/file/mT6Z3KLN/grafana%202022-02-07%20ApiStash%20before.png [01:25:07] ^ example of a case where the "default" of shaded lines are imho not useful or readable [01:26:23] esp when there's a few rare metrics stacked in between it's not obvious what the big area refers to [01:29:45] ok, well hosts and mysql etc all look fine from my side. Will catch some daylight while I can and look at wmf-config next [04:11:52] I need to turn off read-only mode in codfw [04:13:00] ->#ms