[14:07:04] anyone else around to look at that alert? I need to leave in 15 [14:11:08] restarting scholia tool since that's what was causing the noise last time... [14:23:22] dhinus: around? I'm trying to replay your process from T429738 but not getting the graphs you got [14:23:22] T429738: ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738 [14:33:45] I'm less convinced that scholia is the sole cause of the issue but stopping it has cleared the alert so I'm leaving it stopped, left another note on https://github.com/WDscholia/scholia/issues/2794 [14:33:50] and... now I need to go [14:47:30] sorry, I'm here [14:47:45] thank andrewbogot.t [15:40:32] anyhow, it seems fourohfour is holding, the number of sessions is high, but steady, so I'll log off for now and keep an eye in case it increases [15:59:19] thanks a.ndrew and dcar.o [15:59:34] it looks like it was 90% driven by scholia [15:59:59] I quickly created a new dashboard to reproduce the graphs I used in T429738, so they are easier to find [15:59:59] T429738: ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738 [16:00:05] https://grafana-rw.wmcloud.org/d/fnhp8st/tool-error-rates