[08:39:42] * dhinus paged ToolforgeWebHighErrorRate
[08:42:12] <dhinus>	 the alert flapped a few times
[08:44:08] <dhinus>	 https://usercontent.irccloud-cdn.com/file/1vtdqSWq/Screenshot%202026-06-21%20at%2010.43.41.png
[08:52:34] <dhinus>	 seems to be driven by a single tool ("scholia") https://usercontent.irccloud-cdn.com/file/Jq9VcmpX/Screenshot%202026-06-21%20at%2010.51.53.png
[08:54:46] <dhinus>	 the scholia webservice pod is in CrashLoopBackOff, I stopped it
[09:06:24] <dhinus>	 toolforge error rate is back to normal, I have now restarted the tool
[09:07:01] <dhinus>	 https://usercontent.irccloud-cdn.com/file/ihjx26pb/Screenshot%202026-06-21%20at%2011.04.37.png
[09:07:08] <dhinus>	 detailed analysis is at T429738
[09:07:08] <stashbot>	 T429738: ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738
[09:19:51] <dhinus>	 a few minutes after restarting it, the tool went into CrashLoopBackOff again
[09:20:17] <dhinus>	 I stopped it and filed an issue at https://github.com/WDscholia/scholia/issues/2794
[09:20:52] <dhinus>	 logging off for now
[15:00:12] <andrewbogott>	 ty dhinus! Nice to wake up to an alert already being sorted out