[08:46:40] <hashar>	 godog: alert manager apparently has some issue: jinxer-wm> (DatasourceError) firing: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError :)
[08:47:15] <hashar>	 apparently with the error `failed to build query 'A': database is locked`
[09:20:49] <godog>	 hashar: interesting, thank you for reporting, I'll look into that
[09:21:12] <godog>	 hashar: where did you see the error ?
[09:21:27] <hashar>	 godog: irc notification in #wikimedia-releng
[09:21:50] <hashar>	 the dashboard showed a few more alarms, one was for some wikibase/wikidata issue
[09:22:00] <hashar>	 but it is empty now so I guess the issue self solved / the lock vanished
[09:22:27] <hashar>	 we had an other in -releng: 08:49:54(UTC) <jinxer-wm> (DatasourceError) resolved: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
[09:22:45] <godog>	 indeed, I believe that's a grafana alert sending to alertmanager, in other words alertmanager itself is fine and grafana (its alerts) are not
[09:23:01] <godog>	 confirmed that I can't see the alert anymore
[10:11:33] <claime>	 godog: metrics dir sizes have increased quite a bit on prometheus1006 (575M/day before august 14, 1+GB/day after the 15), did something change?
[10:11:47] <claime>	 (just saw the disk space warning, don't know if it's important)
[10:14:22] <godog>	 claime: yes we enabled cadvisor on k8s nodes, kinda expected
[10:14:33] <claime>	 ack
[10:14:35] <godog>	 claime: thank you for the notice though, I'll bump the disk space
[10:15:18] <godog>	 ah nevermind the alert is for services instance, curious
[10:15:41] <godog>	 but fine overall, I'll bump that instead
[12:12:14] <godog>	 hashar: not sure exactly what's going on btw, please let us know if it happens again though
[16:44:33] <hashar>	 godog: I will :) thanks :)