[08:40:12] <jinxer-wm>	 FIRING: ThanosQueryHttpRequestQueryRangeErrorRateHigh: Thanos Query is failing to handle requests. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/af36c91291a603f1d9fbdabdd127ac4a/thanos-query - https://alerts.wikimedia.org/?q=alertname%3DThanosQueryHttpRequestQueryRangeErrorRateHigh
[08:44:12] <jinxer-wm>	 FIRING: ThanosQueryRangeLatencyHigh: Thanos Query has high latency for queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/af36c91291a603f1d9fbdabdd127ac4a/thanos-query - https://alerts.wikimedia.org/?q=alertname%3DThanosQueryRangeLatencyHigh
[08:45:12] <jinxer-wm>	 RESOLVED: ThanosQueryHttpRequestQueryRangeErrorRateHigh: Thanos Query is failing to handle requests. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/af36c91291a603f1d9fbdabdd127ac4a/thanos-query - https://alerts.wikimedia.org/?q=alertname%3DThanosQueryHttpRequestQueryRangeErrorRateHigh
[08:45:36] <tappof>	 &away
[08:49:12] <jinxer-wm>	 RESOLVED: ThanosQueryRangeLatencyHigh: Thanos Query has high latency for queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/af36c91291a603f1d9fbdabdd127ac4a/thanos-query - https://alerts.wikimedia.org/?q=alertname%3DThanosQueryRangeLatencyHigh
[10:46:43] <arnaudb>	 👋 https://gerrit.wikimedia.org/r/c/operations/alerts/+/1098479 I got a new rule that would benefit from a review, if somebody has time to check it, the aim is to raise awareness when a host becomes unfit for production due to a too low innodb buffer pool usage
[11:35:46] <tappof>	 arnaudb: Just reviewed, thank you.
[11:43:13] <arnaudb>	 thanks tappof !
[17:49:11] <jinxer-wm>	 FIRING: SystemdUnitFailed: statograph_post.service on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:52:06] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: statograph_post.service on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:53:12] <jinxer-wm>	 FIRING: ThanosQueryHttpRequestQueryRangeErrorRateHigh: Thanos Query is failing to handle requests. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/af36c91291a603f1d9fbdabdd127ac4a/thanos-query - https://alerts.wikimedia.org/?q=alertname%3DThanosQueryHttpRequestQueryRangeErrorRateHigh
[17:58:12] <jinxer-wm>	 RESOLVED: ThanosQueryHttpRequestQueryRangeErrorRateHigh: Thanos Query is failing to handle requests. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/af36c91291a603f1d9fbdabdd127ac4a/thanos-query - https://alerts.wikimedia.org/?q=alertname%3DThanosQueryHttpRequestQueryRangeErrorRateHigh