[08:36:06] (ThanosQueryIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosQueryIsDown [08:36:11] (ThanosCompactIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosCompactIsDown [08:36:15] (ThanosStoreIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosStoreIsDown [08:36:20] (ThanosRuleIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosRuleIsDown [08:36:25] (ThanosSidecarIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosSidecarIsDown [11:33:29] (RSyslogDeliveryFailure) firing: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure [11:38:29] (RSyslogDeliveryFailure) resolved: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure [12:36:01] (ThanosStoreIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosStoreIsDown [12:36:01] (ThanosSidecarIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosSidecarIsDown [12:36:11] (ThanosQueryIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosQueryIsDown [12:36:11] (ThanosCompactIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosCompactIsDown [12:36:11] (ThanosRuleIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosRuleIsDown [13:13:29] (RSyslogDeliveryFailure) firing: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure [13:18:29] (RSyslogDeliveryFailure) resolved: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure [14:50:00] (ThanosStoreObjstoreOperationLatencyHigh) firing: Thanos Store is having high latency for bucket operations. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/e832e8f26403d95fac0ea1c59837588b/thanos-store - https://alerts.wikimedia.org/?q=alertname%3DThanosStoreObjstoreOperationLatencyHigh [14:55:00] (ThanosStoreObjstoreOperationLatencyHigh) resolved: Thanos Store is having high latency for bucket operations. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/e832e8f26403d95fac0ea1c59837588b/thanos-store - https://alerts.wikimedia.org/?q=alertname%3DThanosStoreObjstoreOperationLatencyHigh [16:36:01] (ThanosStoreIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosStoreIsDown [16:36:11] (ThanosRuleIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosRuleIsDown [16:36:11] (ThanosCompactIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosCompactIsDown [16:36:11] (ThanosQueryIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosQueryIsDown [16:36:16] (ThanosSidecarIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosSidecarIsDown [16:50:19] hah surely the test bot shouldn't be here [20:36:01] (ThanosRuleIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosRuleIsDown [20:36:11] (ThanosQueryIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosQueryIsDown [20:36:11] (ThanosSidecarIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosSidecarIsDown [20:36:16] (ThanosCompactIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosCompactIsDown [20:36:20] (ThanosStoreIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosStoreIsDown [22:43:29] (RSyslogDeliveryFailure) firing: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure [22:48:29] (RSyslogDeliveryFailure) resolved: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure