[00:24:00] (ThanosStoreObjstoreOperationLatencyHigh) firing: Thanos Store is having high latency for bucket operations. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/e832e8f26403d95fac0ea1c59837588b/thanos-store - https://alerts.wikimedia.org/?q=alertname%3DThanosStoreObjstoreOperationLatencyHigh [00:34:00] (ThanosStoreObjstoreOperationLatencyHigh) resolved: Thanos Store is having high latency for bucket operations. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/e832e8f26403d95fac0ea1c59837588b/thanos-store - https://alerts.wikimedia.org/?q=alertname%3DThanosStoreObjstoreOperationLatencyHigh [00:36:11] (ThanosRuleIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosRuleIsDown [00:36:11] (ThanosQueryIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosQueryIsDown [00:36:11] (ThanosSidecarIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosSidecarIsDown [00:36:16] (ThanosCompactIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosCompactIsDown [00:36:20] (ThanosStoreIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosStoreIsDown [00:45:42] (SystemdUnitFailed) firing: curator_actions_cluster_wide.service Failed on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:35:39] I've disabled the test bot, apologies for the alert spam. [04:36:01] (ThanosSidecarIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosSidecarIsDown [04:36:01] (ThanosCompactIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosCompactIsDown [04:36:01] (ThanosRuleIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosRuleIsDown [04:36:11] (ThanosQueryIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosQueryIsDown [04:36:11] (ThanosStoreIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosStoreIsDown [04:45:42] (SystemdUnitFailed) firing: curator_actions_cluster_wide.service Failed on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:48:29] (RSyslogDeliveryFailure) firing: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure [04:53:29] (RSyslogDeliveryFailure) resolved: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure [07:59:29] (RSyslogDeliveryFailure) firing: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure [08:04:29] (RSyslogDeliveryFailure) resolved: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure [08:36:11] (ThanosRuleIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosRuleIsDown [08:36:11] (ThanosSidecarIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosSidecarIsDown [08:36:11] (ThanosCompactIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosCompactIsDown [08:36:16] (ThanosQueryIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosQueryIsDown [08:36:20] (ThanosStoreIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosStoreIsDown [08:45:57] (SystemdUnitFailed) firing: curator_actions_cluster_wide.service Failed on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:45:57] (SystemdUnitFailed) firing: curator_actions_cluster_wide.service Failed on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:03:29] (RSyslogDeliveryFailure) firing: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure [16:08:29] (RSyslogDeliveryFailure) resolved: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure [16:45:57] (SystemdUnitFailed) firing: curator_actions_cluster_wide.service Failed on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:30:42] (SystemdUnitFailed) resolved: (2) ifup@ens13.service Failed on logstash1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:31:42] (SystemdUnitFailed) firing: ifup@ens13.service Failed on logstash1024:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:35:57] (SystemdUnitFailed) firing: (4) ifup@ens13.service Failed on logstash1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:40:57] (SystemdUnitFailed) resolved: (4) ifup@ens13.service Failed on logstash1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:53:12] (SystemdUnitFailed) firing: (3) ifup@ens13.service Failed on logstash1025:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:55:57] (SystemdUnitFailed) resolved: (3) ifup@ens13.service Failed on logstash1025:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:03:12] (SystemdUnitFailed) firing: (3) ifup@ens13.service Failed on logstash1032:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:05:57] (SystemdUnitFailed) resolved: (3) ifup@ens13.service Failed on logstash1032:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:11:42] (SystemdUnitFailed) firing: statograph_post.service Failed on alert2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:16:42] (SystemdUnitFailed) resolved: statograph_post.service Failed on alert2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed