[00:19:25] PROBLEM - Check systemd state on an-web1001 is CRITICAL: CRITICAL - degraded: The following units failed: hardsync-published.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:30:26] RECOVERY - Check systemd state on an-web1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [01:25:00] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: monitor_refine_event.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:38:12] (VarnishkafkaNoMessages) firing: (4) varnishkafka on cp2027 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:43:22] (VarnishkafkaNoMessages) resolved: (6) varnishkafka on cp2027 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:44:15] (VarnishKafkaDeliveryErrors) firing: (2) varnishkafka has cache_text errors on cp5018:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [07:44:15] (VarnishKafkaDeliveryErrors) firing: varnishkafka has cache_upload errors on cp5029:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?panelId=20&fullscreen&orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp5029 - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [07:44:20] (VarnishKafkaDeliveryErrors) firing: varnishkafka has cache_upload errors on cp5031:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?panelId=20&fullscreen&orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp5031 - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [07:50:11] (VarnishKafkaDeliveryErrors) firing: (7) varnishkafka has cache_upload errors on cp5025:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [07:50:15] (VarnishKafkaDeliveryErrors) firing: (6) varnishkafka has cache_text errors on cp5017:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [07:54:50] (VarnishKafkaDeliveryErrors) resolved: (2) varnishkafka has cache_upload errors on cp5029:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [07:54:54] (VarnishKafkaDeliveryErrors) firing: (7) varnishkafka has cache_upload errors on cp5025:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [07:54:59] (VarnishKafkaDeliveryErrors) firing: (8) varnishkafka has cache_text errors on cp5017:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [07:59:15] (VarnishKafkaDeliveryErrors) firing: (7) varnishkafka has cache_upload errors on cp5025:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [08:09:15] (VarnishKafkaDeliveryErrors) firing: (7) varnishkafka has cache_upload errors on cp5025:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [08:24:15] (VarnishKafkaDeliveryErrors) resolved: (2) varnishkafka has cache_upload errors on cp5028:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [11:54:15] (VarnishKafkaDeliveryErrors) firing: (8) varnishkafka has cache_text errors on cp5017:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [15:54:15] (VarnishKafkaDeliveryErrors) firing: (8) varnishkafka has cache_text errors on cp5017:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [19:54:15] (VarnishKafkaDeliveryErrors) firing: (8) varnishkafka has cache_text errors on cp5017:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [20:09:15] (VarnishKafkaDeliveryErrors) firing: (8) varnishkafka has cache_text errors on cp5017:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [20:36:53] (VarnishkafkaNoMessages) firing: varnishkafka on cp4044 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp4044%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:41:35] (VarnishKafkaDeliveryErrors) firing: varnishkafka has cache_upload errors on cp5026:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?panelId=20&fullscreen&orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp5026 - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [20:42:16] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4044 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp4044%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:45:12] (VarnishKafkaDeliveryErrors) firing: varnishkafka has cache_upload errors on cp5026:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?panelId=20&fullscreen&orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp5026 - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [20:46:09] (VarnishKafkaDeliveryErrors) firing: (7) varnishkafka has cache_text errors on cp5017:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [20:50:11] (VarnishKafkaDeliveryErrors) resolved: (2) varnishkafka has cache_upload errors on cp5026:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [20:50:40] (VarnishKafkaDeliveryErrors) resolved: (2) varnishkafka has cache_upload errors on cp5026:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [20:54:15] (VarnishKafkaDeliveryErrors) firing: (7) varnishkafka has cache_text errors on cp5017:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [20:59:15] (VarnishKafkaDeliveryErrors) firing: (7) varnishkafka has cache_text errors on cp5017:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [21:04:15] (VarnishKafkaDeliveryErrors) firing: (6) varnishkafka has cache_text errors on cp5017:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [21:09:15] (VarnishKafkaDeliveryErrors) firing: (6) varnishkafka has cache_text errors on cp5017:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors [21:14:15] (VarnishKafkaDeliveryErrors) resolved: (4) varnishkafka has cache_text errors on cp5018:9132 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishKafkaDeliveryErrors