[00:05:12] (VarnishkafkaNoMessages) firing: varnishkafka for instance cp3064:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-source=statsv&var-cp_cluster=cache_text&var-instance=cp3064:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:10:12] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:15:12] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:20:12] (VarnishkafkaNoMessages) firing: (5) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:25:12] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:30:12] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:35:12] (VarnishkafkaNoMessages) firing: (5) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:40:12] (VarnishkafkaNoMessages) firing: (5) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:45:12] (VarnishkafkaNoMessages) resolved: (4) varnishkafka for instance cp3054:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:45:42] (VarnishkafkaNoMessages) firing: (2) varnishkafka for instance cp3056:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:50:42] (VarnishkafkaNoMessages) firing: (4) varnishkafka for instance cp3056:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:55:42] (VarnishkafkaNoMessages) firing: (5) varnishkafka for instance cp3052:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:00:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:00:57] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:05:42] (VarnishkafkaNoMessages) firing: (4) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:05:57] (VarnishkafkaNoMessages) firing: (5) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:10:42] (VarnishkafkaNoMessages) firing: (4) varnishkafka for instance cp3052:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:10:57] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:15:42] (VarnishkafkaNoMessages) resolved: (5) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:17:42] (VarnishkafkaNoMessages) firing: (2) varnishkafka for instance cp3056:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:22:42] (VarnishkafkaNoMessages) firing: (5) varnishkafka for instance cp3052:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:27:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:32:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:42:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3054:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:47:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:52:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:57:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:02:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:07:43] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:12:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:17:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:22:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:27:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:32:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:37:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:42:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:47:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:52:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:02:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:07:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:12:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:17:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:22:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:27:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:32:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:37:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:42:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:47:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:52:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:57:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:02:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:07:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:12:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:17:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:22:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:27:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:32:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:37:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:42:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:47:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:52:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:57:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:02:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:07:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:12:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:17:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:22:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:27:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:32:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:37:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:42:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:47:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:52:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:57:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:02:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:07:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:12:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:17:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:22:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:27:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:32:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:37:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:42:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:47:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:52:42] (VarnishkafkaNoMessages) firing: (5) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:57:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:02:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:07:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:12:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:17:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:22:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:27:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:32:33] 10Analytics, 10Data-Engineering, 10SRE: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373 (10fgiunchedi) (my two cents) agreed option 1. seems preferable, and to clarify my position on T276972: I'm not against it per-se, I am questioning the "dc... [07:32:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:37:42] (VarnishkafkaNoMessages) firing: (5) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:42:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:47:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:52:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:57:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3052:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:02:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:07:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:10:17] <_joe_> lol [08:10:24] <_joe_> we depooled esams yesterday [08:10:38] <_joe_> i came here and had a moment of panic [08:12:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:17:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:17:57] <_joe_> btullis: I guess you want to downtime varnishkafka alerts in esams for the time being [08:20:01] <_joe_> although it seems possible actually vk crashed? [08:22:17] <_joe_> it was actually down for some reason heh [08:22:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:27:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:31:19] varnishkafa-statsv is up and running on cp3050 [08:31:36] I'm assuming this is triggered by esams being drained from production traffic [08:32:42] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:37:42] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp3050:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:42:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:47:42] (VarnishkafkaNoMessages) firing: (6) varnishkafka for instance cp3052:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:52:42] (VarnishkafkaNoMessages) firing: (5) varnishkafka for instance cp3052:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:56:04] _joe_: vgutierrez: Yeah, apologies for this noise. I'm working on getting confctl pooled/depooled status into prometheus so that we can filter out intentionally depooled instances. https://phabricator.wikimedia.org/T300246 [08:56:47] <_joe_> btullis: these instances are not depooled [08:56:58] <_joe_> vgutierrez: I did restart vk-statsv on cp3050 [08:57:06] <_joe_> it was indeed logging ssl errors [08:57:18] Oh right. [08:57:22] <_joe_> but... yeah it was before the alert fired again [08:57:40] _joe_: SSL errors? [08:57:42] (VarnishkafkaNoMessages) resolved: (3) varnishkafka for instance cp3056:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:57:43] <_joe_> I think in this case your problem is that the datacenter is not receiving traffic [08:58:06] <_joe_> vgutierrez: "could not connect to kafka" [08:58:14] <_joe_> it could've been a network blip causing that [08:58:21] <_joe_> but after a restart, the alert persists [08:58:30] <_joe_> so it is indeed caused by the absence of traffic [08:58:31] vgutierrez: This sounds like just the kind of error we deployed it to catch. [08:58:47] ack [08:58:56] <_joe_> btullis: it was an error from hours ago, I decided to restart that instance just to be sure [08:59:09] <_joe_> but the reason the alert is firing is that esams is receiving no traffic [08:59:12] _joe_: Can you point me to the process for depooling a whoe datacentre please, so I can think how to integrate that into the alert? [08:59:13] <_joe_> well, almost [08:59:24] <_joe_> btullis: it's done via operations/dns [08:59:34] <_joe_> there's an admin_state file there [08:59:35] Ah, OK. [08:59:56] <_joe_> https://gerrit.wikimedia.org/r/plugins/gitiles/operations/dns/+/refs/heads/master/admin_state#76 [09:05:13] 10Data-Engineering, 10Data-Engineering-Kanban: Add alert for varnishkafka low/zero messages per second to alertmanager - https://phabricator.wikimedia.org/T300246 (10BTullis) We had a genuine alert for this today, from cp3050. {F35033313,width=100%} Investigating now. [09:05:50] !log restarted varnishkafka-eventlogging.service on cp3050 T300246 [09:05:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:05:53] T300246: Add alert for varnishkafka low/zero messages per second to alertmanager - https://phabricator.wikimedia.org/T300246 [09:07:09] OK, the de-pooled datacentre is not a case that I had considered yet. I'm going to have to think about how to deal with that, but in the meantime I'm going to create a silence for it in alertmanager. [09:07:51] _joe_: For how long would you reckon esams will be depooled in this way, as a guess? [09:08:23] <_joe_> btullis: I would guess at least for the weekend? but I'd ask the traffic team [09:08:33] ack, thanks [09:17:42] (VarnishkafkaNoMessages) firing: varnishkafka for instance cp3064:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-source=statsv&var-cp_cluster=cache_text&var-instance=cp3064:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [09:27:02] (03CR) 10Tchanders: [C: 04-1] "Thanks for incrementing the version. We should remove the changes from the files marked version 1.0.0" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/774980 (https://phabricator.wikimedia.org/T296428) (owner: 10AGueyte) [09:39:54] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Update DNS for the DataHub MVP services - https://phabricator.wikimedia.org/T301460 (10BTullis) [09:40:00] 10Data-Engineering, 10Data-Catalog, 10SRE, 10serviceops, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) [10:07:23] btullis: per https://phabricator.wikimedia.org/T304089#7819217 it looks like esams is gonna be depooled till 23:00 UTC today [10:08:51] vgutierrez: Great, thanks. I'll adjust the silence in alertmanager accordingly. The true positive for cp3050 is still troubling me more than all the noise from the false positives though. Thanks for brining it to my attention. [10:10:55] that's a manual operation, so allow some room on those alerts... don't expect it to be at 23.00 sharp :) [10:22:42] (VarnishkafkaNoMessages) firing: varnishkafka for instance cp2037:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=statsv&var-cp_cluster=cache_text&var-instance=cp2037:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [10:23:48] --^ This one is low throughput to for the statsv source in codfw. We had another of these at about this time yesterday. [10:27:42] (VarnishkafkaNoMessages) resolved: varnishkafka for instance cp2037:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=statsv&var-cp_cluster=cache_text&var-instance=cp2037:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [10:50:51] 10Data-Engineering-Radar: Decomission the UploadWizard instrument - https://phabricator.wikimedia.org/T305238 (10phuedx) [10:59:59] 10Data-Engineering-Radar: Decomission the UploadWizard instrument - https://phabricator.wikimedia.org/T305238 (10phuedx) [11:00:22] 10Data-Engineering-Radar: Decomission the UploadWizard instrument - https://phabricator.wikimedia.org/T305238 (10phuedx) [11:00:34] 10Data-Engineering-Radar: Decomission the UploadWizard* instruments - https://phabricator.wikimedia.org/T305238 (10phuedx) [11:05:09] 10Analytics, 10Analytics-Wikistats, 10Data-Engineering, 10Data-Engineering-Kanban: Confusing filtering on "Active editors by country" topic - https://phabricator.wikimedia.org/T300365 (10Pcoombe) 05Resolved→03Open The original task is not resolved, only some other unrelated things that were brought up... [11:19:01] 10Data-Engineering-Radar: Decommission the UploadWizard* instruments - https://phabricator.wikimedia.org/T305238 (10phuedx) [12:24:46] (03PS3) 10Aqu: Add archiving job for Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/774383 (https://phabricator.wikimedia.org/T300039) [12:26:37] 10Data-Engineering, 10Data-Engineering-Kanban, 10Infrastructure-Foundations, 10SRE, and 3 others: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10ayounsi) Documentation updated: https://wikitech.wikimedia.org/wiki/Netflow/sflow [12:40:52] 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, 10Research, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10BTullis) It's clear from the above that we have two distinct use cases that have emerged for the web proxies: | # |... [13:38:09] (03PS4) 10Aqu: Add archiving job for Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/774383 (https://phabricator.wikimedia.org/T300039) [13:49:43] (03PS5) 10Aqu: Add archiving job for Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/774383 (https://phabricator.wikimedia.org/T300039) [14:28:03] 10Analytics-Radar, 10SRE, 10observability: Set up cross DC topic mirroring for Kafka logging clusters - https://phabricator.wikimedia.org/T276972 (10Ottomata) In https://phabricator.wikimedia.org/T304373#7823916 @fgiunchedi wrote > to clarify my position on T276972: I'm not against it per-se, I am questionin... [14:33:08] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Low Risk Oozie Migration: APIs - https://phabricator.wikimedia.org/T300028 (10EChetty) p:05Triage→03Low [14:33:20] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Low Risk Oozie Migration: APIs - https://phabricator.wikimedia.org/T300028 (10EChetty) p:05Low→03Lowest [14:33:30] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Low Risk Oozie Migration: APIs - https://phabricator.wikimedia.org/T300028 (10EChetty) p:05Lowest→03Unbreak! [14:33:44] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Low Risk Oozie Migration: APIs - https://phabricator.wikimedia.org/T300028 (10EChetty) p:05Unbreak!→03Medium [14:36:16] 10Data-Engineering, 10Data-Engineering-Kanban: Add the commons-entity dataset to the refinery-drop-mediawiki-snapshots script - https://phabricator.wikimedia.org/T303993 (10EChetty) p:05Triage→03High [16:16:15] btullis: I've noticed https://gerrit.wikimedia.org/r/c/operations/puppet/+/776225 [16:16:30] btullis: that's aiming to fix the esams issue? [16:18:05] Well, this one is more aiming to fix the single depooled cp-* host first. The depooled datacentre is a case that I hadn't thought of until today :-) [16:18:12] ack [16:18:32] I was going to say that that approach wouldn't work for the depooled DC [16:18:45] as we depool DCs on via DNS [16:19:11] https://gerrit.wikimedia.org/r/c/operations/dns/+/776004 <-- this is how we depooled esams last night [16:19:58] (03PS4) 10AGueyte: Add new event action [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/774980 (https://phabricator.wikimedia.org/T296428) [16:20:39] Cool. Thanks. Yes I was wondering whether we could get this admin_state into prometheus somehow. [16:20:45] (03CR) 10jerkins-bot: [V: 04-1] Add new event action [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/774980 (https://phabricator.wikimedia.org/T296428) (owner: 10AGueyte) [16:21:36] Anyway, I'm going to try to get the confctl integration working first and come back to the depooled data centre case/ [16:30:49] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10MediaWiki-extensions-EventLogging, and 2 others: EventLogging needs to enque events to avoid draining users' battery on mobile - https://phabricator.wikimedia.org/T225578 (10Krinkle) [16:32:44] btullis: ack [20:14:39] (03CR) 10Tchanders: [C: 04-1] Add new event action (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/774980 (https://phabricator.wikimedia.org/T296428) (owner: 10AGueyte) [23:31:45] (03PS1) 10Gerrit maintenance bot: Add kcg.wikipedia to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/776269 (https://phabricator.wikimedia.org/T305281)