[17:23:03] <jinxer-wm>	 FIRING: ErrorBudgetBurn: logstash-availability eqiad - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[17:26:43] <jinxer-wm>	 FIRING: BenthosKafkaConsumerLag: Too many messages in jumbo-eqiad for group benthos-webrequest_live - TODO - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=jumbo-eqiad&var-datasource=eqiad%20prometheus/ops&var-consumer_group=benthos-webrequest_live - https://alerts.wikimedia.org/?q=alertname%3DBenthosKafkaConsumerLag
[17:41:43] <jinxer-wm>	 RESOLVED: BenthosKafkaConsumerLag: Too many messages in jumbo-eqiad for group benthos-webrequest_live - TODO - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=jumbo-eqiad&var-datasource=eqiad%20prometheus/ops&var-consumer_group=benthos-webrequest_live - https://alerts.wikimedia.org/?q=alertname%3DBenthosKafkaConsumerLag
[17:48:03] <jinxer-wm>	 RESOLVED: ErrorBudgetBurn: logstash-availability eqiad - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[18:17:33] <jinxer-wm>	 FIRING: ErrorBudgetBurn: logstash-availability eqiad - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[18:30:54] <cwhite>	 Something's not quite adding up about that errorbudgetburn alert.  One of six logstash hosts wasn't ingesting but we didn't experience a service outage.
[18:53:39] <herron>	 cwhite: I'm thinking we should wrap that recording rule expression in a sum by (site), current expr is on the top and proposed on the bottom https://w.wiki/E3fW
[19:02:33] <jinxer-wm>	 RESOLVED: ErrorBudgetBurn: logstash-availability eqiad - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[19:04:10] <cwhite>	 herron: that sounds reasonable.  Should be better than allowing a single host to hit the SLO so hard :)
[19:35:31] <swfrench-wmf>	 hi folks - is there anything amiss with grafana at the moment? folks are reporting that it's unavailable, and I'm getting an envoy 503 when attempting to access grafana.wm.o
[19:36:27] <denisse>	 Hi swfrench-wmf , I recently updates it to Grafana V11. I tested and everything looked fine, but let me take a look again.
[19:36:42] <swfrench-wmf>	 ack, thank you!
[19:51:30] <denisse>	 swfrench-wmf: Could you please try to access grafana again?
[19:51:44] <swfrench-wmf>	 denisse: up for me!
[19:51:53] <denisse>	 swfrench-wmf: Thank you!
[19:51:58] <swfrench-wmf>	 out of curiosity, what was it? :)
[19:52:02] <swfrench-wmf>	 thank _you_!
[19:53:44] <denisse>	 It seems like envoy was trying to use the previous grafana version, after the Grafana upgrade the grafana server was restarted but envoyproxy was not. Grafana introduced new headers in version 10 which envoy didn't know about until it was restarted.
[19:55:07] <denisse>	 I tested grafana-rw and it works fine. Now both grafana hosts have v11.6.1.
[19:55:26] <denisse>	 I'm puzzled as to why I didn't had to reboot envoy after upgrading the first host tho...
[19:55:31] <denisse>	 restart*
[19:57:45] <inflatador>	 denisse Looks like it's working, thanks for your help!
[19:58:08] <denisse>	 inflatador: Awesome, thank you!
[20:01:15] <denisse>	 swfrench-wmf: Just FYI, I'm still trying to figure out the root cause, looking at the grafana logs I see there's an envoyproxy restarter service, so maybe it wasn't that envoy didn't restart. It seems like it did, I'm wondering if it could be related to the Grafana service not reloading correctly after the package upgrade, I'll keep you posted.
[20:01:21] <denisse>	 > May 08 19:51:01 grafana1002 envoyproxy-hot-restarter
[20:04:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: stunnel4.service on grafana1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[20:04:48] <denisse>	 ^ On it.
[20:07:25] <denisse>	 It must resolve soon.
[20:09:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: stunnel4.service on grafana1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[20:22:16] <denisse>	 Okay, so looking at the logs it seems like envoyproxy-hot-restarter didn't trigger on time after the grafana upgrade. Grafana was upgraded at `19:47:02` and performed migrations successfully but from that same time period the oldest envoyproxy log is `May 08 19:47:30 grafana1002 systemd[1]: Stopping envoyproxy.service - Envoy proxy...` which is when I manually restarted the `envoyproxy` service.
[20:23:01] <denisse>	 So that's why the issue happened. Now, I'll investigate why `envoyproxy-hot-restarter` didn't trigger after the upgrade, it seems like it didn't detect it.
[21:41:40] <swfrench-wmf>	 interesting! I take it there was a configuration change that needed to be picked up by envoy in conjunction with the grafana upgrade?