[06:51:21] thank you Cole [07:46:33] I've asked the mobileapps change to be reverted, lag is going down in codfw [13:45:14] godog: I've silenced and alarm, for a rather long time, and now I kinda want to remove the silence, but can't seem to figure out how. This one: https://alerts.wikimedia.org/?q=%40cluster%3Dwikimedia.org&q=team%3Dinfrastructure-foundations&q=alertname%3DNetboxAccounting&q=%40state%3Dsuppressed [13:45:42] Oh, nevermind, the button is just the reverse of what I thought [13:48:12] slyngs: hah ok, I take it you were able to delete the silence now ? [13:48:29] at any rate, bell icon on top right then "browse" will show you silences [13:49:08] Yeah, there's the ^ icon, I took that to mean that I could colaps the box, it is in fact "expand" [13:49:31] * godog nods [15:58:45] hi folks. I have an alert that is currently firing as per the rule but it's not alerting anywhere. what am I missing? [15:58:50] alert in question: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/701ff8ba1aaec34911a117a6f62a4dba78dffe44/team-traffic/dnsbox.yaml [15:58:54] severity is set to crtiical [15:58:56] *critical [15:59:18] if I manually copy the expr and run here, https://grafana-rw.wikimedia.org/alerting/new/alerting?returnTo=%2Falerting%2Flist I see it firing [15:59:28] however I expected it to alert but it's actually not alerting. [16:00:22] I have the right team as well [16:00:32] ah ok I see it now. that took some time [16:00:43] 12:00:20 <+jinxer-wm> FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns7002:9100 - [16:01:41] I wonder why it took so much time (~5 mins?) to go from the rule being matched to an actual alert. [21:52:22] This confuses me a some as well. DnsboxServiceMismatch doesn't have a "for" setting which makes me wonder if prometheus sets a default (of 5m)? [21:54:37] I haven't found a reference to a default in docs, but there I do see three instances where it should have fired, but only one alert actually reached AlertManager. That one about 7m after the metric reached the alerting threshold.