[06:51:21] <godog>	 thank you Cole
[07:46:33] <godog>	 I've asked the mobileapps change to be reverted, lag is going down in codfw
[13:45:14] <slyngs>	 godog: I've silenced and alarm, for a rather long time, and now I kinda want to remove the silence, but can't seem to figure out how. This one: https://alerts.wikimedia.org/?q=%40cluster%3Dwikimedia.org&q=team%3Dinfrastructure-foundations&q=alertname%3DNetboxAccounting&q=%40state%3Dsuppressed
[13:45:42] <slyngs>	 Oh, nevermind, the button is just the reverse of what I thought
[13:48:12] <godog>	 slyngs: hah ok, I take it you were able to delete the silence now ?
[13:48:29] <godog>	 at any rate, bell icon on top right then "browse" will show you silences
[13:49:08] <slyngs>	 Yeah, there's the ^ icon, I took that to mean that I could colaps the box, it is in fact "expand" 
[13:49:31] * godog nods
[15:58:45] <sukhe>	 hi folks. I have an alert that is currently firing as per the rule but it's not alerting anywhere. what am I missing?
[15:58:50] <sukhe>	 alert in question: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/701ff8ba1aaec34911a117a6f62a4dba78dffe44/team-traffic/dnsbox.yaml
[15:58:54] <sukhe>	 severity is set to crtiical
[15:58:56] <sukhe>	 *critical
[15:59:18] <sukhe>	 if I manually copy the expr and run here, https://grafana-rw.wikimedia.org/alerting/new/alerting?returnTo=%2Falerting%2Flist I see it firing
[15:59:28] <sukhe>	 however I expected it to alert but it's actually not alerting.
[16:00:22] <sukhe>	 I have the right team as well
[16:00:32] <sukhe>	 ah ok I see it now. that took some time
[16:00:43] <sukhe>	 12:00:20 <+jinxer-wm> FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns7002:9100 - 
[16:01:41] <sukhe>	 I wonder why it took so much time (~5 mins?) to go from the rule being matched to an actual alert. 
[21:52:22] <cwhite>	 This confuses me a some as well.  DnsboxServiceMismatch doesn't have a "for" setting which makes me wonder if prometheus sets a default (of 5m)?
[21:54:37] <cwhite>	 I haven't found a reference to a default in docs, but there I do see three instances where it should have fired, but only one alert actually reached AlertManager.  That one about 7m after the metric reached the alerting threshold.