[15:15:33] <claime>	 herron: fair play that socat :D
[15:53:29] <herron>	 ha, needed one adjustment and now is running :)
[15:54:48] <herron>	 https://usercontent.irccloud-cdn.com/file/D7Q3dVLa/giphy-1.gif
[16:02:36] <claime>	 x)
[19:01:47] <ryankemper>	 \o
[19:01:54] <ryankemper>	 With the recent work around setting an SLO for wdqs, search team would like to explore our options for reducing/eliminating paging for WDQS (https://phabricator.wikimedia.org/T325324)
[19:02:03] <ryankemper>	 the tricky part is the presence of generic (non-service-specific alerts) that will fire
[19:02:31] <ryankemper>	 to start with something concrete, in this incident https://wikitech.wikimedia.org/wiki/Incidents/2022-12-12_wdqs_codfw_brief_outage what fired was the following:
[19:02:40] <ryankemper>	 `2022-12-12 20:13:18 <+jinxer-wm> (ProbeDown) firing: Service wdqs-ssl:443 has failed probes (http_wdqs-ssl_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#wdqs-ssl:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown`
[19:03:06] <ryankemper>	 are you all aware of there being a way to suppress that alert on a per-service basis (ie just for wdqs)? the metric/probe itself seems like something we'd want to keep, but just be able to disable the part that actually emits the page
[19:05:22] <rzl>	 ryankemper: those per-service alerts are defined in hieradata/common/service.yaml -- I think you can just add `page: false` under wdqs-ssl
[19:06:02] <herron>	 yes was just typing that out as well, what rzl said
[19:06:34] <rzl>	 (also hi! a lot of us have `#p-age` as an IRC notification with varying levels of insistent-ness, consider nerfing it with an extra character like I did there, when you don't mean to notify everybody <3)
[19:09:22] <ryankemper>	 rzl: ah yes, duly noted
[19:10:13] <ryankemper>	 thanks all