[15:10:32] <cdanis>	 did the grafana upgrade break old goto links?  I tried to follow https://grafana.wikimedia.org/goto/QKFq4aD4k?orgId=1 from https://phabricator.wikimedia.org/T264021#8385758 but now it is a 404
[15:44:51] <cwhite>	 cdanis: I can't find that short url in two old database backups (2023-02-03 or 2023-02-21)
[15:45:02] <cdanis>	 hm, interesting
[15:45:07] <cdanis>	 maybe it expired a lot earlier
[15:45:09] <cdanis>	 it's from last Nov
[15:45:17] <cdanis>	 thanks for looking :)O
[15:47:36] <godog>	 do you know what was the underlying dashboard? I tried e.g. https://grafana.wikimedia.org/goto/75EQjmO4z?orgId=1 from https://phabricator.wikimedia.org/T303534 also from last Nov and it works
[15:48:07] <godog>	 maybe the dashboard has been deleted in the meantime (or renamed, but I'd hope that doesn't break things)
[16:20:06] <cdanis>	 it was likely one of the traffic dashboards, but I don't know for sure
[21:58:37] <mutante>	 I am using blackbox::http monitoring and have configured "task creation" as a notification method. Even though I copied the code from an existing one I feel like it's not working. But I don't know for sure. So far I would only know of one way to test it.. to actually take down my service. Is there another way though? I just want to confirm I actually get the phab tickets if it triggers. I 
[21:58:43] <mutante>	 do know that I can lookup "probe_success" in Thanos.
[22:14:19] <cwhite>	 mutante: As you say, taking it down is probably easiest.  Another way is to create and send the http call Prometheus would send to Alertmanager to create the task.
[22:32:25] <mutante>	 cwhite: hmm. that second option sounds nice actually. but that would need some research to find the exact call ?
[22:33:13] <mutante>	 well.. here is what I can do.. find a service that is active-passive, add monitoring to check the same virtual host on BOTH backends.. not just the active one.. then take the passive one down!
[22:33:45] <mutante>	 I wanted to check on all backends anyways. rather than the current practive where it's mostly like "if $active_server..."
[22:34:22] <mutante>	 my alerting will be only tasks, no IRC, no email.. and we would like it that way for a couple cases
[22:44:21] <cwhite>	 Sounds reasonable.  FWIW, the bot appears active even today: https://phabricator.wikimedia.org/p/phaultfinder/