[10:31:24] <dcausse>	 ls
[10:31:29] <dcausse>	 oops
[10:32:47] <dcausse>	 o/ we're experimenting with a tool to kill the jvm if it starts to misbehave but before activating it for real it has a mode where it can touch a file to indicate that it would have killed the JVM
[10:33:36] <dcausse>	 what would be the best way to do emit an alert if a file is present on the filesystem
[10:33:58] <dcausse>	 (and possibly delete it once the alert is fired)
[10:36:07] <dcausse>	 even a datapoint in prometheus would be good (it's just meant for verification/tuning purposes)
[15:23:09] <herron>	 hey dcausse, could the tool emit a syslog message when this action happens?  yeah a datapoint in prometheus would be good too.  we could use that to fire an alert, along with some history of the details of the action taken via the log
[15:56:00] <dcausse>	 herron: sadly the tool is pretty rudimentary it can just touch a file
[15:56:44] <dcausse>	 I suppose we could write a small script triggered by cron that scan for this particular file
[15:57:14] <herron>	 gotcha, ok
[18:08:21] <herron>	 hey rzl, re: https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/768108/3#message-ee7b70391456aa89437a4cc28f77c5a052ddbd7c is something like this more what you have in mind? https://usercontent.irccloud-cdn.com/file/C2zw0oFB/Screen%20Shot%202022-03-11%20at%201.06.03%20PM.png
[18:15:56] <rzl>	 yes! that looks great
[18:16:12] <herron>	 awesome, I'll update the patch this way
[18:17:22] <rzl>	 amazing, thank you!