[08:29:27] <moritzm>	 the grafana role has been switched to Puppet 7 now
[15:40:09] <denisse>	 @moritzm: Thank you! 
[17:06:25] <jinxer-wm>	 (SystemdUnitFailed) firing: rsync-loki-data.service on grafana2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:06:45] <denisse>	 ^ looking
[17:16:25] <jinxer-wm>	 (SystemdUnitFailed) resolved: rsync-loki-data.service on grafana2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:21:02] <lmata>	 Thanks
[19:01:21] <mutante>	 I have one Icinga check that uses "check_lastmod.py". A script I once wrote myself to check if a website has been updated recently enough. Just checks the last modified header of any website. How could I replace this to move it out of Icinga? Can i check any header with blackbox::http ?
[19:05:38] <denisse>	 I think that Blackbox Exporter does not directly expose a way to check or alert based on the 'Last-Modified' HTTP header of a website.
[19:07:08] <mutante>	 thanks denisse. it seems like there is no replacement for it
[19:07:19] <denisse>	 But you can write a custom exporter that acts similarly to your "check_lastmod.py" script.
[19:08:39] <denisse>	 Your custom exporter can send requests to your target website(s), check the 'Last-Modified' header, and then expose that information for Prometheus to scrape.
[19:10:00] <mutante>	 hmm. yea, I don't think I want to build an entire package and all that for one check
[19:12:38] <moritzm>	 mutante: check prometheus::node_textfile, it's a define I once added to simply setting up such a check without the need for a package
[19:12:56] <mutante>	 moritzm: ah! thank you, will check
[19:13:25] <denisse>	 :o good idea!
[19:55:51] <cwhite>	 mutante: is this http endpoint currently probed with blackbox exporter?
[20:02:49] <cwhite>	 if so, there's a chance that the last modified timestamp is available in Prometheus
[20:08:08] <mutante>	 cwhite: there is "prometheus::blackbox::check::http { 'en.planet.wikimedia.org':", yes.  but there is also still  @monitoring::host { 'en.planet.wikimedia.org': and a "check_ssl_http_letsencrypt" on it. 
[20:14:41] <cwhite>	 mutante: we have the last modified header in Prometheus
[20:14:46] <cwhite>	 `probe_http_last_modified_timestamp_seconds{module="http_en_planet_wikimedia_org_ip4"}`
[20:17:29] <mutante>	 cwhite: cool, thank you!
[20:18:34] <mutante>	 I dont know how to use that but it sounds like I just need to rtfm then or something
[20:25:54] <cwhite>	 mutante: The goal is to alert if the delta between last-modified and now exceeds a threshold?
[20:30:40] <mutante>	 cwhite: yes, it was set to warn afer 24 hours and crit after 48 hours
[20:35:29] <cwhite>	 mutante: you could try to warn on something like `probe_http_last_modified_timestamp_seconds{module="http_en_planet_wikimedia_org_ip4"} < time() - 86400`
[20:35:48] <cwhite>	 then crit on 172800
[20:36:26] <cwhite>	 see also: https://prometheus.io/docs/prometheus/latest/querying/functions/#time
[20:40:57] <mutante>	 cwhite: thank you. so that would mean  https://wikitech.wikimedia.org/wiki/Alertmanager#Create_alerts  ?
[20:41:57] <cwhite>	 Yep!  That should do the trick :)
[20:42:32] <mutante>	 alright, thanks again