[12:42:15] godog: question regarding https://gerrit.wikimedia.org/r/q/897833 . Did this prevent configuration of newly added blackbox checks? [12:42:15] Looking at https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes%2Fcustom&var-module=All&orgId=1 it seems that some checks (like releases_jenkins and rpki) were added and failed right after this change was merged. [12:56:43] jelto: yeah that's righ [12:56:45] right [12:56:54] and led me to discover/debug https://phabricator.wikimedia.org/T331867 [13:11:57] ok thanks for the feedback :) I guess the releases/jenkins blackbox probes also need some more debugging. I re-opened T327975 [13:14:02] *nod* [13:14:56] not sure if relevant to your debugging but new probes wouldn't be configured (in codfw) since last thurs [21:00:20] How do you absent a prometheus::blackbox::check::http? It doesn't have an ensure parameter. Does it just go away if it's not in code anymore? [21:01:27] for those cases where you need an alert to work on the "active server", and only there.. and then flip what the active server is [21:02:04] with other things like rsync::quickdatacopy we need to properly absent them or there will be follow-up issues [21:16:26] mutante: prometheus::blackbox::check::http uses exported resources to declare the checks within the manifest of the monitored service and render configs on the prometheus hosts. It should be as simple as removing the declaration from the target service manifest. [21:16:36] Are you seeing some unexpected behavior? [21:17:38] cwhite: thanks! ok, I will know for sure in a few minutes [21:31:25] cwhite: ACK, I made a code change and it was removed from /srv/prometheus/ops/targets/probes-custom_puppet-http.yaml after running puppet on monitored host and then prometheus. thanks [21:31:42] \o/ [21:34:23] well, except that doesnt mean it stops showing up in logstash .. at least until now [21:34:39] checks back in a couple minutes .. I guess [21:35:20] ah, more than one prometheus server is making the requests in codfw [21:35:29] 2006 but also 2005 [21:35:40] runs puppet on 2005 too [21:41:34] good now:) cya