[09:04:41] <godog>	 herron: looks like thanos-rule on titan2001 never got reloaded
[09:10:17] <elukey>	 sigh
[09:10:52] <elukey>	 Maybe we could add a notify in puppet so when pyrra reloads it will signal thanos-rule as well?
[09:23:41] <elukey>	 reloaded thanos-rule on 2001
[09:26:32] <godog>	 yes that's what needs to happen, for some reason I thought that was the case already
[09:31:37] <elukey>	 or we could add a dependency in the Pyrra filesystem's systemd unit
[09:33:36] <elukey>	 mmm no I had in mind PartOf/Requires but it would need to be on the thanos side
[09:55:56] <godog>	 I'll file a tracking task
[09:57:45] <godog>	 https://phabricator.wikimedia.org/T353691
[10:01:01] <elukey>	 <3
[11:27:26] <elukey>	 after the reload I still see the alerts firing :(
[11:27:41] <elukey>	 <jinxer-wm> (ErrorBudgetBurn) firing: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[12:00:32] <godog>	 ack, I'll take a look
[12:03:16] <godog>	 looks like /etc/pyrra on titan2001 is not getting updated? e.g. /etc/pyrra/config/varnish.yaml is on titan1001 but isn't present on titan2001, equally /etc/pyrra/output-rules/ on titan2001 is not updated and still contains liftwing rules
[14:40:05] <herron>	 doh! thanks
[14:42:09] <inflatador>	 godog heads up that I'm merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/983948 
[14:42:51] <herron>	 yes the reload handling could be improved, thing is the operator will detect changed files and create new output rules without being reloaded itself.  I was thinking something like PathModified on output-rules to issue a reload, until upsteram adds a custom reload command/url feature 
[14:45:44] <godog>	 inflatador: +1'd
[14:53:25] <inflatador>	 godog excellent..merged. Working on a patch to use the new param ;)
[15:36:35] <jayme>	 since when did the CI pipeline for alerts get so useless? https://integration.wikimedia.org/ci/job/trigger-alerts-pipeline-test/1483/console :D
[15:37:29] <volans>	 you have to follow the links
[15:37:33] <volans>	 to https://integration.wikimedia.org/ci/job/alerts-pipeline-test/1485/console
[15:37:41] <volans>	 not sure why :D
[15:38:46] <jayme>	 that's very unintuitive...I clicked all tree links (two are the same) and none of those led to the output
[15:39:04] <jayme>	 one has to click *again* to get the output of the job
[15:39:34] <jayme>	 IIRC that used to work "properly" in the past
[15:41:41] <volans>	 I'm not saying it wasn't :D just how to get there from it
[15:41:50] <volans>	 probably some parent/child job relation?
[15:42:02] <jayme>	 yeah, sure - did not mean to sound harsh. sorry
[15:42:15] <volans>	 you weren't ;)
[15:42:47] <volans>	 it has 2 jobs from operation-misc.yaml:
[15:42:47] <volans>	  - 'trigger-{name}-pipeline-{pipeline}'
[15:42:53] <volans>	 - '{name}-pipeline-{pipeline}'
[15:43:22] <volans>	 possibly related