[09:21:05] <_joe_>	 cwhite: I was noticing that we don't have the ecs access log format you introduced in https://gerrit.wikimedia.org/r/c/operations/puppet/+/668231 in the kubernetes configurations for mediawiki
[09:21:54] <_joe_>	 I was wondering if we should add it and switch to use those instead of the "wmfjson" format 
[09:22:05] <_joe_>	 before we start sending actual traffic to k8s :)
[09:26:48] <btullis>	 godog: Sorry to trouble you, but it looks to me like these changes (https://gerrit.wikimedia.org/r/c/operations/alerts/+/731919 and https://gerrit.wikimedia.org/r/c/operations/alerts/+/732300) haven't been deployed. Or if they have then I'm missing something.
[09:28:08] <btullis>	 I've tried an SSH tunnel to both of the prometheus ops servers in eqiad. e.g. `ssh prometheus1004.eqiad.wmnet -L9900:localhost:9900 -N` and I'm browsing http://localhost:9900/ops/classic/rules
[09:30:32] <btullis>	 I've also logged in via SSH to prometheus1003 and 1004  and checked the contents of `/srv/alerts/` but I can't see any reference to the new `team-data-engineering.yaml`file. Puppet runs are succeeding. What am I missing?
[09:36:49] <godog>	 btullis: sure no problem, I'll take a look, I'm not excluding a bug in the deploy script
[09:37:23] <btullis>	 Thanks. No hurry.
[09:38:36] <godog>	 btullis: I think it is .yml vs .yaml
[09:38:47] <godog>	 definitely need to catch that in CI
[09:41:11] <btullis>	 Ah, great. I'll change that filename now and send a CR your way. I'm glad I went to the effort of checking that it was present though (and finding that it wasn't).
[09:42:07] <godog>	 indeed, thank you for that 
[09:48:32] <godog>	 btullis: sth like         if yml_files:
[09:48:33] <godog>	         if set(p.glob("**/*.yml"))
[09:48:34] <godog>	 no
[09:48:39] <godog>	 https://gerrit.wikimedia.org/r/c/operations/alerts/+/732621
[10:04:40] <btullis>	 Ah, now my test itself is failing :-) Will try to fix that.
[10:20:47] <godog>	 ack! I'll merge my change
[15:10:42] <cwhite>	 _joe_: Good catch!  Looks like the configuration lives in the production-images repo?  I'm happy to to together a CR.
[15:11:06] <cwhite>	 *to put together
[15:13:25] <cwhite>	 It doesn't seem easy to keep the two copies in sync.  Perhaps we can add a note in the puppet repo asking folks to update the one in production-images too.