[07:03:00] <elukey>	 hello on-callers
[07:03:21] <elukey>	 I am going to deploy the new changeprop docker image to eqiad (the codfw one has been running fine)
[07:10:31] <elukey>	 {{done}}
[07:11:52] <akosiaris>	 nice
[07:13:31] <elukey>	 also, me and Alex are testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/1133740 - we are limiting the uploads from the deployment node (from scap basically) to the docker registry
[07:13:53] <elukey>	 hopefully this should mitigate the issue described in https://phabricator.wikimedia.org/T390251#10706432
[07:14:13] <elukey>	 if you see issues reported during deployments and I am not around, the patch is easy to revert
[07:14:22] <akosiaris>	 elukey: since you are our local envoy expert, I got  https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1133745 this for you
[07:15:07] <akosiaris>	 this registry corrupt blob thing is very bizarre. The concurrency thing will hopefully solve this, but it's a real Heisenbug
[07:16:21] <elukey>	 I am the envoy expert? Really? 
[07:16:45] <elukey>	 I'll check but I am not convinced
[07:18:03] <akosiaris>	 you 've been awarded the barnstar, nothing to check
[07:18:25] <akosiaris>	 btw, this is already live in eqiad
[07:18:42] <akosiaris>	 but with a filterChain just for the mw-wikifunctions-ingress.discovery.wmnet SNI
[07:19:07] <akosiaris>	 I 've found how to dump the envoy configuration from ingressgateway and verified it's applied correctly
[07:19:13] <akosiaris>	 and works
[07:19:40] <akosiaris>	 the difference between what's applied in eqiad is an sni: mw-wikifunctions-ingress.discovery.wmnet line in the filterChain 
[07:20:25] <akosiaris>	 oh damn, it's malformatted, 
[07:20:27] <akosiaris>	 I just noticed
[07:20:31] <akosiaris>	 oof
[07:20:53] <akosiaris>	 damn yaml
[07:21:00] <elukey>	 what do you mean it is already live in eqiad? (just to understand, it is morning :D)
[07:22:28] <akosiaris>	 deploy1003:$ sudo -i ; kube_env admin eqiad; kubectl -n istio-system get EnvoyFilter servername-tweaks -o yaml
[07:23:02] <akosiaris>	 this won't work as is ^, you 'll need the sudo -i on a separate line but the rest will work
[07:23:05] <akosiaris>	 also
[07:23:31] <akosiaris>	 let me document how I dump the envoy configuration in a phaste
[07:23:37] <akosiaris>	 it's probably going to be useful in the future
[07:23:44] <akosiaris>	 ah no, wikitech better
[07:24:22] <elukey>	 yep yep, there is a page about the istio ingress with come useful commands
[07:24:25] <elukey>	 could be a good place
[07:25:31] <akosiaris>	 ahahah it already exists https://wikitech.wikimedia.org/wiki/Kubernetes/Ingress#Dump_complete_envoy_config
[07:25:53] <akosiaris>	 Suits me for not RTFMing
[07:26:00] <akosiaris>	 on the plus side, I 'll probably never forget now
[07:26:41] * akosiaris fixing the damn tabs in the gerrit patch
[07:27:22] <elukey>	 (going to relocate, back in a bit)
[08:20:28] <elukey>	 since all metrics are good for changeprop, I am going to deploy changeprop-jobqueue-codfw as well
[08:26:51] <elukey>	 done
[10:03:07] <akosiaris>	 elukey: well, the previous patch did pass CI, but ... https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1133855/
[10:03:13] <akosiaris>	 wanna know the best part?
[10:03:23] <akosiaris>	 it applied to staging-{eqiad,codfw}
[10:04:41] <akosiaris>	 ignoring apparently that entire stanza
[10:04:57] <akosiaris>	 but in the other clusters the validator complained
[10:05:06] <akosiaris>	 is that a regression? I wonder
[10:18:55] <elukey>	 ahhhh snappppp
[10:18:59] <elukey>	 I didn't really see it :(
[10:19:31] <elukey>	 yaml engineering
[11:57:15] <sukhe>	 4
[11:57:40] <taavi>	 5
[12:04:48] <akosiaris>	 6?
[13:30:50] <_joe_>	 vgutierrez: definitely a libera.chat issue yes