[14:34:18] <brouberol>	 cdanis: I'm wondering how I can test https://phabricator.wikimedia.org/T404162#11172921 in CI, to make sure what I'm doing works as expected. I submitted https://gerrit.wikimedia.org/r/c/labs/private/+/1187463, but the next step looks like it should be reflecting that change in the private puppet repo. Given that this looks like a change limited to
[14:34:18] <brouberol>	 the private git repo, I'm not sure how to check the change via PCC
[14:35:20] <brouberol>	 _or_, we could inject these secrets from within puppet/modules/profile/manifests/kubernetes/deployment_server/helmfile.pp, but that seems very out-ot-place
[14:56:18] <cdanis>	 brouberol: so, PCC uses the labs-private repo -- it doesn't have access to the real secrets
[14:56:40] <cdanis>	 merging the labs-private change (and maybe also a puppet-merge after you do that) should be enough to get PCC to see the new secrets
[14:56:46] <cdanis>	 and to test that patch
[15:06:59] <brouberol>	 that was my understanding as well. However, the change I'm planning to make is solely located in the git private repo, and PCC allows us to see the change on a puppet (as in https://gerrit.wikimedia.org/g/operations/puppet) patch, doesn't it? 
[15:09:56] <cdanis>	 ohhhhh
[15:10:01] <cdanis>	 right, since you have to... mm
[15:11:38] <cdanis>	 you could wholesale replace all references to the old $passwords:: object with the secret :)
[15:11:44] <cdanis>	 that would test in PCC just fine 😅
[15:12:25] <cdanis>	 you could also just perform the change on private puppet, and then run puppet manually on several affected hosts, to see quickly for failures
[15:16:06] <cdanis>	 I don't have a strong preference, wdyt brouberol?
[15:16:53] <cdanis>	 (there is not a real way to test this within CI, aside from no longer using the passwords class at all)
[15:19:04] <brouberol>	 I think I'll do the change in 3 steps: 1) add the password to the file in the real private git repo 2) add a reference to that password in the hieradata/role/common/deployment_server/kubernetes.yaml file, which won't break anything even if I get it wrong 3) once we validate that 2) works, we then disable puppet on all db hosts, update the "real"
[15:19:04] <brouberol>	 wikiadmin_password puppet variable definition with the `secret(...)`, run puppet on a depooled db host, make sure everything works fine, and re-enable puppet everywhere
[15:20:45] <cdanis>	 ack, that sounds good
[15:33:05] <brouberol>	 thanks again for your help