[05:22:31] I'm glad we have additional security controls in our k8s cluster, to reduce attack surface in cases like that ^^^ [07:16:08] things can still happen though, some extra action is needed [07:37:14] we could stop the webservice while they fix the code [08:04:43] taavi: thanks for the fix yesterday to puppet, looking into it now [10:09:42] hmm, cloudwebs and such are not using the latest openstack packages :/ [10:09:59] (that's why the patches failed yesterday on those) [10:10:47] why do cloudwebs need the openstack packages in the first place? [10:19:08] no idea, looking [10:22:51] it comes from openstack horizon docker_deploy [10:25:04] not sure if it needs it for anything [10:25:55] maybe it's from before it was dockerized? [10:27:29] added a comment for later: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1035148 [10:27:41] andrewbogott: ^ maybe you know from the top of your head [11:07:58] today, maintain-kubeusers writes in each tool account configmap the creation date (in addition to the cert expiration date). Is that information useful for anything in particular place? I'm thinking on dropping it as part of my refactor [11:08:54] something like [11:08:55] status: 'user created: 2023-11-02T19:23:17.254569' [11:09:02] but I can't think what is the use of that information [11:09:24] again, this is different from the expiration date of the cert, which is stored with something like [11:09:30] expires: 2024-11-01T19:18:15 [11:11:04] dhinus: did you have any thoughts on how/where to run opentofu for cloud vps infra? [11:11:21] taavi: not really, too many other things in my head :) [11:11:51] arturo: I'm guessing for debugging, to see that it's working [11:12:10] though that might be in the metadata of the configmap no? [11:12:24] yeah, a bit redundant [11:12:49] dhinus: ok, I will write a task with my thoughts then I think [11:13:07] taavi: sounds good, thanks [11:28:53] * dcaro lunch [12:07:56] dcaro: I don't remember off the top but likely they're leftovers like you said. [12:11:44] dhinus: filed T365696, and now I'm wondering whether that needs to be a proper decision request [12:11:45] T365696: Investigate how to run OpenTofu to manage Cloud VPS admin-only resources - https://phabricator.wikimedia.org/T365696 [14:01:09] taavi: thanks, I added some thoughts to the task. it could also be a decision request, up to you [15:20:25] The task about rotating credentials (feel free to add/refine) T365724 [15:20:26] T365724: [infra] Allow users to self-rotate the all the credentials - https://phabricator.wikimedia.org/T365724 [15:39:21] bd808: sorry, I just sent a patch on top of yours xd [15:43:46] dcaro: no worries. eoghan is trying to help too. I think the real secret maybe needs to move to another hiera file too. The actual cloudweb runs are failing now. [15:44:15] dcaro: thanks for the ticket, I made a couple of changes [15:44:18] eoghan wanted to reuse the existing key instead of jsut pasting the value over to a new one. [15:45:33] Hello [15:45:36] bd808: yep, the error now on pcc is different though, it complains about the parameter not being passed, but not about the lookup... [15:45:41] \o hi! [15:46:05] * bd808 invited eoghan so y'all could coordinate a fix [15:46:25] Current status is that pcc works but the actual run on cloudweb hosts doesn't, right? [15:46:30] yes [15:46:43] wait, there's a missing parameter not passed around [15:46:50] class { '::openstack::wikitech::web': [15:46:52] to that call [15:47:18] (pcc is still not working for me at least: https://puppet-compiler.wmflabs.org/output/1035148/2613/cloudweb1003.wikimedia.org/prod.cloudweb1003.wikimedia.org.err) [15:47:37] lol. I only pulled it into the class params :facepalm: [15:47:37] right [15:47:54] Ah yes [15:48:35] anyhow, as I already said in -sre-collab, the real-private hiera key is also defined in the gitlab role hiera which means it's not available for the cloudweb class, and needs to be moved to the profile hiera file [15:49:42] ^ good point [15:49:48] Got it. I need to split in a few minutes to pick up the kids. Maybe the best thing for right now is to back out the change and I can look at moving it later this evening. [15:49:56] Unless there's a faster fix right now [15:50:17] I'm about to push the fix to the ops/puppet bit. the secret will still need sorting [15:50:28] I'll sort out the secret location [15:50:32] 👍 [15:50:38] Cool, thanks taavi. [15:50:42] Sorry for the hassle [15:51:03] I'll sit back and relax :) [15:51:05] The secret is used by another tool though, so it'll need to be updated or duplicated [15:51:10] (Please not duplicated) [15:51:30] maybe we can use lookups inside hiera itself? [15:52:02] taavi: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1035478 [15:53:43] thanks, looking [15:54:55] I'm running PCC for it too (which I should have done for the prior patch as well) [15:55:13] so am I :P [15:56:24] I gotta go, back in ~90 minutes. Thanks for sorting that, taavi. [15:56:42] PCC seems not pleased still [15:57:04] bd808: how so? [15:57:22] https://puppet-compiler.wmflabs.org/output/1034532/2618/cloudweb1003.wikimedia.org/change.cloudweb1003.wikimedia.org.err [15:57:33] was that transient? [15:57:49] that's against the broken patch, not the one fixing things [15:58:03] * dcaro does that quite often [15:58:03] i merged the fix and ran puppet on cloudweb1003, which looks ok to me [15:58:19] and gitlab hosts are also running puppet without change after I moved the hiera key [15:58:28] cool. I'm just slow and confused this morning then ;) [15:58:31] I see pcc fixed on my other patch too \o/ [15:58:50] the pcc script in the puppet repo takes 'latest' as a patch number too :-) [15:59:10] uooooo! that's great! [15:59:12] I see the value I needed to see in /etc/mediawiki/WikitechPrivateSettings.php so things are good on my end [16:00:11] dcaro: quick review for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1035479? [16:00:59] +1d [16:01:11] ty [16:04:35] * arturo offline [16:14:13] * dcaro off