[07:39:26] * arturo online [07:39:30] o/ [09:04:43] taavi: https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/38 [09:06:05] approved [09:06:21] thanks [10:02:22] * arturo brb [10:44:20] so since it seems like the OVS migration will force us to come up with a new set of flavors, I'm thinking of writing a script to manage the flavors based on a config file instead of creating them by hand [10:49:39] taavi: +1, that would be awesome! [10:50:38] * dhinus wonders if the opentofu provider can manage flavors? [10:51:11] good question [10:54:00] looks like yes https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs/resources/compute_flavor_v2 [10:54:53] so if we can figure out a good way to run tofu as the cluster admin, that might be a good option too [10:57:12] nice! I think this is one of the usecases where tofu is really a good fit, if authn/authz is not too hard [11:15:05] +1 to more things asCode [11:27:06] I'm looking for reviews of this https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/37 [13:43:55] taavi: I think the reason I did an '--add safe.directory' a few times is that I can't e.g. "git branch" in /srv/git/operations/puppet as root. [13:44:26] everything in that directory should be done as gitpuppet, not as root [13:44:27] That's a problem for the sync script but it also seems like it mostly breaks any user workflow that involves applying local patches... [13:44:40] Hm... [13:45:52] This is a puppet7 change? [13:46:13] Since I'm sure it wasn't the case that you had to be gitpuppet to mess with local puppet back when it was in /var/lib/git [13:46:55] yes [13:47:53] Hm... even if /I/ know not to add safe, our users are surely going to do that since git tells them to [13:48:08] I guess we could have puppet chown the dir on every run [13:58:38] I miss the context, but in production::puppetmaster::production we puppetise a few git::systemconfig overrides, maybe the same can be used here? [14:00:30] moritzm: the problem is that on puppet 7 servers the puppet.git and labs/private.git clones are owned and interacted by gitpuppet:gitpuppet, but people keep manually touching them as root which breaks the sync script [14:01:10] so adding a safe.directory for them is the exact opposite of what we want. however if there's a git config to suppress the 'do this to add a safe.directory setting' message I'd be very interested in that :-) [14:02:15] ah, right [14:03:21] one thing that comes to my mind is to add a systemd timer which chowns the checkout to gitpuppet:gitpuppet regularly [14:03:40] or have Puppet manage the permissions, but that's a certain catch 22 :-) [14:07:27] yeah, forcing the ownership may be the only good way forward [14:07:45] that or convince puppet7 to not care about ownership, or abandon the 'gitpuppet' service user and have everything owned by root [14:14:52] taavi: what do you think about the latter? [14:16:56] i would prefer to avoid that if possible [14:18:17] I suspect the initial motivation for that user was to avoid running the sync cron as root but now we do that anyway [14:19:05] like, for this issue to happen someone needs to get an error message, and then run a 'if you know what you are doing run this to fix it' command without properly looking at how the setup is done nor reading the documentation for the service, or documenting/trying to automate that change, and that just should not be happening anyway [14:19:33] you are an optimist :) [14:19:56] yes :-) [14:21:08] I fear that this ticket is an example of just that https://phabricator.wikimedia.org/T360470 [14:21:58] I guess jelto at least set upon the right fix, 'Make sure to use the correct user in the future, like sudo -u gitpuppet git status.' [14:25:26] in my opinion, the permissions settings have been an issue for a long time [14:26:13] example: T152059 [14:26:15] T152059: role::puppetmaster::standalone clones Git repositories as gitpuppet, git-sync-upstream overwrites them as root - https://phabricator.wikimedia.org/T152059 [14:29:03] and a bunch of other tickets have been opened over the years related to the permission settings [14:29:32] T325280, T184259, T152060 [14:29:34] T325280: cloud puppetmasters fail to run git-sync-upstream - https://phabricator.wikimedia.org/T325280 [14:29:34] T184259: labspuppetmaster1001: have consistency in owner of git repos - https://phabricator.wikimedia.org/T184259 [14:29:35] T152060: Inconsistent groups for Git repositories with role::puppetmaster::standalone - https://phabricator.wikimedia.org/T152060 [14:30:52] arturo: do you have a fix in mind? I'm leaning towards puppet forcing ownership [14:31:16] I don't have a fix in mind, because also I don't fully understand the setup or its requirements [14:31:43] I _can_ imagine why running all this as non-root is desirable [15:43:57] Wow, the hackathon was HUGE [15:47:22] 2019 was 221 people: https://phabricator.wikimedia.org/T223973#5199915 [15:47:35] It was a good event though for sure. [15:47:43] how many did we have this year? [15:48:29] I don't have rewind on the live stream of the dept meeting. It was a bit under 200? [15:50:02] 150 in person I think [15:51:40] "150+ in person participants" according to https://docs.google.com/presentation/d/1J944tuHi1GAs1p7y7IRyoc6WjKsmoWjorFEsbJaftSw/edit#slide=id.g2da79e0e703_0_205 [16:06:29] * arturo offline [17:09:22] taavi and arturo, for your future consideration, I'm back to thinking we should just eliminate the gitpuppet user. I was going to wait and see if there were more problems and I already found a few. T364492 [17:09:23] T364492: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492 [22:25:11] I gave T364509 a +1 when somebody has time to run the cookbook [22:25:13] T364509: Request creation of Huma VPS project - https://phabricator.wikimedia.org/T364509 [22:26:32] Ladsgroup tried gently to get me to give him more resources without asking for a new project during the hackathon, but I was able to resist the temptation. ;)