[06:52:24] morning [07:52:17] morning [07:53:10] what might be up with the pipeline status here: https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/70? [07:57:51] blancadesal: it did not trigger as ahmon is not part of the toolforge project, you have to manually runit [07:57:56] (unless we change how that works) [07:59:01] from https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/pipelines/new and select the branch [07:59:54] dcaro: thx. the gitlab ui is a bit confusing in that sense [08:00:21] yep, it kind of just fails if you try from the MR to run the pipeline [08:17:49] o/ [08:20:09] is this a valid reason for a floating ip? T365641 [08:20:10] T365641: Floating IP for Wikispore - https://phabricator.wikimedia.org/T365641 [08:33:52] there's no official support yet for vanity domains I think T342398 [08:33:53] T342398: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398 [08:35:33] so I think that the floating IP is the only way to set it up right now [08:36:50] though we don't allow https/http on floating ip addresses ("Only request a Floating IP address if you need to expose non-HTTP/HTTPS endpoints. Floating IP addresses are not automatically available to projects, the default quota is 0. ", https://wikitech.wikimedia.org/wiki/Help:Manage_floating_IP_addresses_assigned_to_Cloud_VPS_instances#Request_a_floating_IP_address) [08:38:32] blancadesal: I think you can add it to the team meeting to discuss next thursday [08:39:42] with the current setup, support for custom domains _does_ require http/s on a floating IP if I'm reading T342398 right? [08:39:43] T342398: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398 [08:40:22] does any project use a custom domain currently? I though our policy was against this [08:40:33] dcaro: ok, adding to etherpad [08:45:37] 👍 [08:52:42] hmm, as they are asking for +1TB storage, per the guidelines this request should also be discussed at the team meeting: T365733 [08:52:43] T365733: Increase storage for parsoid visualdiff testing - https://phabricator.wikimedia.org/T365733 [08:52:55] I kind of remember t.aavi working on extending nova-proxy support for additional domains, but I could be wrong [08:53:23] but considering that this is the parsoid team and their specific use case, could we not expedite it quicker? [08:56:31] blancadesal: I think we have more than 50TB free at the moment, I think we can +1 right away [08:57:13] +1'd [08:57:31] xd, I was +1 ing too [08:57:36] thanks! [09:54:49] dcaro: if you would like to review the maintain-kubeusers refactor, I think now/today is the right time! [09:55:12] I would like to do a deployment in toolsbeta to expose the code to more real data (compared to lima-kilo) [09:55:25] arturo: I'll give it a look [09:55:35] thanks [09:55:53] this is the MR I'm interested in https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 [09:56:20] 28k lines feels scary xd (probably mostly vcr) https://usercontent.irccloud-cdn.com/file/LKcNQVKC/image.png [09:56:38] yes, lots of changes to cassetes (basically, drop old ones, generate new ones) [09:57:05] but the python code changes are also big, I'm afraid [10:28:27] * dcaro lunch [13:02:23] heads up I'm gonna merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1029158 which sets a 10-minute timeout for Redis connections [13:02:38] I will keep an eye on issues, but ping me if you see anything [13:05:56] 👍 [13:14:17] tools-puppetserver-01 is failing with the "dubious ownership" error [13:14:59] (the puppet-git-sync-upstream unit is failing) [13:15:50] I guess it's T364492 [13:15:51] T364492: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492 [13:24:05] I tried "chown -R gitpuppet:gitpuppet /srv/git/operations/puppet/" but that did not fix it [14:00:10] dhinus: I can look if it's still broken [14:00:48] I think I fixed it by applying manually https://gerrit.wikimedia.org/r/c/operations/puppet/+/1030962 [14:01:10] which was merged but never made it to tools-puppetserver-01 (and potentially to other puppetservers) [14:01:42] seems possible, has it been broken that long? [14:04:12] last puppet commit on that server was from 20 days ago [14:06:09] 10 days actually [14:08:01] maybe we should add an alert to check if puppet is running /but/ the last commit is too old [14:08:24] similar to the alert we have for puppet not running since... [14:11:29] now onto a new problem: Puppet is up to date, but my puppet change did not modify the Redis config file :) [14:19:42] I don't think tools-puppetserver-01 is its own puppet server [14:21:42] looking at /etc/puppet/puppet.conf, I see server = tools-puppetserver-01.tools.eqiad1.wikimedia.cloud [14:21:59] so my understanding is that it is? [14:26:11] hm, I guess so [14:26:23] so there's also a deployment step, even if the git repo is correct [14:26:29] that is usually called automatically, but not always [14:26:59] /usr/local/bin/puppetserver-deploy-code [14:27:03] if you haven't tried that already [14:32:17] yep that's the part that was breaking, and your patch above fixes it, but it was never deployed to that server [14:32:28] so I manually applied your patch, then ran manually puppetserver-deploy-code [14:33:27] hmmmm I definitely remember manually applying it, I hope it doesn't somehow reverse itself [14:34:04] I am entirely unhappy with the way puppet is managed after the move to puppet 7, feel free to rething/redesign. I was trying to reuse the upstream process + add some extra bits but maybe we should just scrap the upstream entirely. [14:34:09] hmm I think it does reverse itself if you do "run-puppet-agent", so I had to manually call puppetserver-deploy-code [14:34:22] before doing run-puppet-agent, but hopefully it will not reverse again [14:36:30] I don't have specific ideas right now on how to redesign the setup, but I subscribed to T364492, we can discuss there [14:36:30] T364492: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492 [15:59:45] does this ring a bell for anyone? "Repository 'http://mirrors.wikimedia.org/osbpo bullseye-victoria-backports-nochange InRelease' changed its 'Origin' value from 'Infomaniak' to 'osbpo'" [16:10:47] * arturo offline [17:30:23] nope [17:30:26] * dcaro off