[06:18:43] greetings [07:11:33] morning. looking for a review on this istio metric tuning patch: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1206 [07:14:02] LGTM [07:14:11] also TIL kind: Telemetry, nice [10:57:52] * dcaro lunich [12:22:17] I have found out WMCS has a backup system for instances and I found it has some rules to exclude some instances. One of them was matching a hostname that would I think caused it to backup a 320G attached volume ;) [12:22:25] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1268540 cloudbackup: do not back integration-castor* instances [12:22:36] I am adding andrewbogott as a reviewer (based on git log on that file) [12:31:04] hashar: thanks! LGTM [12:31:35] hopefully that will save some tiny bit of disk space :-] [12:43:04] oh, I have to merge it right? [12:43:40] btw. when anyone has some time https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/86 unblocks apt buildpack on the new ubuntu24-based buildpack environment [12:44:17] dcaro: up to you, I don't have puppet rights :b Maybe andrew needs to +1 it as well [12:44:48] it is up to you, I have absolutely no idea how that backup system works or what is the impact of thatchange [12:45:11] beside I renamed that instance two years ago and again this morning. I caught that using a git grep "just in case ™" [12:45:24] it's ok, I have been working on it quite a bit [12:45:27] https://usercontent.irccloud-cdn.com/file/fVKbAQDw/image.png [12:45:34] \o/ [12:45:35] (in the past xd) [12:45:47] I knew I should have used git blame ahah [12:46:35] merci! [12:48:03] de rien! [13:18:57] dhinus: thanks! (for the review) [13:40:28] If there are no objections I'm going to start refreshing toolforge etcd nodes now [13:42:03] SGTM [13:42:29] actually team meeting in ~20m andrewbogott, not sure how long it takes ? [13:42:49] not very log, and it'll be one node at a time [13:43:04] ok! [13:58:23] ack [14:29:45] do we still allow nested KVM on the instances? I am rebuilding one that makes use of QEmu and it does not seem kvm is available inside the instance. [14:29:45] In Puppet I found: [14:29:45] modules/openstack/files/nova/kvm_intel.conf:options kvm_intel nested=1 [14:29:45] modules/openstack/manifests/nova/compute/service.pp: file { '/etc/modprobe.d/kvm_intel.conf': [14:29:45] So tentatively the compute hosts do have the kernel module loaded [14:30:06] this was done in early 2020 via https://phabricator.wikimedia.org/T267433 [14:30:27] so I wonder whether our compute hosts have the module loaded or if I am missing something in my VM config :) [14:37:13] IIRC nested KVM is mostly not supported; getting it to work was a deep, unpleasant rabbithole. If you have a compelling use-case we can revisit the question. [14:37:38] hm, although that task disagrees... [14:39:37] no, this task discusses removing the feature https://phabricator.wikimedia.org/T276208 [15:03:29] who has two thumbs and keeps stubbing them into openstack project names with dashes? [15:09:30] I was convinced I filed a task similar to https://phabricator.wikimedia.org/T422515 and I'm failing to find it heh [15:18:01] godog: some (most, old) projects have an ID == the project name, new projects have a uuid ID that is != name... is it possible that that's what you're encountering and not really an issue with a - ? [15:20:21] andrewbogott: ah I wasn't aware, definitely possible the root cause is that [15:20:56] I've tried to sort out the wording in cookbooks &c to distinguish between 'name' and 'ID' but there are probably still a few places where we conflate the two [15:21:00] having the id == name was something we relied on in many places, so yep, you might find that assumption is old scripts and such xd [15:21:59] hashar: andrewbogott about kvm nested, I think it opens some nice possiblities (like running lima-kilo inside VMs, for generating the images, or even running functional tests on demand) [15:22:31] not that we have the resources right now to act on it, but enables being able to do it xd [15:22:49] no worries! my use case does not use KVM currently, which I gues might make it slow [15:22:55] but it is probably not an issue :] [15:23:29] I think I got confused because of https://phabricator.wikimedia.org/T267433 being marked solved and Puppet having a modprobe configuration with `options kvm_intel nested=1` [15:23:34] andrewbogott dcaro ack, thank you that's useful info [15:23:56] andrewbogott dcaro do you mind if I copy/pasta the conversation on task / [15:23:59] ? [15:24:02] and I guess to have the module loaded that would require another stanza. So most probably those files should be removed from Puppet [15:24:03] not at all [15:24:07] godog: go for it [15:24:12] thank you [15:25:08] ok gotta go, see you tomorrow [15:31:41] \o cya [16:20:53] easy review if anyone has a minute https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/318 and https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1210 [16:31:08] dcaro: I think my confusion about the toml file is because on wikitech you wrote "pyproject.toml", but the upstream doc say "project.toml"? [16:32:11] xd dammit, I'll fix [16:42:23] thanks :) [16:44:14] dcaro: +1d your MRs [16:52:07] thanks! [17:05:55] * dcaro off [17:05:56] cya!