[09:33:36] that's probably chuckonwu :), @chuckonwu feel free to ask around on how to fix it if you don't find it yourself [09:33:51] hmm, someone has been NFSing during the nigh! https://usercontent.irccloud-cdn.com/file/jap4vUAc/image.png [09:37:06] 👀 [09:49:39] o/ I'm back from holidays and catching up... let me know if there's anything I should look at first. [09:51:08] welcome back [10:58:17] heads up, merging tofu-infra refactor [11:03:01] tofu-infra refactor merged & applied successfully :) [11:03:13] 🎉 [11:04:27] thanks arturo for all the work on this MR [11:06:02] thank you for the assistance! dhinus [11:06:22] on other topic, T389196 is ready for the clinic duty person, who is that? [11:06:22] T389196: Increase quota for Traffic cloud project - https://phabricator.wikimedia.org/T389196 [11:13:23] clinic duty is k.omla I think [11:47:56] T389775 may be of interest for us? [11:47:57] T389775: Stuck on "Waiting for an available worker" - https://phabricator.wikimedia.org/T389775 [11:51:11] not really, that's an individual tool with a history of getting stuck that's again gotten stuck? unless you see something I'm not seeing that suggests a platform issue [11:51:58] do you suspect of a quota problem? [11:52:43] that's my guess, have not checked, but if they are trying to start too big of a container it will have a hard time getting scheduled [11:53:19] huh [11:53:22] > Error creating: admission webhook "volume-admission.tools.wmcloud.org" denied the request: No hostPath volumes allowed, got one under /spec/volumes/0 Name:dumps HostPath:&HostPathVolumeSource{Path:/public/dumps/,Type:*,} [11:53:39] has there recently been changes to that webhook? [11:53:54] taavi: yes [11:54:19] related to https://phabricator.wikimedia.org/T389775 [11:54:31] sorry, https://phabricator.wikimedia.org/T386921 [11:54:51] https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [11:55:43] that's not quota xd [11:56:15] hmm, that seems to be denying quite a lot of things that were previously allowed [11:56:30] is the idea that all NFS mounts would have to use `toolforge: tool` now? [11:56:36] yep [11:57:01] rely on the admission controller to add the mounts for them [11:58:05] (side note, using a mutating controller to do validation feels wrong, although I'm not sure if that will actually break anything) [11:59:49] it should not, tekton does it quite often [12:00:47] though we could create a validation config and a mutating config that point to the same service instead of just mutation (like they do) [12:01:19] do you want to add a not in the task? I'll do if you you prefer [12:01:22] please do [12:01:26] ack [12:01:45] also, anyone mind if I make https://phabricator.wikimedia.org/T386921 public? as the patch has been deployed I think that should be fine now [12:02:44] ok from me [12:07:16] * dcaro lunch [12:08:50] also is there some specific reason to allow the `home` mount to be manually configured? [12:25:05] (many thanks again for the help with T389775 — was trying to take a half-look between work things) [12:25:06] T389775: Stuck on "Waiting for an available worker" - https://phabricator.wikimedia.org/T389775 [12:29:10] TheresNoTime: i assume you're already aware, but usual reminder that running things via https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework will avoid these kinds of surprise breaking changes [12:35:33] yeaaaaaah ^^' I've somewhat inherited keeping a half eye on this tool, but it might be worth putting in the effort to stop these sorts of things happening [13:19:00] taavi: that it comes from webservice iirc [13:19:39] arturo: I think there's something going on on coludcontrol1005 opentofu repo, it's making puppet fail [13:19:44] are you doing anything there? [13:19:45] ``` [13:20:01] https://www.irccloud.com/pastebin/mqggtio3/ [13:29:27] mmm [13:29:36] dcaro: no, we merged the refactor a few hours ago [13:29:52] let me take a look [13:31:02] I wrote something somewhere that it's not here (so not sure which chat/window it went to xd) [13:31:23] I ran th ecommand manually using a more specific ref (refs/remotes/origin/HEAD) and puppet is running now [13:51:54] Do you remember how to get somebody to have sudo powers in toolforge VMs? I'm trying to get that for chuckonwu [13:53:01] he is already member of the admin tool in Toolforge [13:53:12] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#What_makes_a_root/Giving_root_access [13:53:35] so the sudo policy that can be edited via Horizon [13:56:16] I was reading this doc piece https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolforge_roots_and_Toolforge_admins#sudo_policy_%22roots%22 and not understanding that I needed to look in horizon [13:57:03] we should maybe replace that separate list with a direct reference to the admin tool [13:57:56] taavi: do you remember how to get Chuck access to cloudcontrols for interacting with the openstack CLI ? he is not a SRE, so no general prod access [13:58:14] was there like a group that I could add membership via ops/puppet? [13:59:20] exactly, he needs to follow https://wikitech.wikimedia.org/wiki/SRE/Production_access to get added to the `wmcs-roots` group [14:00:12] thanks! [14:00:46] Thanks taavi! [14:07:45] I'm confused by how the "sudo policy" works, where do you enable it in horizon? [14:08:53] access -> project sudo in horizon [14:09:14] ah ok! I think the doc page that arturo linked to is misleading, I created that page but it was mostly a copy-paste. I'll edit that. [17:01:11] * arturo offline