[08:56:51] <blancadesal>	 morning!
[08:58:03] <dcaro>	 morning :), I might be in-out at times today, had a rough night
[08:59:53] <blancadesal>	 oh, sorry to hear that :(
[09:00:19] <blancadesal>	 could I get a +1 on this quota increase request? https://phabricator.wikimedia.org/T352136
[09:05:17] <dcaro>	 done 👍
[09:06:20] <blancadesal>	 thanks :)
[09:08:36] <blancadesal>	 cookbook fails :( https://usercontent.irccloud-cdn.com/file/xyHulBsa/Screenshot%202023-12-06%20at%2010.08.14.png
[09:21:59] <dcaro>	 I think there's a typo in the error message xd "run_one_as_dict -> expecting a list" should be "dict" I guess
[09:22:08] <dcaro>	 it might be related to the openstack upgrade
[09:30:36] <blancadesal>	 did it with `sudo wmcs-openstack quota set` instead
[09:34:39] <dcaro>	 blancadesal: can you open a task for it? (if you are not already fixing it xd)
[09:35:18] <blancadesal>	 yup, creating a task :)
[09:44:58] <blancadesal>	 dcaro: for projects requesting deletion: if they still have instances, do we ask the owners to delete them, or do we just do it ourselves?
[10:09:25] <TheresNoTime>	 is there a way for me, a WMCS root, to download a snapshot/image of a VM?
[10:11:59] <TheresNoTime>	 (curiosity, no specific use-case really)
[10:14:26] <dcaro>	 blancadesal: We can delete, but double-check with them just in case they want to save anything, a precaution would be to shelve the VMs for a bit before deleting the project completely (not sure what's the specific case though)
[10:16:27] <dcaro>	 TheresNoTime: hmmm, not sure xd, I guess if you have access to the cloudvirts/cloudcontrol nodes you can always dump it from ceph with rbd, but there must be a way to do something like that with openstack/cinder
[10:17:06] <TheresNoTime>	 would have thought Horizon would have A Button(tm) to do something like that, but *shrug* :)
[10:17:22] <dcaro>	 oh, you might need to download both, the VM and each volume also for a complete setup
[10:18:56] <dcaro>	 hmm, it seems horizon has no such feature yet (from 2018 post): `
[10:18:56] <dcaro>	     Currently there is no option for downloading snapshots and images from Horizon dashboard, python-glanceclient already supports this by calling the image-download sub-command, but Horizon misses this feature.
[10:18:56] <dcaro>	 `
[10:21:08] <TheresNoTime>	 is an odd feature to be missing, but ho hum ^^
[10:28:16] <blancadesal>	 dcaro: https://phabricator.wikimedia.org/T352727
[10:29:48] <dcaro>	 blancadesal: sounds like it would be safe to delete yes, it has been shut down for a while already, and the owners acked that there's no data they want saved
[10:36:20] <blancadesal>	 dcaro: ack
[10:49:47] <blancadesal>	 dcaro: I'm not sure what's going on here: https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/pipelines/33009 
[10:50:53] <dcaro>	 precondition failed :/, maybe the image/version already exists and it's immutable somehow? Or the quota (though iirc that ended up in entity too large)
[10:51:42] <dcaro>	 no such image, and the quota is unlimited xd
[10:53:05] <dcaro>	 wait, I was looking tools, this pushes to toolsbeta
[10:53:23] <dcaro>	 hmm, there we have 100G of quota
[10:53:56] <dcaro>	 and for some reason the dev tag is immutable
[10:53:58] <dcaro>	 https://usercontent.irccloud-cdn.com/file/NsyYmP5q/image.png
[10:56:21] <dcaro>	 let me check the latest maintain-harbor changes again xd
[11:01:08] <dcaro>	 I'm guessing that the immutable rule was disabled for toolsbeta, and with the images cleanup, we re-enable all the immutability rules at the end (even if they were disabled)
[11:03:49] <blancadesal>	 ah yes that was the patch that was merged yesterday
[11:05:16] <dcaro>	 I think we can either change it so it does not match the -dev images, or just delete it
[11:05:20] <dcaro>	 (from toolsbeta)
[11:06:25] <dcaro>	 I just deleted it
[11:08:27] <dcaro>	 you can retry the steps there in the pipeline :)
[11:13:27] <blancadesal>	 what did you delete? 
[11:15:40] <dcaro>	 from toolsbeta, in the toolforge project, inside policies, there's a `immutability` tab, there was a rule there saying that every tag on every repo was meant to be immutable (you can see it in the tools-harbor instance), that's what I deleted
[11:16:26] <dcaro>	 hahaha, or maybe not, I deleted it from tools xd
[11:16:30] <blancadesal>	 haha
[11:16:43] <dcaro>	 done
[11:17:10] <dcaro>	 hmm, maybe we can tweak the UI of one of them to look different....
[11:19:31] <blancadesal>	 locally, I have toolsbeta in light mode and tools in dark ¬‿¬
[11:19:51] <dcaro>	 ooohhh, neat
[11:21:46] <dcaro>	 hmm... you have to rebuild the portal component to change the look it seems: https://goharbor.io/docs/1.10/build-customize-contribute/customize-look-feel/
[11:22:54] <blancadesal>	 interesting! we should try it :)
[11:23:20] <blancadesal>	 yay, jobs are passing! time for lunch :))
[15:37:23] <taavi>	 is the solution you're going with for T352774 flexible enough to also easily add a rust buildpack?
[15:37:23] <stashbot>	 T352774: [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools - https://phabricator.wikimedia.org/T352774
[15:40:57] <dcaro>	 taavi: should be yes, most of the "bother" comes from getting a copy of the buildpack in our gitlab repo and keeping it up to date so we don't pull it from the internet on every build
[15:41:08] <dcaro>	 still working on it though
[15:41:50] <taavi>	 cool
[15:54:06] <andrewbogott>	 taavi: have time for a quick review?  It turns out keystone needs to be rotating credential keys just like fernet keys (I'm assuming this is why existing app creds stopped working when we rebuilt cloudcontrols, although the story doesn't totally add up)  https://gerrit.wikimedia.org/r/c/operations/puppet/+/980465
[15:55:26] <taavi>	 andrewbogott: unfrotunately not at this moment, I'm just about to head to the office for breakfast
[15:55:30] <andrewbogott>	 ok!
[15:55:40] <andrewbogott>	 I'm rereading the docs and think that my patch might be totally wrong anyway :/
[15:59:31] <TheresNoTime>	 seen a few irritated folx making comments about the Toolforge grid decom, so just wanted to let y'all know you're awesome & appreciated ✨
[16:07:17] <dcaro>	 <3 thanks! It does mean a lot :)
[16:09:10] <balloons>	 TheresNoTime, thank you!
[16:14:21] <dcaro>	 nice, I got the apt buildpack injected before the detect step
[16:14:25] <dcaro>	 https://www.irccloud.com/pastebin/rVssLVb3/
[16:14:49] <balloons>	 dcaro, you are a wizard!
[16:15:35] <blancadesal>	 dcaro: 🚀
[16:21:35] <dcaro>	 andrewbogott: did you recreate tf-infra-test vm yesterday by hand?
[16:21:56] <andrewbogott>	 dcaro: no. but the terraform daily test might have
[16:22:22] <dcaro>	 andrewbogott: where does that live? it's missing adding that VM to the default security group
[16:23:33] <andrewbogott>	 I don't know a whole lot, but I believe the tests live in tf-bastion.tf-infra-test.eqiad1.wikimedia.cloud:/tf-infra-test
[16:23:41] <andrewbogott>	 (and github someplace)
[16:24:30] <dcaro>	 we should probably move that to gitlab
[16:27:26] <balloons>	 https://github.com/toolforge/tf-infra-test
[16:28:11] <dcaro>	 oh, it uses one of those git-crypt secret files
[16:29:19] <dcaro>	 I'll need someone to add me I guess
[16:30:23] <dcaro>	 anyone has access to it?
[16:30:27] <dcaro>	 Rook: ?
[16:34:39] <andrewbogott>	 The key is in their home dir on that host
[16:36:58] <dcaro>	 so we are using only one key for everyone?
[16:37:44] <andrewbogott>	 ...I don't know enough about git-crypt to answer that usefully.
[16:37:48] <andrewbogott>	 I think in this case yes
[16:38:48] <balloons>	 I believe that's correct. Rook can give more details, but terraform needing state meant it had to be stored somewhere. I believe with object storage now we can design something better
[16:41:13] <dcaro>	 the state is a different thing, this is just secrets (I guess that to avoid putting them in puppet they were put in the repo)
[16:42:02] <Rook>	 I've been running into problems with object storage for tofu state https://phabricator.wikimedia.org/T352164
[16:43:25] <taavi>	 Rook: seems like you're missing the project name from the bucket name? see the example on https://wikitech.wikimedia.org/wiki/Help:Using_OpenTofu_on_Cloud_VPS#State_management
[16:43:30] <Rook>	 Yes, we've been using a shared key per repo for git-crypt where it is used. It apparently supports multiple keys, though I'm not immediately convinced that it would be useful
[16:44:15] <Rook>	 taavi: I'll give that a try. It was working without the project name, but not with variables
[17:16:12] <Rook>	 Looks like volume detachment problems https://www.irccloud.com/pastebin/SFX3ZQGU/
[17:17:39] <Rook>	 VM test volume looks stuck too https://www.irccloud.com/pastebin/bIcVGrnu/
[17:17:43] <Rook>	 andrewbogott: thoughts?
[17:20:02] <dcaro>	 Rook: andrewbogott just created T352895
[17:20:02] <stashbot>	 T352895: [tf-infra-tests] Failing to destroy - volumes stuck - https://phabricator.wikimedia.org/T352895
[17:20:10] <dcaro>	 can you add your findings there? I got to go for the day
[17:20:30] <Rook>	 Can do, have a good evening :)
[17:20:37] <andrewbogott>	 Rook, I'm in a call now sorry
[17:20:51] <Rook>	 c'est pas grave
[17:20:54] <dcaro>	 I'll follow up tomorrow, would appreciate also a look to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed and https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed and make sure what I updated is ok
[17:21:00] <dcaro>	 thanks, cya!