[00:16:00] * bd808 off
[10:11:25] <blancadesal>	 dcaro: builds-api uses a robot account for harbor, right?
[10:11:48] <dcaro>	 let me check
[10:19:08] <blancadesal>	 hmm, I think it's the maintain-harbor user
[10:20:42] <blancadesal>	 either way, there's no way with the current permissions to retrieve the default project quota  
[10:33:50] <dcaro>	 it uses maintain-harbor yes
[13:11:43] <blancadesal>	 dcaro: hmm, I was hoping to build and deploy builds-api locally because the gitlab pipelines take forever today, but I'm getting this error:
[13:11:53] <blancadesal>	 https://www.irccloud.com/pastebin/fuNoBPYb/
[13:13:16] <taavi>	 which docker version is that?
[13:15:59] <blancadesal>	 20.10.24
[13:16:58] <taavi>	 does it work if you pass DOCKER_BUILDKIT=1 to the `docker build` command?
[13:21:08] <blancadesal>	 it does
[13:28:52] <dcaro>	 nice, would be interesting to make sure everything works on lima-kilo
[13:28:56] * dcaro lunch
[13:51:03] <arturo>	 an early preview of the jobs-api openapi schema: https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/59
[13:51:06] <arturo>	 early comments welcome
[13:51:10] * arturo food break
[14:33:25] <blancadesal>	 relevant fosdem talk: https://mirror.librelabucm.org/fosdem/2024/aw1126/fosdem-2024-2531-introduction-to-openapi.av1.webm :)
[14:35:58] <blancadesal>	 https://fosdem.org/2024/schedule/event/fosdem-2024-2531-introduction-to-openapi/
[15:39:48] <arturo>	 cool
[16:12:30] <andrewbogott>	 dhinus: does my rambling on T356904 help explain anything?
[16:12:31] <stashbot>	 T356904: [toolsdb] [cinder] [ceph] Deleting snapshot does not work - https://phabricator.wikimedia.org/T356904
[16:15:06] <dhinus>	 andrewbogott: yes, thanks! though I'm not sure exactly what happened
[16:15:38] <andrewbogott>	 I'm pretty sure the issue is the same as always, which is that the backup process takes rbd snapshots which cinder doesn't know about.
[16:16:06] <dhinus>	 ahhh now I get it
[16:16:08] <andrewbogott>	 So there's a secret dependency chain that causes rbd to error out when cinder asks to delete things.
[16:16:36] <dhinus>	 so the problem will only happen on the volumes that have a non-cinder backup
[16:16:51] <andrewbogott>	 Yes, but that's almost everything more than 24 hours old.
[16:17:01] <dhinus>	 btw how did you find that log line in logstash? what did you filter for?
[16:17:14] <andrewbogott>	 We  had a similar issue with VM images not getting cleaned up after deletion, we now have a daily cron that purges them.
[16:17:26] <andrewbogott>	 I proposed an upstream patch for the VM thing but it's unpopular.  https://review.opendev.org/c/openstack/nova/+/843228
[16:18:12] <andrewbogott>	 I think I switched to cinder-only logs, and then just watched them tick by as a I tried to delete :)
[16:18:24] <andrewbogott>	 But searching for uuids often works too
[16:21:45] <andrewbogott>	 (on this dashboard https://logstash.wikimedia.org/app/dashboards#/view/3ef008b0-c871-11eb-ad54-8bb5fcb640c0?_g=h@865c245&_a=h@cd6aa03)
[16:22:15] <andrewbogott>	 I usually start by editing the 'openstack services' filter to select the service(s) I'm interested in
[17:02:31] * arturo offline
[19:27:57] * bd808 lunch