[06:45:47] <taavi>	 morning
[07:50:10] <arturo>	 morning 
[08:57:24] <dcaro>	 morning
[08:57:41] <dcaro>	 hmm... I think that the ceph cluster has grown over the weekend :/
[08:58:16] <dcaro>	 and it will not support taking out the rest of osds without exceeding 90% usage
[09:00:20] <dcaro>	 current (mean) usage is 79.7%, and we have 229 osds in, there's 21 osds left to drain, but if we drain those, it will go to >91% usage :/
[09:04:49] <dcaro>	 let me undrain the hosts, see if the cluster returns to the <60% :/, it feels like the cluster grew extra (from the original 272 osds at ~56% capacity, 201 osds would be at ~76%...)
[10:54:23] * taavi lunch
[10:55:05] * dcaro lunch
[10:59:44] <TheresNoTime>	 hm, today on "odd stuff I don't want to touch as it'll break things" https://usercontent.irccloud-cdn.com/file/SvhDEyhR/Screenshot%202023-10-09%20at%2011.57.40.png
[11:00:10] <TheresNoTime>	 volume attached twice to the same instance/same mount point..?
[11:00:25] <arturo>	 probably an openstack-side mishap
[11:01:18] <TheresNoTime>	 (everything works fine, just odd..!)
[11:01:44] * TheresNoTime knows that if they try to "detach" one of those, it'll break everything
[11:05:58] <arturo>	 hopefully not :-)
[11:06:20] <arturo>	 maybe an operation (detach, reattach) will resync the state with the reality
[11:06:23] <arturo>	 (but just maybe)
[11:06:44] <arturo>	 the odds that it breaks is also non 0% hehe
[12:39:10] <dcaro>	 taavi: is this still being used/needed?  https://gerrit.wikimedia.org/r/admin/repos/cloud/toolforge/delete-crashing-pods
[12:39:24] <dcaro>	 also  https://gerrit.wikimedia.org/r/admin/repos/cloud/toolforge/kube-container-updater
[12:40:12] <taavi>	 dcaro: https://phabricator.wikimedia.org/T334399
[12:40:44] <taavi>	 I'm not interested in delete-crashing-pods anymore. kube-container-updater could become a repo in my personal namespace for now I think
[12:41:39] <taavi>	 imported as https://gitlab.wikimedia.org/taavi/kube-container-updater, so both of those gerrit repos can be archived
[13:10:54] <dcaro>	 taavi: thanks!
[13:50:19] <andrewbogott>	 Thanks for whoever ack'd and dealt with the cloudbackup200x things overnight. I'm running an epic cleanup job there that should free up disk space and make things happy... it seems to  have worked on 2002 but crashed on 2001 so I'm re-running
[13:50:27] <andrewbogott>	 (and then will try to figure out why it's not running automatically)
[13:50:44] * andrewbogott out today but will be around for a few minutes in case anyone needs unblocking
[15:58:12] * arturo offline