[08:00:23] Cteam: welcome to today 🦄! Don’t forget to post your update in thread. [08:00:23] Feel free to include: [08:00:23] 1. 🕫 Anything you'd like to share about your work [08:00:23] 2. ☏ Anything you'd like to get help with [08:00:23] 3. ⚠ Anything you're currently blocked on [08:00:23] (this message is from a toolforge job under the admin project) [09:03:07] Done: [09:03:07] * Removed a bunch of old backups from ceph (cloudbackup1004) [09:03:07] Doing: [09:03:07] * Continue looking for things to cleanup in ceph, after re-adding all but one host, we are still at 67% usage, for a safe rack shutdown we need to be at ~60% (because we are losing ~25% capacity + 5% leeway for osds not to be perfectly rebalanced + 10% leeway for background operations before cluster writes are blocked) [09:03:07] * I looked into s3 CSI for kubernetes, though it seems that the only one around is yandex specific one (that uses their fuse libraries too), not sure if we want to use that. [09:03:07] * Will try to prepare everything for the open beta (wiki/email/etc.) [09:03:07] Blockers: [09:03:08] * None [10:34:39] Yesterday: [10:34:49] * T348531 [10:34:50] T348531: hw troubleshooting: disk failure for cloudvirt2004-dev.codfw.wmnet - https://phabricator.wikimedia.org/T348531 [10:35:11] * T341285, upgraded cloudvirts [10:35:11] T341285: Upgrade cloud-vps openstack to version 'Antelope' - https://phabricator.wikimedia.org/T341285 [10:35:16] Today: [10:35:47] * the hardware problem is apparently resolved, so I will continue with the OpenStack upgrade T341285 [10:38:11] * I will also fix some CI errors in the wmcs-cookbooks repo (some dependency update have broken our CI tests)