[08:00:19] Cteam: welcome to today 🦄! Don’t forget to post your update in thread. [08:00:19] Feel free to include: [08:00:19] 1. 🕫 Anything you'd like to share about your work [08:00:19] 2. ☏ Anything you'd like to get help with [08:00:19] 3. ⚠ Anything you're currently blocked on [08:00:19] (this message is from a toolforge job under the admin project) [09:57:27] Tuesday: [09:57:30] * T345811 started to upgrade cloudcontrols [09:57:30] T345811: [openstack] Upgrade eqiad hosts to bookworm - https://phabricator.wikimedia.org/T345811 [09:57:32] * T350172 regenerated openstack tokens [09:57:32] T350172: Unable to retrieve instances - https://phabricator.wikimedia.org/T350172 [09:57:34] * T350154 copied the openstack prometheus exporter to bookworm [09:57:34] T350154: [openstack] prometheus exporter broken in bookworm - https://phabricator.wikimedia.org/T350154 [09:57:36] Yesterday: [09:57:38] * I didn't work (public holiday) [09:57:40] Today: [09:57:42] * T345811 upgrading the remaining two cloudcontrols [09:57:44] ** T350188 checking ceph-common versions [09:57:44] T350188: [openstack] Fix ceph-common version in Bookworm - https://phabricator.wikimedia.org/T350188 [09:57:46] ** T350154 enabling the exporter in codfw [09:57:48] * T349695 debugging the recurring toolsdb OOMs [09:57:48] T349695: [toolsdb] MariaDB process is killed by OOM killer (October 2023) - https://phabricator.wikimedia.org/T349695 [13:55:25] Yesterday/Today: Pretty much just draining and undraining ceph OSDs. It's not very demanding but it's about all I have the bandwidth for. T348643 [13:55:25] T348643: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643