[11:43:45] Sorry for not sending updates here for a few days! [11:43:47] Done: [11:43:49] * helped with the runc upgrade (T356507) [11:43:51] Doing: [11:43:53] * understanding toolforge standalone images (T355231 and T277749) [11:43:55] * T344717 creating a new toolsdb replica [11:43:57] Blockers: [11:43:59] * T356904 cannot delete a cinder snapshot (not really a blocker, I'm investigating) [12:02:25] Done: [12:02:25] * Found the root of the connectivity issues on k8s (rate-limiting concurrent connections from a single IP at the wiki CDN layer) [12:02:25] Doing: [12:02:25] * Toolforge grooming, did not have much time yesterday [12:02:25] * Continue doing a full pass to all the tool migration tasks, did not have much time yesterday [12:02:25] * Connectivity issues between cloudcontrols and cloudrabbit nodes, the heartbeat changes did not help :/ [12:02:25] * Toolforge refinement today in the collab session [12:02:26] * Ceph outage yesterday, will fill up some logs in the task and try to play with snaptrim (thanks dhinus and andrewbogott !!) [12:02:26] Blockers: [12:02:27] * None [16:01:27] Did: [16:01:27] • Tinkered with the k8s.reboot cookbook a bit, patches pending in gerrit [16:01:27] • designate-sink is flaky in codfw1dev since I moved it to cloudcontrols. I found at least one issue there but at least one remains. [16:01:27] Doing: [16:01:27] • trying to find what else is broken with designate-sink in codfw1dev [16:01:28] • Understanding (and fixing?) why it's hard to delete snapshots T356904