[09:41:34] anybody knows why the s1 schema change is taking so long in db1154? T367856 [09:41:35] T367856: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856 [09:41:41] current replag is 71 hours [10:12:50] dhinus: m.arostegui or Amir.1 would be the ones to comment with authority (but they're both OoO); my impression was that it was expected that this would take some time to run (and that dumps has been causing some additional lag in s1 too, which is probably not helping). [10:13:37] I'm just the storage person, though, so take the above with a pinch of salt ;) [10:34:34] Emperor: thanks, dumps might explain the additional slowness... anyway it's progressing, I'll keep an eye on it [14:00:06] I'm depooling clouddb1020, then reimaging it to bookworm (T365424) [14:00:18] T365424: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424 [14:27:50] hnowlan, elukey: sorry for being so late to the party on https://gerrit.wikimedia.org/r/c/generated-data-platform/datasets/image-suggestions/+/1058588, I've been out on sickleave [14:28:25] urandom: o/ np, I hope you feel better now.. no rush, I am just trying to tidy-up thinks in k8s land :) [14:28:39] I see that it's been merged, but not deployed. I reckon it's probably safe(ish) to do so, but we don't have any way to test [14:30:20] that service is meant to be replaced entirely by the data-gateway service (which is ready to go), and migration is currently pending a way to test :/ (https://phabricator.wikimedia.org/T368096) [14:31:49] (not having a way to test is what caused me to balk when I saw the gerrit yesterday) [14:32:58] urandom: ahh I see - on the beta -> k8s staging access, verrrry unlikely. would deploying a data-gateway instance in staging require us to first create the universe? [14:33:20] our docker shim is actually remarkably forgiving, I just stood up a shellbox there and it was surprisingly easy [14:34:53] it would require cassandra, with the schema used on the aqs cluster, and some data, and ofc the service itself [14:35:08] and ownership :) [14:36:02] hehhhh [14:36:07] hnowlan: with mw mov(ed|ing) to k8s, will staging there be a thing? [14:36:29] how will k8s staging be utilized vis-a-vis mw? [14:36:39] (if at all) [14:36:56] urandom: no, we just have canaries and no staging. But testing locally will be easier with more and more dockerisation [14:37:33] locally? [14:38:30] yeah not so useful in this case [15:14:12] urandom: given that buster is completely EOL, I'd suggest to deploy the new image sooner rather than later [17:15:19] clouddb1020 is reimaged and repooled [17:17:44] in other news, the s1 schema change has finally completed on db1154 (sanitarium) and it's now being applied to replicas downstream of that sanitarium