[07:23:41] arnaudb: Can I deploy a schema change on the old s5 master? [07:24:20] it's actually repooling in slow mo as I wanted to be cautious [07:24:30] so if you want to stop the process sure! [07:24:33] ah no problem [07:24:36] let me know when done [07:24:55] otherwise I can ^C and depool it again? [07:25:00] you tell me! [07:25:15] nah it is okay [07:25:16] no rush [07:25:31] I will check when it is fully repooled [07:25:38] sure [08:32:18] marostegui: would you mind me upgrading db1245 before you run your schema update? [08:43:53] I don't think I am running any schema change there [08:44:06] I just need the master [08:44:09] (the old one) [08:44:18] Which is done now! [08:44:18] [10:42:23] <+logmsgbot> !log arnaudb@cumin1002 dbctl commit (dc=all): 'db1183 (re)pooling @ 100%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60873 and previous config saved to /var/cache/conftool/dbconfig/20240418-084223-arnaudb.json [08:45:05] oh then please proceed [08:45:07] I'll wait :) [08:45:20] I don't need db1245 for anything [08:45:24] great [09:37:44] marostegui: for T360029 can I try to change without committing the IP of a DB via spicerack? I'll revert it right after, just to check the current library works for this. Maybe a depooled one or an x2 replica that is not in mwconfig [09:37:44] T360029: Integrate dbctl IP changes as part of VLAN changes. - https://phabricator.wikimedia.org/T360029 [09:42:52] volans: sure thing [09:43:14] any x2 would replica would work [09:43:14] any hostname you suggest? :) [09:43:16] ok [09:43:17] thx [09:43:24] if yiu want me to depool another host just let me know [09:43:32] nah should be fine [09:48:12] {done} updating task, all good [09:57:32] marostegui: arnaudb Can I play with the old s5 master? [09:57:40] Amir1: not yet [09:57:49] I don't mind! [09:57:53] 💔 [09:59:19] Amir1: I will let you know when you can [09:59:32] aesome [10:50:11] I just tested db-mysql in cumin2002 and it does connect to prod hosts via cumin2024 user instead of root and things look fine. Please let me know if things break. I won't drop the user yet. cc marostegui arnaudb [10:56:09] Nice [10:56:19] I haven't seen anything strange so far [10:56:57] Most of the stuff should be using using db-mysql, I don't think we've got anything with user hardcored or anything like that [10:59:00] I think you have bash that connects to every host in every section, wanna try running show processlist on everything to see if any host is missing the user/grant [11:00:21] nothing urgent, my tool said it's been everything and I'm sure it's at least covering 99% of them anyway [11:08:05] Amir1: Mine uses db-mysql [11:11:55] Amir1: I am done with db1183 [11:34:49] marostegui: sorry I was in a meeting, you I mean just checking if the new cumin user is created and has the rights on all all hosts (e.g. recovery, reimage, etc. etc.) [11:35:05] thanks. I go play with db1183 [11:35:54] Amir1: yep, I can. However what you can do is also remove the old one from a production host and see if something screams [11:36:17] oh yeah, good idea [15:05:29] urandom: o/ aqs codfw is on pki! [15:05:48] \o/ [15:35:56] elukey: woo! \o/ [15:36:15] <3 [15:40:27] I think we can do eqiad next week, if these days pass without any hiccup [15:41:26] one question that I have regarding restbase - is the nodejs app the only client of the cassandra instances? [15:41:38] IIRC it should be, and it connects to cassandra via localhost [15:42:04] ah no maybe I am wrong [15:43:15] my main doubt is if the nodejs service uses TLS to connect to cassandra, to update its settings [15:43:23] otherwise we can proceed anytime [15:43:29] should be the same as aqs [15:49:49] urandom: --^ [16:09:58] there is also echoseen, which is a different instance of kask [16:10:15] and also PCS [16:10:27] which used to use restbase, but now connects directly to the db [16:11:29] all of which use encryption, even if they are not doing validation [16:11:54] elukey: --^ [16:18:44] ack thanks!