[05:17:12] I'm going to try to switch s6, s1 and es4 before 8am [06:47:39] All done, I am now going to go for x2, which is going to be fun.... [06:54:31] Enabling writes back on es4 [07:26:10] ALmost 1h to switchover x2.... [07:26:15] Incredible [07:26:19] 😱 [07:26:20] anyway, I think it is done [07:30:27] And now I am going to switch s3 [07:46:55] marostegui: yeah, that is fixed, db2136 has pymysql 1.0.2 now [07:47:13] arnaudb: shall we move forward and upgrade the remaining bullseye DB hosts now? [07:47:35] I think it'll be OK yep! marostegui wdyt? [07:49:05] arnaudb: I am busy, sorry [07:53:52] moritzm arnaudb ok to merge from my side [07:54:45] I have done 3 hours of non stop switchovers on my own, I am mentally exhausted, I am going to be out for the rest of the day. [07:55:25] As a team we need to discuss the network maintenance task spread, because this is not sustainable [07:56:16] sure thing, take care meanwhile :) [07:57:01] take some rest, I'll take care of the pymysql rollouts in half an hour and will sync up with Arnaud [08:46:41] pymysql has been upgraded to 1.0.2 on all DB hosts running bullseye now [08:48:08] jynus: backup[12]002, the dbprov* hosts and ms-backup are also on bullseye and have pymysql installed (possibly pulled in for the same tools as for https://phabricator.wikimedia.org/T355531 [08:48:30] shall I also upgrade these or do you want to test some backup-specific things first with 1.0.2? [09:14:39] I belive wmfbackups doesn't use (or very minorly- just for storing data) the mysql bindings and it was prepared for bookworm already: https://gitlab.wikimedia.org/repos/sre/wmfbackups/-/compare/main...bookworm?from_project_id=1639&straight=false [09:15:32] dbprov1/2003 should be already in bookworm [09:15:45] I wouldn't touch ms-backup, though [09:15:54] is there any reason you want to push its upgrade now? [09:17:24] none of these tools should be installed or used from those hosts [09:24:35] however, I suggested creating a ticket for tracking the compatibility issues for 1.0.2 - in particular, the way exceptions are handled [09:42:41] there's no reason to immediately upgrade these now, I can also simply keep them on 0.9.3, it's just that pending updates are a bit of a lurking issue, if e.g. a dbprov host would get reimaged on bullseye it would end up with 1.0.2 [09:43:15] but if the next reimage would target bookworm anywa, we can also leave them as-is, just wanted to go through the list of non-DB core users of pymysql for some headsup [09:56:07] I'd prefer to leave them as is- I prepared for bookworm, but this exception issue, I would like to track it and check it on all of my software before upgrading [09:56:34] I also made sure all the software is ready for puppet 7 and the new cert already [10:03:59] ok, sounds good to me [10:57:02] marostegui: FYI, I'm starting maint work on the old replicas