[09:54:32] always a good start: `Unknown/unsupported storage engine: InnoDB` [10:01:50] try MyISAM :D [10:07:25] I heard TokuDB is webscale [10:14:38] turns out this was code for "the database is corrupted". it's so obvious now [10:26:42] gotta love those helpful error messages [10:32:48] * Emperor remembers iRODS having a bug where it would sometimes bomb out 'terminating with uncaught exception of type boost::filesystem::filesystem_error: boost::filesystem::directory_iterator::operator++: Not a directory: "/proc/self/fd"' [10:33:27] ...which is quite an alarming error message [10:36:57] (this turned out to be a race condition in the logging code(!)) [10:38:21] yikers [10:40:14] https://github.com/irods/irods/issues/4943 has all the gory details (TL;DR: don't close() before dup2() ) [10:41:55] 👌 [10:43:29] sounds like some excellent debugging [11:00:59] ...of course, there was a systemd foul-up making things worse too [11:01:50] FYI, I am going to do some tests of backups with cumin2002 with the new packages (e.g. on a smaller db, like x1) [11:02:25] this shouldn't affect production dbs or production backups, but giving a heads up [11:02:31] ooh, backup testing. Can I delete some things? :-D [11:02:46] we are testing backups, not recoveries :-D [11:06:41] I now have to find the yaml key where I disabled backups on that host somewhere on hiera [11:32:02] Sometimes puppet threw error at me parameter foo is type Str but must be type Str. Very helpful (it was uppercase but the error made both uppercase so they became identical) [12:41:31] Amir1: 🤦‍♀️ [13:50:51] jynus: wmfmariadbpy 0.9 is now released and deployed, so there _shouldn't_ be anything blocking you from my side. if reality differs, please let me know :) [14:12:12] yeah, I am relatively confident about the unit side of things, it is the integration testing where things may explode :-) [14:13:02] so many variables chaning: new packages, new dependencies, new os, new python version, new mariadb versions, etc. [14:13:06] *changing [15:07:38] [15:07:12]: INFO - Running XtraBackup at db2101.codfw.wmnet:3320 and sending it to dbprov2003.codfw.wmnet [15:08:19] apparently mkdir and tcp implementation hasn't changed from buster to bullseye :-P [15:35:08] sobanski: Hi