[08:10:54] morning [08:11:23] hi [09:37:56] is m1 down? [09:38:20] or db1128 is under maintenance? [09:43:56] PROBLEM - pt-heartbeat-wikimedia service on db1128 is CRITICAL: CRITICAL - Expecting active but unit pt-heartbeat-wikimedia is failed https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [09:46:06] RECOVERY - pt-heartbeat-wikimedia service on db1128 is OK: OK - pt-heartbeat-wikimedia is active https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [14:44:16] so the MariaDB and MariaDB-log this is both good and bad news [14:44:28] good because it is not a backups issue, only automation [14:44:43] bad because this is the first time we get that error [14:45:11] not sure if there are some changes on packaging or something that could cause that issue [14:46:17] e.g. maybe backups need a restart for the right binary to be used or something [14:47:24] or something was upgraded on dbprovs or some other issue [14:49:32] db1139 says "Server version: 10.4.22-MariaDB MariaDB Server" but db1128 says "Server version: 10.4.22-MariaDB-log MariaDB Server" marostegui [14:50:14] but both have the same package :-( [14:50:19] that's weird [14:50:29] db1139 is buster? [14:50:37] ah, let me see [14:51:06] 10.4.22+deb11u2 in db1139 [14:51:23] 10.4.22+deb11u2 in db1128 [14:51:37] :-/ [14:51:49] same debian version on both [14:51:52] 11.3 [14:51:57] weird, isn't it? [14:52:23] the workaround is easy- the backup technically not failed, it stopped as a protection (intended) [14:52:34] but would like to know what the different identification [14:52:52] *why [14:52:53] maybe they are using the buster package and not the bullseye one? [14:52:57] for when it *does* matter [14:53:01] I stopped packaging for buster [14:53:34] the part that changes is the one reported by @@version [14:53:34] it is the bullseye one [14:54:10] -log General logging, slow logging or binary (replication) logging is enabled. [14:54:13] https://mariadb.com/kb/en/version/ [14:54:26] yes, that is a common thing on some builds [14:54:33] not worried about that, it is a non-issue [14:54:39] but it shouldn't happen [14:55:14] because we have disabled that behaviour? [14:55:41] so the "protections" I put for making sure versions are the same go off [14:55:47] in this case we could ignore them [14:56:03] but I would like to know why it is happening, in case next time they matter [14:56:15] e.g. 10.4.22-MariaDB-percona-version [14:56:33] I want to be sure what is going on before confidently ignore it [14:57:58] e.g. if the package has changed, and we knew about it- I can just ignore the -log [14:58:35] but that is the issue, it shouldn't have changed, they have the same package version installed [14:59:05] jynus: my understanding is taht the same version reports different suffixes based on live configuration [14:59:17] as reported by the link I pasted above [14:59:22] oh, what? [15:00:00] I interpret that as "as configured on compile time" [15:00:03] jynus: 10.4.25-MariaDB-log [15:00:06] wow [15:00:07] that's the new package too [15:00:19] so it changes based on live config? [15:00:34] I think so at least for the -log one "General logging, slow logging or binary (replication) logging is enabled." [15:00:44] e.g. if I reboot db1139 and enable the binlog, it would say MariaDB-log? [15:00:45] those seems to be runtime config, not build config [15:01:02] that's how I interpreted it, but I might be wrong :D [15:01:11] volans: you understand why I think that is not great :-) [15:01:24] mixing "The server is compiled for debugging." [15:01:39] vs "I just decided at runtime to enable binlog" [15:01:55] let me be 100% sure [15:02:15] I will restart db1139- I trust volans, but it seems so silly I have to check it by myself [15:02:24] the compilation options haven't changed at all from my side [15:02:28] yeah [15:02:40] I didn't know about this, did you, marostegui? [15:02:55] the on and of -log? [15:03:00] I didn't [15:03:34] I had seen the -log, but I thought it was based on the "compile with binlog enabled by default" [15:04:48] yeah same here [15:04:55] are testing on db1139? [15:04:58] I think it was becase I don't normally handle hosts without bilong [15:05:01] marostegui: yes [15:05:06] about to downtime and restart [15:05:27] volans: do you want to be a DBA? [15:05:29] ok [15:05:33] he is at heart [15:05:50] volans: you know more than at least me about databases [15:06:27] marostegui: is it log_bin, log_bin = 1 ? [15:06:56] uh? [15:07:08] the option to enable, I never remember [15:07:11] bin_log ? [15:07:30] ah it is log-bin [15:07:41] ahahahaha [15:07:57] volans: I am not being sarcastic this time [15:08:03] but it is log-bin=path [15:08:05] first time I learn about this [15:08:13] now you made me doubt [15:08:25] (not the log bin, the version string changing at runtime) [15:08:36] log_bin = /var/log/mysql/mysql-bin [15:08:50] I will leave it on the default [15:10:19] new to me too [15:10:23] indeed it is how volans says [15:10:24] I just found the page and read it ;) [15:10:32] it's a non-sense [15:10:35] I totally agree [15:11:12] now I understand why version is a variable, not a status value [15:11:54] we should store a private key on the version, what could possible go wrong! [15:12:36] hahaha [15:12:59] rotfl [15:13:15] you can also fake it [15:13:15] https://mariadb.com/kb/en/server-system-variables/#version [15:13:47] Dynamic: No, so not on "real runtime" [15:13:58] only on start [15:14:11] that is why I propose to store the entire database contents there! [15:14:22] ahahah [15:14:40] mysqld --version=$(mysqldump --all-databases) [15:14:51] for easy access [15:16:00] ok, so actionables on my side [15:16:11] 1) restart db1139 back without binlog [15:16:28] 2) fake the version with the current backup package to remove the -log part [15:16:39] 3) prepare the backups again so the finish [15:17:06] 4) update the automation to ignore - just -log?, anything after the last -? [15:17:21] just the 4 default variants? [15:17:44] ^will create a ticket about this, as this would limit any improtu backups/transfers [15:20:14] 5) file a bug upstream to make version() accept a bolean parameter to get just the actual version without additional stuff :D [15:20:33] nah this is so crazy that is "intended behaviour" [15:20:54] I am trying to compare what other vendors use [15:21:00] surely doesn't happen by chance [15:21:32] e.g. if percona does subversioning, like 8.0.31-percona or something [15:22:47] as the idea of that part of the check was to not do cross-vendor prepare [15:23:17] marostegui: in theory you should be able to reimage db1128, but I would wait until tomorrow for confirmation [15:23:49] jynus: I can wait till Monday even [15:24:05] percoan doess indeed subversioning: 8.0.23-14 [15:25:31] not sure if to ignore everything after the dash, or just the few default ones, I am unsure about xtrabackup compatibility between same upstream version [15:25:54] I think for now I will just ignore the 4 on the mariadb documentation, just to be safe [15:26:07] and like this, better error out [15:26:32] as in the worse case scenario- data has not been lost, it is kept [15:30:09] to add more confusion- tool_version = 10.4.22-MariaDB [15:30:14] ibbackup_version = 10.4.22-MariaDB [15:30:20] server_version = 10.4.22-MariaDB-log [15:30:22] haha [15:30:53] we will call this feature "dynamic versioning" [15:32:35] * jynus executes "man remote_backups.cnf" on cumin1001 [15:38:52] for reference, I updated the xtrabackup_info server version to remove the -log, and rerun the backup with --only-postprocess option (which assumes the transference is done and only does the prepare + compression)