[01:12:03] PROBLEM - MariaDB sustained replica lag on m1 on db1217 is CRITICAL: 17.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321 [01:13:29] RECOVERY - MariaDB sustained replica lag on m1 on db1217 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321 [01:14:37] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 3.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [01:16:05] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [13:39:17] marostegui: sorry, I think I might need your help here, I'm a bit lost. db-compare is reporting difference and possible data corruption on content_models table in most of s7 dbs, the thing is that I don't see a difference [13:39:20] run [13:39:21] sudo db-compare bgwiki content_models model_id db2175.codfw.wmnet:3306 db2104 [13:39:45] the table is small, so you can fully query the whole thing and check it yourself [13:39:57] https://www.irccloud.com/pastebin/dtp3yCKV/ [13:40:05] https://www.irccloud.com/pastebin/8SYduuSc/ [13:40:47] my guess is padded space in the text? [13:40:54] probably [13:40:55] let me check [13:43:08] hex also looks the same [13:43:11] https://www.irccloud.com/pastebin/lWFbzE5P/ [13:43:36] is that table written a lot? [13:44:48] those hosts have different mariadb versions [13:46:13] -INSERT INTO `content_models` VALUES (1,'wikitext'),(2,'javascript'),(3,'css'),(4,'Scribunto'),(5,'MassMessageListContent'),(6,'sanitized-css'),(7,'json'); [13:46:13] +INSERT INTO `content_models` VALUES (5,'MassMessageListContent'),(4,'Scribunto'),(3,'css'),(2,'javascript'),(7,'json'),(6,'sanitized-css'),(1,'wikitext'); [13:46:27] interesting how mysqldump shows them [13:46:43] not much, but read a lot [13:47:23] the query by the db-compare is this [13:47:24] SELECT crc32(GROUP_CONCAT(IFNULL(model_id, '\0'),'|',IFNULL(model_name, '\0'),'|')) FROM content_models WHERE model_id BETWEEN 1 AND 7 ORDER BY model_id [13:47:37] since orders the model_id, it should be fine to have a different order [13:50:28] Amir1: for compare db to work well [13:50:35] it is something related to the different versio [13:50:40] sometimes is needed a separate order different from the pk [13:50:55] I was about to say that [13:50:59] your issue is that the compare you ran returns data in different order [13:51:09] https://phabricator.wikimedia.org/P52240 [13:51:10] so in some cases you need to extra --order-by clause [13:51:20] additional to the PK [13:51:33] so you put there the PK and adiitional columns to order by [13:51:55] ping me if need more context, I am busy with something else atm [13:52:44] I'm not sure the --order-by clause is at fault here beside auto increment PK [13:53:08] The rows are clearly different from mysql storage point of view as you can see above [13:53:13] the tables return the exact same result if you put order by model_id there [13:53:16] And it is definitely related to the version [13:53:23] yeah, it's version [13:53:24] they are stored in a different way [13:54:06] Keep in mind that the innodb version is different between 10.4 and 10.6 [13:54:23] yeah, I think that's causing issues [13:54:48] maybe one of the checks should be to compare same mariadb versions [13:54:52] when picking hosts [13:55:24] yeah [13:55:55] weirdly it only happens in content_models table in all s7 wikis but not other tables [13:56:19] anyway, if the data is the same, I don't care. I try to find a way to compare only inside one version [13:56:45] yeah the data is the same [13:57:01] that table has a PK and a varbinary too [13:57:05] only those two