[11:24:33] afk for a bit [12:38:56] PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [12:41:00] RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [15:09:07] marostegui: the broken rows for flaggedtemplates in dewiki are 779,171 [15:09:18] I haven't checked arwiki and co [17:30:27] ACKNOWLEDGEMENT - MariaDB sustained replica lag on s3 on db2074 is CRITICAL: 1.221e+04 ge 2 Kormat Catching up on replication https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2074&var-port=9104 [17:33:24] ACKNOWLEDGEMENT - MariaDB sustained replica lag on s1 on db2130 is CRITICAL: 1.291e+04 ge 2 Kormat Catching up on replication https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2130&var-port=9104 [17:35:32] marostegui: i've acked the icinga checks for replication lag for the machines which are still catching up, and removed all the downtimes [17:35:38] the only remaining thing is db2078 [17:36:03] at this hour i'm going to leave it downtimed, and will bring it back up tomorrow [17:36:45] thank you kormat [21:54:33] Amir1: where is your schema drift script [21:55:10] RhinosF1: https://github.com/Ladsgroup/db-analyzor-tools/blob/master/db_drift_checker.py [21:55:31] it will produce a json, the webservice can then understand the json [21:57:49] * RhinosF1 wants to steal some of the logic to try and catch drifts [21:58:03] I got to hope people use abstract schema though