[09:58:50] it was good to wait so to have a fresh backup of al misc hosts [10:16:49] marostegui: which test hosts were you using? I would like to do a recovery test on either dc [10:19:58] you can pick any [10:20:07] 1124, 1125 and 1133 [10:20:17] I'm on clinic duty, so not using them today [10:21:04] ok, I will just do a quick restore, verify things are going well then just remove its data [10:21:25] I will pick db1133 [11:34:51] done, I also deleted the recovered db there [11:37:06] thanks :) [11:48:59] Deleted scholarships database from m2 [18:03:54] PROBLEM - MariaDB sustained replica lag on s1 on db2146 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2146&var-port=9104 [18:05:00] RECOVERY - MariaDB sustained replica lag on s1 on db2146 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2146&var-port=9104 [18:13:08] PROBLEM - MariaDB sustained replica lag on s1 on db1132 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1132&var-port=9104 [18:14:36] RECOVERY - MariaDB sustained replica lag on s1 on db1132 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1132&var-port=9104