[03:12:32] (MysqlReplicationLag) firing: MySQL instance db1143:9104 has too large replication lag (11m 19s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1143&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [03:15:21] PROBLEM - MariaDB sustained replica lag on s4 on db1143 is CRITICAL: 709 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104 [12:13:12] <_joe_> db1143 is recovering replication after the "cure" (kill -9) [12:18:01] so far everything looks good [12:18:19] it's recovering quickly, I go rest until it recovers [15:14:50] <_joe_> Amir1: the replication should have caught up; I'd repool this server on monday though [15:15:00] <_joe_> I am going off for now