[01:11:00] PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 18 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [01:12:54] RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [07:31:32] (MysqlReplicationLag) firing: MySQL instance db1106:9104 has too large replication lag (1h 9m 55s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1106&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [07:31:50] ^ me [07:46:32] (MysqlReplicationLag) resolved: MySQL instance db1106:9104 has too large replication lag (6m 20s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1106&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag