[07:38:32] (MysqlReplicationLag) firing: MySQL instance db1198:9104 has too large replication lag (1h 28m 55s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1198&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [07:38:38] ^ me [07:48:32] (MysqlReplicationLag) resolved: MySQL instance db1198:9104 has too large replication lag (10m 15s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1198&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [10:01:32] (MysqlReplicationLag) firing: MySQL instance db2180:9104 has too large replication lag (1h 48m 2s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2180&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [10:03:20] ^ me [10:06:32] (MysqlReplicationLag) resolved: MySQL instance db2180:9104 has too large replication lag (22m 49s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2180&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [21:02:32] (MysqlReplicationLag) firing: MySQL instance db1139:13311 has too large replication lag (1h 24m 11s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1139&var-port=13311 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [21:07:32] (MysqlReplicationLag) firing: (2) MySQL instance db1139:13311 has too large replication lag (24m 25s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [21:12:32] (MysqlReplicationLag) resolved: (2) MySQL instance db1139:13311 has too large replication lag (8m 27s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [21:23:02] (MysqlReplicationLag) firing: (3) MySQL instance db1139:13311 has too large replication lag (8m 27s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [21:28:02] (MysqlReplicationLag) firing: (2) MySQL instance db1150:13314 has too large replication lag (12m 35s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [21:33:02] (MysqlReplicationLag) resolved: (2) MySQL instance db1150:13314 has too large replication lag (12m 35s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [21:45:32] (MysqlReplicationLag) firing: MySQL instance db1171:13318 has too large replication lag (1h 55m 29s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1171&var-port=13318 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [21:50:32] (MysqlReplicationLag) resolved: MySQL instance db1171:13318 has too large replication lag (29m 1s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1171&var-port=13318 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [21:53:02] (MysqlReplicationLag) firing: (2) MySQL instance db1171:13318 has too large replication lag (11m 23s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [21:53:17] (MysqlReplicationLag) resolved: MySQL instance db1171:13318 has too large replication lag (11m 23s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1171&var-port=13318 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag