[03:05:45] PROBLEM - MariaDB sustained replica lag on s3 on db1154 is CRITICAL: 98 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13313 [03:08:07] RECOVERY - MariaDB sustained replica lag on s3 on db1154 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13313 [06:20:56] jynus: morning, for when you are online, can you let me know when I could reboot db1117 and db1116? thanks! [10:30:54] marostegui: if today, any time between now and 0UTC [11:53:55] sorry jynus I was busy with one issue [11:53:58] I will do it now thanks [13:29:32] (MysqlReplicationLag) firing: MySQL instance db1189:9104 has too large replication lag (7d 19h 8m 10s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1189&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [13:31:07] ^expected [17:29:47] (MysqlReplicationLag) firing: MySQL instance db1189:9104 has too large replication lag (2d 10h 17m 25s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1189&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [19:34:32] (MysqlReplicationLag) resolved: MySQL instance db1189:9104 has too large replication lag (23m 24s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1189&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag