[00:17:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 17.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [00:20:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 2.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [00:26:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 51.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [00:27:26] PROBLEM - MariaDB sustained replica lag on s3 on db2156 is CRITICAL: 15.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [00:28:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 14.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [00:28:26] PROBLEM - MariaDB sustained replica lag on s3 on db2149 is CRITICAL: 16 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [00:28:58] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 26.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [00:29:26] RECOVERY - MariaDB sustained replica lag on s3 on db2149 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [00:29:26] RECOVERY - MariaDB sustained replica lag on s3 on db2156 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [00:30:00] RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [00:30:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [00:30:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [01:32:26] PROBLEM - MariaDB sustained replica lag on s3 on db2190 is CRITICAL: 18.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [01:32:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 43.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [01:33:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 19.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [01:34:32] PROBLEM - MariaDB sustained replica lag on s3 on db2177 is CRITICAL: 20.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [01:35:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 14 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [01:36:36] RECOVERY - MariaDB sustained replica lag on s3 on db2227 is OK: (C)10 ge (W)5 ge 4.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [01:37:32] RECOVERY - MariaDB sustained replica lag on s3 on db2177 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [01:38:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [01:38:26] RECOVERY - MariaDB sustained replica lag on s3 on db2190 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [01:39:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 1.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [02:20:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 71.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [02:21:26] PROBLEM - MariaDB sustained replica lag on s3 on db2190 is CRITICAL: 42.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [02:22:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 23.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [02:23:00] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 38 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [02:24:36] RECOVERY - MariaDB sustained replica lag on s3 on db2227 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [02:25:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 44.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [02:25:26] PROBLEM - MariaDB sustained replica lag on s3 on db2149 is CRITICAL: 22.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [02:28:00] RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [02:28:48] FIRING: MysqlReplicationLag: MySQL instance db2205:9104@s3 has too large replication lag (1m 48s). Its replication source is db1223.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2205&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [02:28:48] FIRING: [8x] MysqlReplicationLagPtHeartbeat: MySQL instance db2149:9104 has too large replication lag (1m 48s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [02:31:32] PROBLEM - MariaDB sustained replica lag on s3 on db2177 is CRITICAL: 29.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [02:35:32] RECOVERY - MariaDB sustained replica lag on s3 on db2177 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [02:37:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 99.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [02:38:26] RECOVERY - MariaDB sustained replica lag on s3 on db2149 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [02:41:26] RECOVERY - MariaDB sustained replica lag on s3 on db2190 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [02:43:00] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 57 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [02:46:26] PROBLEM - MariaDB sustained replica lag on s3 on db2190 is CRITICAL: 145.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [02:47:30] PROBLEM - MariaDB sustained replica lag on s3 on db2177 is CRITICAL: 151.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [02:48:28] PROBLEM - MariaDB sustained replica lag on s3 on db2156 is CRITICAL: 77.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [02:49:32] RECOVERY - MariaDB sustained replica lag on s3 on db2177 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [02:50:26] PROBLEM - MariaDB sustained replica lag on s3 on db2149 is CRITICAL: 83.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [02:50:28] RECOVERY - MariaDB sustained replica lag on s3 on db2156 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [02:53:26] RECOVERY - MariaDB sustained replica lag on s3 on db2149 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [02:53:36] RECOVERY - MariaDB sustained replica lag on s3 on db2227 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [02:56:26] RECOVERY - MariaDB sustained replica lag on s3 on db2190 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [02:58:00] RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [02:58:32] PROBLEM - MariaDB sustained replica lag on s3 on db2177 is CRITICAL: 158 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [02:58:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 178.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [02:58:48] FIRING: [8x] MysqlReplicationLagPtHeartbeat: MySQL instance db2149:9104 has too large replication lag (1m 17s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [02:59:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [03:00:36] RECOVERY - MariaDB sustained replica lag on s3 on db2227 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [03:01:26] PROBLEM - MariaDB sustained replica lag on s3 on db2190 is CRITICAL: 86.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [03:01:28] PROBLEM - MariaDB sustained replica lag on s3 on db2156 is CRITICAL: 52 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [03:02:32] RECOVERY - MariaDB sustained replica lag on s3 on db2177 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [03:03:26] RECOVERY - MariaDB sustained replica lag on s3 on db2190 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [03:03:28] RECOVERY - MariaDB sustained replica lag on s3 on db2156 is OK: (C)10 ge (W)5 ge 2.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [03:03:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 1 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [03:03:48] RESOLVED: MysqlReplicationLag: MySQL instance db2205:9104@s3 has too large replication lag (1m 47s). Its replication source is db1223.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2205&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [03:03:48] RESOLVED: [8x] MysqlReplicationLagPtHeartbeat: MySQL instance db2149:9104 has too large replication lag (2m 28s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [09:50:13] ugh [09:50:44] thanks zabe [09:50:50] went with 1000 [22:01:00] PROBLEM - MariaDB sustained replica lag on s2 on db2175 is CRITICAL: 12.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2175&var-port=9104 [22:02:00] RECOVERY - MariaDB sustained replica lag on s2 on db2175 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2175&var-port=9104