[06:09:33] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 11.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [06:12:33] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 1.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [13:20:40] PROBLEM - MariaDB sustained replica lag on m3 on db2234 is CRITICAL: 13.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2234&var-port=9104 [13:22:40] RECOVERY - MariaDB sustained replica lag on m3 on db2234 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2234&var-port=9104 [14:10:26] PROBLEM - MariaDB sustained replica lag on s2 on db2225 is CRITICAL: 25.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2225&var-port=9104 [14:10:26] PROBLEM - MariaDB sustained replica lag on s2 on db2204 is CRITICAL: 21.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2204&var-port=9104 [14:11:12] PROBLEM - MariaDB sustained replica lag on s2 on db2238 is CRITICAL: 21.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2238&var-port=9104 [14:11:28] PROBLEM - MariaDB sustained replica lag on s2 on db2226 is CRITICAL: 11.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2226&var-port=9104 [14:12:26] RECOVERY - MariaDB sustained replica lag on s2 on db2204 is OK: (C)10 ge (W)5 ge 3.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2204&var-port=9104 [14:13:12] RECOVERY - MariaDB sustained replica lag on s2 on db2238 is OK: (C)10 ge (W)5 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2238&var-port=9104 [14:13:26] RECOVERY - MariaDB sustained replica lag on s2 on db2225 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2225&var-port=9104 [14:13:26] RECOVERY - MariaDB sustained replica lag on s2 on db2226 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2226&var-port=9104 [14:29:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 28.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [14:31:26] PROBLEM - MariaDB sustained replica lag on s3 on db2190 is CRITICAL: 11.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [14:33:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 31.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [14:34:00] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 11.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [14:35:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 20.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [14:35:26] PROBLEM - MariaDB sustained replica lag on s3 on db2190 is CRITICAL: 12.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [14:36:26] RECOVERY - MariaDB sustained replica lag on s3 on db2190 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [14:37:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [14:38:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [14:39:00] RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [14:50:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 18.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [14:53:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 3.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [14:55:26] PROBLEM - MariaDB sustained replica lag on s7 on db2220 is CRITICAL: 77 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104 [14:56:26] PROBLEM - MariaDB sustained replica lag on s7 on db2222 is CRITICAL: 67.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2222&var-port=9104 [14:56:26] PROBLEM - MariaDB sustained replica lag on s7 on db2208 is CRITICAL: 36 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104 [14:57:00] PROBLEM - MariaDB sustained replica lag on s7 on db2159 is CRITICAL: 30.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2159&var-port=9104 [14:59:12] PROBLEM - MariaDB sustained replica lag on s7 on db2221 is CRITICAL: 41.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104 [15:00:28] PROBLEM - MariaDB sustained replica lag on s7 on db2218 is CRITICAL: 73.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2218&var-port=9104 [15:02:48] FIRING: MysqlReplicationLag: MySQL instance db2220:9104@s7 has too large replication lag (4m 33s). Its replication source is db1236.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2220&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [15:02:48] FIRING: [9x] MysqlReplicationLagPtHeartbeat: MySQL instance db2150:9104 has too large replication lag (4m 35s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [15:03:00] RECOVERY - MariaDB sustained replica lag on s7 on db2159 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2159&var-port=9104 [15:04:40] PROBLEM - MariaDB sustained replica lag on s7 on db2182 is CRITICAL: 48.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2182&var-port=9104 [15:05:26] PROBLEM - MariaDB sustained replica lag on s7 on db2150 is CRITICAL: 58.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2150&var-port=9104 [15:07:40] RECOVERY - MariaDB sustained replica lag on s7 on db2182 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2182&var-port=9104 [15:12:32] PROBLEM - MariaDB sustained replica lag on s7 on db2168 is CRITICAL: 87.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104 [15:12:40] PROBLEM - MariaDB sustained replica lag on s7 on db2182 is CRITICAL: 83.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2182&var-port=9104 [15:14:26] RECOVERY - MariaDB sustained replica lag on s7 on db2150 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2150&var-port=9104 [15:15:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 75.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [15:15:40] RECOVERY - MariaDB sustained replica lag on s7 on db2182 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2182&var-port=9104 [15:16:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 21.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [15:16:26] PROBLEM - MariaDB sustained replica lag on s3 on db2149 is CRITICAL: 11.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [15:16:26] PROBLEM - MariaDB sustained replica lag on s3 on db2156 is CRITICAL: 25.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [15:16:30] RECOVERY - MariaDB sustained replica lag on s7 on db2168 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104 [15:16:58] PROBLEM - MariaDB sustained replica lag on s7 on db2159 is CRITICAL: 108.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2159&var-port=9104 [15:17:23] ^-- expected? [15:17:26] PROBLEM - MariaDB sustained replica lag on s3 on db2190 is CRITICAL: 39.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [15:17:26] RECOVERY - MariaDB sustained replica lag on s7 on db2208 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104 [15:18:26] RECOVERY - MariaDB sustained replica lag on s3 on db2149 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [15:20:28] RECOVERY - MariaDB sustained replica lag on s7 on db2222 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2222&var-port=9104 [15:21:26] RECOVERY - MariaDB sustained replica lag on s3 on db2156 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [15:22:26] PROBLEM - MariaDB sustained replica lag on s7 on db2208 is CRITICAL: 199.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104 [15:22:48] FIRING: [10x] MysqlReplicationLagPtHeartbeat: MySQL instance db2150:9104 has too large replication lag (8m 13s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [15:23:12] RECOVERY - MariaDB sustained replica lag on s7 on db2221 is OK: (C)10 ge (W)5 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104 [15:23:58] RECOVERY - MariaDB sustained replica lag on s7 on db2159 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2159&var-port=9104 [15:23:58] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 32.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [15:25:26] PROBLEM - MariaDB sustained replica lag on s7 on db2222 is CRITICAL: 93.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2222&var-port=9104 [15:25:30] PROBLEM - MariaDB sustained replica lag on s7 on db2168 is CRITICAL: 96.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104 [15:25:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 26.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [15:26:26] PROBLEM - MariaDB sustained replica lag on s3 on db2149 is CRITICAL: 13.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [15:26:26] RECOVERY - MariaDB sustained replica lag on s3 on db2190 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [15:26:58] RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [15:27:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [15:27:26] RECOVERY - MariaDB sustained replica lag on s3 on db2149 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [15:27:32] RECOVERY - MariaDB sustained replica lag on s7 on db2168 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104 [15:27:36] RECOVERY - MariaDB sustained replica lag on s3 on db2227 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [15:27:48] FIRING: [17x] MysqlReplicationLagPtHeartbeat: MySQL instance db2149:9104 has too large replication lag (1m 1s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [15:28:12] PROBLEM - MariaDB sustained replica lag on s7 on db2221 is CRITICAL: 182.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104 [15:29:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [15:29:40] PROBLEM - MariaDB sustained replica lag on s7 on db2182 is CRITICAL: 173.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2182&var-port=9104 [15:31:26] RECOVERY - MariaDB sustained replica lag on s7 on db2208 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104 [15:32:30] PROBLEM - MariaDB sustained replica lag on s7 on db2168 is CRITICAL: 244 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104 [15:33:00] PROBLEM - MariaDB sustained replica lag on s7 on db2159 is CRITICAL: 244.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2159&var-port=9104 [15:34:26] PROBLEM - MariaDB sustained replica lag on s7 on db2150 is CRITICAL: 154.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2150&var-port=9104 [15:36:28] PROBLEM - MariaDB sustained replica lag on s7 on db2208 is CRITICAL: 70.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104 [15:38:30] RECOVERY - MariaDB sustained replica lag on s7 on db2168 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104 [15:39:26] RECOVERY - MariaDB sustained replica lag on s7 on db2222 is OK: (C)10 ge (W)5 ge 3.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2222&var-port=9104 [15:39:26] RECOVERY - MariaDB sustained replica lag on s7 on db2150 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2150&var-port=9104 [15:40:28] RECOVERY - MariaDB sustained replica lag on s7 on db2218 is OK: (C)10 ge (W)5 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2218&var-port=9104 [15:41:00] RECOVERY - MariaDB sustained replica lag on s7 on db2159 is OK: (C)10 ge (W)5 ge 1.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2159&var-port=9104 [15:41:40] RECOVERY - MariaDB sustained replica lag on s7 on db2182 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2182&var-port=9104 [15:42:48] RESOLVED: MysqlReplicationLag: MySQL instance db2220:9104@s7 has too large replication lag (2m 41s). Its replication source is db1236.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2220&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [15:42:48] RESOLVED: [17x] MysqlReplicationLagPtHeartbeat: MySQL instance db2149:9104 has too large replication lag (1m 1s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [15:43:12] RECOVERY - MariaDB sustained replica lag on s7 on db2221 is OK: (C)10 ge (W)5 ge 1.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104 [15:43:26] RECOVERY - MariaDB sustained replica lag on s7 on db2220 is OK: (C)10 ge (W)5 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104 [15:43:26] RECOVERY - MariaDB sustained replica lag on s7 on db2208 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104 [16:44:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 19.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [16:47:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 3.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [17:19:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 18.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [17:19:26] PROBLEM - MariaDB sustained replica lag on s3 on db2190 is CRITICAL: 16.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [17:19:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 42 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [17:20:26] PROBLEM - MariaDB sustained replica lag on s3 on db2156 is CRITICAL: 10 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [17:20:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 24.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [17:21:00] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 29 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [17:21:28] RECOVERY - MariaDB sustained replica lag on s3 on db2156 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [17:22:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 4.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [17:22:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 12 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [17:23:58] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 10 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [17:25:00] RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [17:25:26] RECOVERY - MariaDB sustained replica lag on s3 on db2190 is OK: (C)10 ge (W)5 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [17:25:36] RECOVERY - MariaDB sustained replica lag on s3 on db2227 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [17:27:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 2.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [17:41:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 35 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [17:44:00] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 11.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [17:44:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 14 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [17:45:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 1.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [17:45:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 2.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [17:46:00] RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [18:45:11] sigh it's the script [18:46:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 21.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [18:46:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 50.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [18:48:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 22.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [18:48:26] PROBLEM - MariaDB sustained replica lag on s3 on db2190 is CRITICAL: 21.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [18:48:26] PROBLEM - MariaDB sustained replica lag on s3 on db2156 is CRITICAL: 45.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [18:48:36] RECOVERY - MariaDB sustained replica lag on s3 on db2227 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [18:49:26] RECOVERY - MariaDB sustained replica lag on s3 on db2190 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [18:50:26] PROBLEM - MariaDB sustained replica lag on s3 on db2149 is CRITICAL: 12.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [18:51:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [18:52:00] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 27.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [18:52:26] RECOVERY - MariaDB sustained replica lag on s3 on db2149 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [18:53:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 65.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [18:54:30] PROBLEM - MariaDB sustained replica lag on s3 on db2177 is CRITICAL: 15.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [18:56:18] these all should recover soon [18:56:26] there was a --throttle option [18:56:30] RECOVERY - MariaDB sustained replica lag on s3 on db2177 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [18:57:26] RECOVERY - MariaDB sustained replica lag on s3 on db2156 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [18:59:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 37.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [19:00:36] RECOVERY - MariaDB sustained replica lag on s3 on db2227 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [19:00:48] FIRING: MysqlReplicationLag: MySQL instance db2205:9104@s3 has too large replication lag (3m 38s). Its replication source is db1223.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2205&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [19:00:48] FIRING: [8x] MysqlReplicationLagPtHeartbeat: MySQL instance db2149:9104 has too large replication lag (3m 38s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [19:05:26] PROBLEM - MariaDB sustained replica lag on s3 on db2190 is CRITICAL: 43.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [19:05:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 83.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [19:06:30] PROBLEM - MariaDB sustained replica lag on s3 on db2177 is CRITICAL: 76.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [19:08:26] PROBLEM - MariaDB sustained replica lag on s3 on db2149 is CRITICAL: 37.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [19:09:26] RECOVERY - MariaDB sustained replica lag on s3 on db2149 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [19:09:30] RECOVERY - MariaDB sustained replica lag on s3 on db2177 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2177&var-port=9104 [19:10:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [19:12:36] RECOVERY - MariaDB sustained replica lag on s3 on db2227 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [19:14:26] PROBLEM - MariaDB sustained replica lag on s3 on db2156 is CRITICAL: 64.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [19:15:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 63.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [19:16:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [19:17:58] RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [19:20:26] RECOVERY - MariaDB sustained replica lag on s3 on db2190 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [19:20:26] RECOVERY - MariaDB sustained replica lag on s3 on db2156 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2156&var-port=9104 [19:21:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 52.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [19:21:26] PROBLEM - MariaDB sustained replica lag on s3 on db2149 is CRITICAL: 91.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [19:21:54] the script hasn't been writing to s3 for at least twenty minutes now [19:22:10] it can be something else [19:22:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 19.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [19:23:00] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 47 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [19:23:26] RECOVERY - MariaDB sustained replica lag on s3 on db2149 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [19:24:00] RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [19:24:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [19:28:26] PROBLEM - MariaDB sustained replica lag on s3 on db2190 is CRITICAL: 28.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [19:29:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 25.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [19:30:48] RESOLVED: MysqlReplicationLag: MySQL instance db2205:9104@s3 has too large replication lag (1m 8s). Its replication source is db1223.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2205&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [19:30:48] RESOLVED: [8x] MysqlReplicationLagPtHeartbeat: MySQL instance db2149:9104 has too large replication lag (1m 0s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [19:31:58] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 17.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [19:33:36] PROBLEM - MariaDB sustained replica lag on s3 on db2227 is CRITICAL: 13.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [19:34:36] RECOVERY - MariaDB sustained replica lag on s3 on db2227 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2227&var-port=9104 [19:35:00] RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [19:35:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 4.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [19:35:26] RECOVERY - MariaDB sustained replica lag on s3 on db2190 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2190&var-port=9104 [19:36:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 3.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [19:41:06] it should have been fixed for good now. unless --throttle 1 is not enough [22:01:42] --throttle 1 would only be one millisecond [22:02:03] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master/maintenance/recountCategories.php#105 takes the value by 1000 [22:02:08] but usleep takes microseconds [22:31:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 24.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [22:33:12] PROBLEM - MariaDB sustained replica lag on s3 on db2194 is CRITICAL: 12.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [22:34:12] RECOVERY - MariaDB sustained replica lag on s3 on db2194 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2194&var-port=9104 [22:34:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [23:30:36] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 38.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [23:32:00] PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 18.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [23:34:58] RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 4.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104 [23:37:36] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 4.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104