[14:13:59] clouddb1014 replication issues? [14:22:34] checking [14:23:22] ` Last_SQL_Error: Could not execute Write_rows_v1 event on table nlwiki.recentchanges; Index for table 'recentchanges' is corrupt; try to repair it, Error_code: 1034; handler error HA_ERR_CRASHED; the event's master log db1155-bin.001893, end_log_pos 431898912` [14:23:41] seems like there was an issue in s2 instance [14:23:48] should I create a ticket for this? [14:25:50] Yeah, best to track it in a task [14:27:23] PROBLEM - MariaDB sustained replica lag on s6 on db2158 is CRITICAL: 29.75 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2158&var-port=9104 [14:29:28] RECOVERY - MariaDB sustained replica lag on s6 on db2158 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2158&var-port=9104 [14:31:21] I have fixed it for now, we'll see if it comes back [14:31:41] I'll create a task if it does [14:31:47] thanks! [16:07:54] PROBLEM - MariaDB sustained replica lag on s6 on db2124 is CRITICAL: 109.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2124&var-port=9104 [16:09:56] RECOVERY - MariaDB sustained replica lag on s6 on db2124 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2124&var-port=9104 [18:00:48] anything I got wrong?