[08:57:33] PROBLEM - MariaDB sustained replica lag on s4 on db2136 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2136&var-port=9104 [08:58:01] PROBLEM - MariaDB sustained replica lag on s4 on db2138 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2138&var-port=13314 [08:58:47] RECOVERY - MariaDB sustained replica lag on s4 on db2136 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2136&var-port=9104 [08:59:17] RECOVERY - MariaDB sustained replica lag on s4 on db2138 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2138&var-port=13314 [10:54:38] PROBLEM - MariaDB sustained replica lag on s4 on db2138 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2138&var-port=13314 [10:55:28] PROBLEM - MariaDB sustained replica lag on s4 on db2073 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2073&var-port=9104 [10:55:28] PROBLEM - MariaDB sustained replica lag on s4 on db2137 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2137&var-port=13314 [10:56:30] RECOVERY - MariaDB sustained replica lag on s4 on db2073 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2073&var-port=9104 [10:56:32] RECOVERY - MariaDB sustained replica lag on s4 on db2137 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2137&var-port=13314 [10:56:42] RECOVERY - MariaDB sustained replica lag on s4 on db2138 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2138&var-port=13314 [11:01:10] is there any on-going maintenance on s4? [11:01:28] hum. that's all codfw. do we really care about getting these alerts.. [11:02:12] I am not doing anything on s4 [11:02:26] * kormat watches everyone turn to look at Amir1 [11:06:52] MetBot is uploading 40 files per minute, could be related [13:02:15] I have the actor migration but that's really slow