[06:43:57] <marostegui>	 I am going to switch es5 codfw master
[08:03:38] <marostegui>	 I had to do something before that (like pooling a different replica), so starting for real now :)
[18:00:37] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104
[18:02:37] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104
[18:08:37] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104
[18:15:37] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s8 on db1214 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104
[19:21:19] <Amir1>	 s3 in the past month: https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=db1157&var-datasource=thanos&var-cluster=mysql&viewPanel=28&from=1712674563020&to=1715196042199
[19:30:37] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104
[19:33:37] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s8 on db1214 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104
[21:00:37] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104
[21:02:37] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s8 on db1214 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104