[09:31:47] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s4 on db2110 is CRITICAL: 6.473e+04 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2110&var-port=9104
[09:35:32] <jinxer-wm>	 (MysqlReplicationLag) firing: MySQL instance db2110:9104 has too large replication lag (16h 58m 50s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2110&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[09:46:10] <jynus>	 Amir1: that is under maintenance/newly setup, right? ^
[09:51:16] <Amir1>	 under maint
[09:51:17] <Amir1>	 sigh
[09:51:27] <jynus>	 ok, no issue then
[09:51:37] <Amir1>	 the schema change finished well, it's quite slow to catch up though 
[09:51:52] <Amir1>	 that's why it expired the downtime
[11:20:32] <jinxer-wm>	 (MysqlReplicationLag) resolved: MySQL instance db2110:9104 has too large replication lag (6m 31s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2110&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[11:24:23] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s4 on db2110 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2110&var-port=9104
[15:09:53] <jynus>	 backups worked well, but will will need a lot of more work for T319383 this quarter
[15:09:54] <stashbot>	 T319383: Mydumper incompatibility with MariaDB 10.6 (was: Logical recoveries (myloader) to db2098:s7 are failing with "Lock wait timeout exceeded; try restarting transaction") - https://phabricator.wikimedia.org/T319383