[01:08:31] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2132 is CRITICAL: 7.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2132&var-port=9104
[01:08:37] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 11.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[01:09:25] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 10 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:10:05] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2132 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2132&var-port=9104
[01:10:57] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:11:43] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[16:55:58] <marostegui>	 jynus: db1117 can have mariadb restarted today?
[16:56:06] <marostegui>	 actually only db1117:3321
[16:56:14] <jynus>	 sorry remindme what that is?
[16:56:19] <jynus>	 m1 replica?
[16:56:22] <marostegui>	 yep
[16:56:39] <jynus>	 let me double check the backups, muy most likely right away
[16:56:50] <marostegui>	 thanks, no rush, going into a meeting now
[16:57:35] <jynus>	 marostegui: I confirm it, all m* backups finished correctly can be rebooted at any time
[16:57:42] <marostegui>	 cool, thank you!
[16:57:52] <jynus>	 only ongoing stuff is es4 and es5 on both dcs
[18:46:32] <jinxer-wm>	 (MysqlReplicationLag) firing: MySQL instance db1107:9104 has too large replication lag (1h 11m 30s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1107&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[19:11:32] <jinxer-wm>	 (MysqlReplicationLag) resolved: MySQL instance db1107:9104 has too large replication lag (6m 30s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1107&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[19:20:03] <marostegui>	 ^ me