[01:08:30] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 6.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:09:06] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 11.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[01:10:00] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:10:34] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[15:56:33] * jynus is going to restore his camera setup to be used as webcam after failing to photograph Comet C/2022 E3 (ZTF)
[21:11:34] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s4 on db1143 is CRITICAL: 746 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104
[21:13:37] <jinxer-wm>	 (MysqlReplicationLag) firing: MySQL instance db1143:9104 has too large replication lag (13m 26s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1143&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:23:32] <jinxer-wm>	 (MysqlReplicationLag) resolved: MySQL instance db1143:9104 has too large replication lag (19m 26s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1143&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:24:32] <jinxer-wm>	 (MysqlReplicationLag) firing: MySQL instance db1143:9104 has too large replication lag (7m 5s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1143&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:28:47] <jinxer-wm>	 (MysqlReplicationLag) resolved: MySQL instance db1143:9104 has too large replication lag (7m 5s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1143&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:57:45] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s4 on db1143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104