[09:47:45] <icinga-wm_>	 PROBLEM - MariaDB sustained replica lag on s7 on db1170 is CRITICAL: 16.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1170&var-port=9104
[09:47:45] <icinga-wm_>	 PROBLEM - MariaDB sustained replica lag on s7 on db1227 is CRITICAL: 11.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1227&var-port=9104
[09:47:59] <icinga-wm_>	 PROBLEM - MariaDB sustained replica lag on s7 on db2122 is CRITICAL: 23.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2122&var-port=9104
[09:47:59] <icinga-wm_>	 PROBLEM - MariaDB sustained replica lag on s7 on db2168 is CRITICAL: 28.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104
[09:49:45] <icinga-wm_>	 RECOVERY - MariaDB sustained replica lag on s7 on db1227 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1227&var-port=9104
[09:49:59] <icinga-wm_>	 RECOVERY - MariaDB sustained replica lag on s7 on db2122 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2122&var-port=9104
[09:49:59] <icinga-wm_>	 RECOVERY - MariaDB sustained replica lag on s7 on db2168 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104
[09:50:47] <icinga-wm_>	 RECOVERY - MariaDB sustained replica lag on s7 on db1170 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1170&var-port=9104
[10:21:25] <jinxer-wm>	 (SystemdUnitFailed) firing: pt-heartbeat-wikimedia.service on db2118:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[23:11:45] <icinga-wm_>	 PROBLEM - MariaDB sustained replica lag on s8 on db1154 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13318
[23:12:47] <icinga-wm_>	 RECOVERY - MariaDB sustained replica lag on s8 on db1154 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13318