[01:10:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: mariadb.service on db2160:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[05:10:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: mariadb.service on db2160:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:10:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: mariadb.service on db2160:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:10:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: mariadb.service on db2160:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:10:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: mariadb.service on db2160:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[21:10:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: mariadb.service on db2160:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[21:11:48] <jinxer-wm>	 FIRING: MysqlReplicationLag: MySQL instance db2249:9104@x1 has too large replication lag (7m 10s). Its replication source is db2191.codfw.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2249&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:11:48] <jinxer-wm>	 FIRING: MysqlReplicationLagPtHeartbeat: MySQL instance db2249:9104 has too large replication lag (7m 10s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2249&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[21:16:48] <jinxer-wm>	 FIRING: [2x] MysqlReplicationLag: MySQL instance db2191:9104@x1 has too large replication lag (6m 20s). Its replication source is db1220.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:16:48] <jinxer-wm>	 FIRING: [7x] MysqlReplicationLagPtHeartbeat: MySQL instance db2186:9104 has too large replication lag (5m 43s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[21:26:48] <jinxer-wm>	 FIRING: [2x] MysqlReplicationLag: MySQL instance db2191:9104@x1 has too large replication lag (9m 6s). Its replication source is db1220.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:26:48] <jinxer-wm>	 FIRING: [7x] MysqlReplicationLagPtHeartbeat: MySQL instance db2186:9104 has too large replication lag (9m 4s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[21:41:48] <jinxer-wm>	 RESOLVED: [2x] MysqlReplicationLag: MySQL instance db2191:9104@x1 has too large replication lag (4m 37s). Its replication source is db1220.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:41:48] <jinxer-wm>	 RESOLVED: [7x] MysqlReplicationLagPtHeartbeat: MySQL instance db2186:9104 has too large replication lag (9m 4s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat