[11:37:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1117:13323) - TODO - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed
[13:20:32] <Emperor>	 Le sigh, thanos-be2004 has spat a disk :(
[15:37:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1117:13323) - TODO - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed
[19:37:31] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1117:13323) - TODO - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed
[21:04:32] <jinxer-wm>	 (MysqlReplicationLag) firing: MySQL instance db2141:13311 has too large replication lag (1h 43m 44s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2141&var-port=13311 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:09:32] <jinxer-wm>	 (MysqlReplicationLag) firing: (2) MySQL instance db1139:13311 has too large replication lag (16m 54s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:14:32] <jinxer-wm>	 (MysqlReplicationLag) resolved: (2) MySQL instance db1139:13311 has too large replication lag (16m 54s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:27:02] <jinxer-wm>	 (MysqlReplicationLag) firing: (4) MySQL instance db1139:13311 has too large replication lag (16m 54s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:32:02] <jinxer-wm>	 (MysqlReplicationLag) firing: (2) MySQL instance db1150:13314 has too large replication lag (12m 23s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:37:02] <jinxer-wm>	 (MysqlReplicationLag) resolved: (2) MySQL instance db1150:13314 has too large replication lag (6m 41s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:47:17] <jinxer-wm>	 (MysqlReplicationLag) firing: (3) MySQL instance db1150:13314 has too large replication lag (6m 41s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:52:02] <jinxer-wm>	 (MysqlReplicationLag) firing: (3) MySQL instance db1171:13318 has too large replication lag (33m 24s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[21:52:17] <jinxer-wm>	 (MysqlReplicationLag) resolved: (2) MySQL instance db1171:13318 has too large replication lag (33m 24s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[23:37:31] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1117:13323) - TODO - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed