[00:55:05] <jinxer-wm>	 FIRING: [6x] MysqlReplicationThreadCountTooLow: MySQL instance db1150:13314 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow
[02:00:05] <jinxer-wm>	 FIRING: [6x] MysqlReplicationThreadCountTooLow: MySQL instance db1150:13314 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow
[03:15:05] <jinxer-wm>	 FIRING: [4x] MysqlReplicationThreadCountTooLow: MySQL instance db1150:13314 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow
[03:20:05] <jinxer-wm>	 FIRING: [4x] MysqlReplicationThreadCountTooLow: MySQL instance db1150:13314 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow
[03:25:05] <jinxer-wm>	 FIRING: [4x] MysqlReplicationThreadCountTooLow: MySQL instance db1150:13314 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow
[03:35:05] <jinxer-wm>	 RESOLVED: [3x] MysqlReplicationThreadCountTooLow: MySQL instance db1150:13314 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow
[05:20:05] <jinxer-wm>	 FIRING: MysqlReplicationThreadCountTooLow: MySQL instance db1150:13313 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1150&var-port=13313 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow
[05:30:05] <jinxer-wm>	 FIRING: [2x] MysqlReplicationThreadCountTooLow: MySQL instance db1150:13313 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow
[06:50:05] <jinxer-wm>	 FIRING: [2x] MysqlReplicationThreadCountTooLow: MySQL instance db1150:13313 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow
[06:55:05] <jinxer-wm>	 RESOLVED: MysqlReplicationThreadCountTooLow: MySQL instance db1150:13313 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1150&var-port=13313 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow
[07:27:41] <Emperor>	 ^-- are those known about?
[07:27:46] <Emperor>	 I mean, the cause thereof
[07:43:32] <arnaudb>	 there is a known issue on dumps that cause replication to suffer a bit T368098
[07:43:33] <stashbot>	 T368098: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098
[07:47:48] <marostegui>	 arnaudb: those hosts aren't s1
[07:50:43] <arnaudb>	 I infered that the problem was a bit wider indeed, it was my conclusion after checking that the hosts were in fact being backuped which seemed a legit cause
[07:50:53] <arnaudb>	 (and they recovered just a few moments after)
[07:51:03] <marostegui>	 is that part of the new prometheus alert?
[07:51:12] <marostegui>	 because we don't get alerted for those
[07:51:20] <arnaudb>	 good question, let me check
[07:51:25] <marostegui>	 maybe they need to be silenced
[07:51:36] <arnaudb>	 or threshold to be moved/exclusion to be made for backup hosts
[07:51:43] <arnaudb>	 checking
[07:53:47] <arnaudb>	 confirmed, will open a task to properly handle this
[07:57:46] <arnaudb>	 T372991
[07:57:52] <stashbot>	 T372991: mariadb - monitoring - MysqlReplicationThreadCountTooLow fix - https://phabricator.wikimedia.org/T372991