[01:08:43] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 15.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:09:51] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 14 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[01:11:29] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[01:11:57] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[07:17:30] <marostegui>	 db1125 (test host) got rebooted
[07:17:42] <marostegui>	 I am not creating a task since it is a test host and will be decommissioned anyways
[09:09:32] <jinxer-wm>	 (MysqlReplicationLag) firing: (2) MySQL instance db1196:9104 has too large replication lag (1h 9m 30s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[09:14:32] <jinxer-wm>	 (MysqlReplicationLag) firing: (3) MySQL instance db1154:13311 has too large replication lag (59m 8s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[09:15:45] <marostegui>	 ^that is all me
[09:19:32] <jinxer-wm>	 (MysqlReplicationLag) firing: (3) MySQL instance db1154:13311 has too large replication lag (31m 18s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[09:23:40] <marostegui>	 iasaw
[09:23:43] <marostegui>	 ups
[09:23:47] <marostegui>	 the cat
[09:24:32] <jinxer-wm>	 (MysqlReplicationLag) resolved: (3) MySQL instance db1154:13311 has too large replication lag (20m 9s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[09:26:31] <jynus>	 "iasaw" I think she may be comunicating murder attempts?
[09:28:30] <marostegui>	 hahahaha
[09:34:47] <jinxer-wm>	 (MysqlReplicationLag) firing: (4) MySQL instance db1166:9104 has too large replication lag (49m 58s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[09:35:46] <marostegui>	 ^ me
[09:39:47] <jinxer-wm>	 (MysqlReplicationLag) resolved: (3) MySQL instance db1166:9104 has too large replication lag (21m 16s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[10:05:05] <jynus>	 db2139 and db1145 with replication stopped is me testing the new host for backups
[10:05:25] <marostegui>	 cool
[10:07:53] <jynus>	 I wonder if I should run the es bacula now early or wait for tomorrow
[10:08:13] <jynus>	 as it takes 17 hours
[10:09:19] <jynus>	 I think I will wait
[10:09:27] <marostegui>	 yeah, maybe tomorrow is safer
[10:10:16] <jynus>	 I will wait for disabling the jobs, as in case bacula restarts, the disabling gets lost
[10:13:06] <jynus>	 I will also now start the es2020 data check
[10:13:21] <marostegui>	 nice, let me know if you need help
[10:40:00] <jynus>	 I will leave compare running on the last 4 million rows of each table on cumin2002
[11:15:39] <marostegui>	 I have started a new compare on db1198, the last time, before the DIMM was changed, it made the host crash. So I have started it again to see if it crashes again
[16:00:07] <jynus>	 taking care of the bacula unscheduling now
[16:10:45] <jynus>	 should be good now
[16:11:10] <jynus>	 what time in the end for tomorrow? marostegui after the x1 thingy?
[16:12:27] <marostegui>	 whenever you feel comfortable with it jynus
[16:12:41] <marostegui>	 I don't mind
[16:13:10] <jynus>	 ok, then I predict around 8 UTC should be ok
[16:15:45] <jynus>	 to finish the day in a positive way, things looking good for the dbprov expansion so far: https://phabricator.wikimedia.org/T327155#8557575
[16:16:58] <marostegui>	 good!