[05:48:23] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on x1 on db1137 is CRITICAL: 3.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1137&var-port=9104
[05:50:13] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on x1 on db1137 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1137&var-port=9104
[11:51:58] <marostegui>	 I am going to reboot both sanitarium hosts in eqiad
[11:52:26] * jynus is relieved he can remove an old, leftover EFI partition and that means he is saved from reimaging lots of servers after checking those reboot normally without it
[13:28:00] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2122 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2122&var-port=9104
[13:30:08] <jynus>	 expected?
[13:30:51] <jynus>	 seems to be deletions, so probably some job or script
[13:31:46] <jynus>	 https://grafana.wikimedia.org/goto/SbBXw9ZVk?orgId=1
[13:35:04] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2122 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2122&var-port=9104
[13:42:14] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2122 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2122&var-port=9104
[14:15:08] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db1127 is CRITICAL: 2.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1127&var-port=9104
[14:19:56] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db1127 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1127&var-port=9104
[15:44:15] <Amir1>	 sigh
[15:44:26] <Amir1>	 I'm not running anything there