[01:09:23] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1217 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321
[01:09:52] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 9.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:11:00] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1217 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321
[01:11:30] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[08:41:38] <jynus>	 I am doing some maintenance on backup1 sections, may generate lag
[08:52:58] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on backup1-eqiad on db1205 is CRITICAL: 129.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1205&var-port=9104
[08:55:54] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on backup1-eqiad on db1205 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1205&var-port=9104
[09:02:21] <jynus>	 amir and manuel, check these query optimizer decisions: https://phabricator.wikimedia.org/P47285 🤦
[09:52:25] <Amir1>	 jynus: MariaDB is full of surprises 
[09:52:45] <Amir1>	 I dislike unions and I don't even know why
[09:53:01] <Amir1>	 jeez, I mean mysql unions, not any other type
[09:53:32] <jynus>	 in this case the union fixes the issue
[09:53:45] <jynus>	 it is the IN where mariadb struggles here
[10:34:05] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on backup1-eqiad on db1205 is CRITICAL: 79.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1205&var-port=9104
[10:34:24] <jynus>	 that's me
[10:36:40] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on backup1-eqiad on db1205 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1205&var-port=9104
[10:44:39] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on backup1-eqiad on db1205 is CRITICAL: 112.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1205&var-port=9104
[10:47:33] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on backup1-eqiad on db1205 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1205&var-port=9104