[01:09:41] <icinga-wm_>	 PROBLEM - MariaDB sustained replica lag on m1 on db1217 is CRITICAL: 12.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321
[01:10:09] <icinga-wm_>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 10.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:11:19] <icinga-wm_>	 RECOVERY - MariaDB sustained replica lag on m1 on db1217 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321
[01:11:47] <icinga-wm_>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[08:33:18] <Emperor>	 sigh, this morning rclone sync failed because we lost a race with someone deleting objects
[08:34:01] <Emperor>	 https://commons.wikimedia.org/wiki/File:Jake_from_ENHYPEN.jpg was deleted at 03:27, and rclone got to trying to sync it across at 03:28:12 
[10:06:21] <Amir1>	 :D
[15:14:24] <dhinus>	 o/ I tried creating an index on the huge linkwatcher db (T328691) and it failed with "The total number of locks exceeds the lock table size". a quick search suggests I should increase innodb_buffer_pool_size, but also suggests I should not increase it to more than the available RAM, which is not enough... suggestions?
[15:14:25] <stashbot>	 T328691: [toolsdb] Migrate linkwatcher db to Trove - https://phabricator.wikimedia.org/T328691