[01:06:35] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 15.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [01:07:27] PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 18.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [01:11:21] RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [01:11:41] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [13:05:22] Emperor: can you confirm swift and swift-ro can be switched over as part of the service switchover on the 28th? They were excluded from the switchdc.services cookbook because of a rebalance https://phabricator.wikimedia.org/T287539#7339799 but afaik there's no reason to keep them out. [13:06:56] claime: he is off today [13:07:12] jynus: ah, thanks [13:07:29] it can wait until monday anyways [13:08:00] Ah he's off until later than that, mb [13:08:19] I'll find someone :D [15:25:35] media backups is currently running; it has to download ~10 million files, and update the status of a bunch more [15:26:20] running things single threaded ATM to keep track of errors/log while I monitor the process [15:34:54] I am happy because I was able to recognize file renames succesfully and archive old metadata- however, if there is garbage on the database, I still output garbage (production would need lots of cleanup) [17:36:24] \o/