[01:07:30] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 3.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [01:07:58] PROBLEM - MariaDB sustained replica lag on m1 on db1217 is CRITICAL: 14.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321 [01:13:26] RECOVERY - MariaDB sustained replica lag on m1 on db1217 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321 [01:15:42] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [06:55:13] https://phabricator.wikimedia.org/P52457 [09:22:28] jynus: yeah, that was me. I moved old content to ES. Most content of wikivoyage ones have been dumped from wikitravel with running moveToExternal [09:22:46] nice work, Amir1! [09:23:03] backup time got reduced almost to half of it [09:23:12] niice [09:24:46] (I ran it on all wikis, dewikivoyage one was also somewhat big but basically every wiki had something to clean up) [09:30:33] We should rotate the ES cluster numbers to cluster28 and cluster29, I haven't forgotten about it, just so much work [09:41:23] I know, that is why I didn't ask to do it :-D, just suggesting to do it for next cluster