[08:30:07] I have fixed test-s4 cluster, so it is now showing the correct topology in orchestrator after Arnaud's fixes <3 [12:11:15] PROBLEM - MariaDB sustained replica lag on s1 on db2146 is CRITICAL: 202.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2146&var-port=9104 [12:12:13] uh [12:12:45] ^ Amir1 arnaudb expected? [12:13:49] seems to be pooled [12:14:02] at meeting [12:14:08] depool it please [12:14:18] investigating [12:14:21] depooling it then [12:16:18] done [12:16:51] I will downtime it [12:26:36] created a task, even if it is a non-issue as a reminder to repool it: https://phabricator.wikimedia.org/T355313 [12:27:49] RECOVERY - MariaDB sustained replica lag on s1 on db2146 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2146&var-port=9104 [12:41:19] thanks jynus will check! [12:51:02] s4 eqiad snapshot fresh 8 hours ago 1.7 TB -4.3 % — [12:51:13] First of two drops [12:55:33] (90GB)