[15:15:24] afk for a bit [15:47:53] forT355866 db2155 db2156 db2105 db2122 db2133 es2024 are all depooled and downtimed [15:48:42] arnaudb: thanks a lot! [15:49:02] np! [15:50:11] I see we have es2024, es2027 and es2028 in that rack too, have they been depooled? [15:50:45] es2024 has been depooled, es2027 and 2028 are standalone: https://phabricator.wikimedia.org/T355866#9494244 [15:51:59] ok great! I didn't realise the significance of the comment, cool :) [15:52:05] I'll let you know when we are done [15:52:10] thanks topranks ! [15:53:00] I'll downtime 2027 and 2028 for an hour as well, good catch [15:56:05] arnaudb: no need to downtime if you haven't already, I'll do that for the whole rack before we begin [15:56:28] oh TIL, thanks (but too late x)) [16:11:40] arnaudb: ok we are done, all looking ok if you want to repool [16:11:56] on it, thanks :-) [16:12:02] was quick! [16:49:33] PROBLEM - MariaDB sustained replica lag on s5 on db2178 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2178&var-port=9104 [16:50:33] RECOVERY - MariaDB sustained replica lag on s5 on db2178 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2178&var-port=9104