[01:11:14] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 11.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [01:13:10] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [14:25:50] For later: claime: I agree, I had a dashboard but wasn't accepted -> what did you mean? [14:26:16] I would like to have a dashboard to check what a host is [14:26:31] I proposed one but Amir shut me down [14:51:37] so as a summary not too worried about the hw down, just trying to figure out how to best replace it [14:51:55] (as part of the service) [14:53:00] s4 is the largest section, backing it up everyday probably makes it the busiest db we have :-D, and the most likely to crash 🫡🫡🫡 [14:53:42] any suggestions for a harder reset than a hard reset? 0:-D [14:56:18] jynus: it needs the crash kart for sure [14:56:27] :-D [14:56:29] I would ping chris or john directly on irc [14:58:50] jynus the harder reset to the hard reset is a complete shutdown for a minute, removing power, etc [14:59:10] yeah, that I knew but wanted to avoid :-D [14:59:20] but I think that's the only way [14:59:25] I am ordering the new dim for db1150 now [14:59:41] thank you cmjohnson1, take your time, just was trying to avoid you work [16:30:50] loving a s5 recovery as it took just 30 minutes to recover that over 1Gbit [18:25:29] jynus: "Amir shut me down" what did I do? 😔 I need a bit of context 😅