[01:10:16] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [01:12:06] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [07:44:33] Emperor: whenever you have time, I would appreciate a review of (mostly double checking the IPs are ok) https://gerrit.wikimedia.org/r/c/operations/puppet/+/895910 [09:14:24] OK, will get to that this morning [other half has Covid, so everything a bit chaotic here this morning] [09:17:11] Emperor: thanks1 [09:17:13] thanks! [09:17:15] good luck [15:39:06] In 20 minutes I am switching over m5 db master, which will affect toolhub, mailman and some other WMCS related databases. Impact: RO for around 1 minute, reads unaffected https://phabricator.wikimedia.org/T330847 [16:27:21] Its been a long run but this is finally done https://phabricator.wikimedia.org/T321312 [16:28:17] \o/