[01:09:21] PROBLEM - MariaDB sustained replica lag on m1 on db1217 is CRITICAL: 4.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321 [01:10:49] RECOVERY - MariaDB sustained replica lag on m1 on db1217 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321 [04:19:49] I'm back, I have hundreds of mails. if someone needs me to focus on something urgent please let me know [06:16:10] welcome back [06:24:34] thanks! [09:23:01] https://www.theregister.com/2023/10/13/mariadb_restructure/ [09:25:46] yep [10:27:55] FYI: I'm running maint script migrating pagelinks stuff in s2, s3, s5, and s7. You'll see more writes there (and db sizes and backups growing for now) [10:46:46] 👍 [10:46:57] hello everybody, by the way [10:47:11] welcome back jynus 👋 [10:48:19] how did things go while I was out? More smoothly I am guessing 😆 [11:20:21] nothing to report afaik! [11:22:40] we had a few mails about long-running backups, but AFAICT they'd run to completion, just slowly [11:54:04] Emperor: that was sorta expected, I made the new clusters but didn't switch them because I was waiting for Jaime to come back, so hopefully for next week onwards you won't see it again (you'll see it again for this week's backup because we need to do the last RO one) [12:30:39] while switching the clusters, I made an interesting discovery. Triggering abuse filter will add an extra ES row [12:30:49] https://www.irccloud.com/pastebin/ngcGdeal/ [12:31:26] which is DB://cluster28/1 [12:43:23] clusters have been rotated [12:43:26] -rw-rw---- 1 mysql mysql 232838397952 Oct 16 12:34 zhwiki/blobs_cluster27.ibd [12:43:26] -rw-rw---- 1 mysql mysql 9437184 Oct 16 12:42 zhwiki/blobs_cluster29.ibd [12:43:44] marostegui jynus ^ [12:43:51] great [12:47:09] nice, great job [13:28:04] so far so good [13:28:07] https://www.irccloud.com/pastebin/GH5PjFPM/