[04:06:21] Wow change_tag is 71G in s8 [05:02:08] Amir1: let me know when around so I can push the MW change to close es4 and es5 [05:02:11] No rush [06:37:47] I've to be AFK to go and pickup access keys to the technical room of my building from a random office 5km away, still available on IRC though I have it with me :) [06:43:25] good luck! [08:01:32] es6 backups are alerting because they grew a +544922.8 % [08:12:31] Expected :) [08:12:42] That's a nice rate [08:18:29] back from roaming around trying to find a signal, turns out there has been an issue on a migration on the ISP's side, 2 buildings here are completely in the dark [09:30:49] marostegui: good "morning". I'm around now [09:31:07] XDD [09:31:17] can you review https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1030918 ? [09:34:41] deploying [09:35:26] \o/ [09:50:37] change deployed, I am not going to confirm that es4 and es5 are indeed not getting any writes [10:03:21] https://phabricator.wikimedia.org/T364447#9793792 [10:18:25] 🎉 [10:19:19] 10h to alter change_tag in s8 [10:19:32] it is going to take a while to get that section done [10:20:41] and it's going to get even bigger [10:20:47] I'm sad [14:51:40] I am leaving, reminder: I am off tomorrow [14:52:05] * volans too :D [14:52:36] Off or leaving? [14:52:51] tomorrow off [14:53:07] slacker [20:40:32] PROBLEM - MariaDB sustained replica lag on s8 on db2154 is CRITICAL: 3.379e+04 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2154&var-port=9104 [21:24:34] RECOVERY - MariaDB sustained replica lag on s8 on db2154 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2154&var-port=9104