[05:16:04] <marostegui>	 great job handling db1131 Amir1 
[05:16:38] <Amir1>	 I wish I could do better, but it's good to have a SRE living in pacific ocean timezone
[05:17:38] <marostegui>	 indeed!
[05:17:49] <marostegui>	 I am going to raise the priority to high though, as it is the candidate master for s6
[05:18:09] <Amir1>	 sgtm
[07:44:32] <Amir1>	 marostegui: I'm off to rest, there is a script running for s7, in mwmaint as my screen. it should be fine though. djvu fixes for s4 is also running in screen.
[07:45:00] <marostegui>	 sounds good
[07:45:03] <marostegui>	 get some rest
[08:29:14] <Emperor>	 ah, good, yesterday's faffing around getting more kern.log entries for T295563 seems to have done the trick (so hopefully the disk will now get replaced)
[08:29:15] <stashbot>	 T295563: hw troubleshooting: disk failure (sdr) for ms-be2059.codfw.wmnet - https://phabricator.wikimedia.org/T295563
[08:29:36] <marostegui>	 :-)
[10:18:34] <jynus>	 backups worked much faster, now that the 3 hosts are back at 100% utilization :-)
[10:19:10] <jynus>	 they were ready by 4am, just as manuel gets up :-)
[10:20:14] <marostegui>	 hahaha
[10:25:12] <jynus>	 I think scheduling backups in the most optimal way is an NP-hard problem 🤔 so I won't waste much time over-optimizing
[11:24:11] <jynus>	 "Reimage db1125 (test cluster) with Bullseye" 👀
[11:24:32] <marostegui>	 :)
[11:24:53] <jynus>	 great idea to try to keep 10.4
[11:33:53] <jynus>	 FYI db1101:s7 seems a bit stressed at the moment: https://logstash.wikimedia.org/goto/8c587eac08efe9f0639720b76e93e18b
[11:35:05] <marostegui>	 probably Amir1's script
[11:35:44] <jynus>	 to keep an eye: https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=3&orgId=1&var-job=All&var-server=db1101&var-port=13317&from=1637192114848&to=1637235314848
[11:35:58] <marostegui>	 it is mainly from the jobrunner anyways
[11:36:34] <jynus>	 it think it is that because it is the main user of wait for replication, it is because it is lagging 1-2 seconds
[12:10:52] <jynus>	 what was the bug about binlog not working properly after an upgrade and that it wasn't on the release notes? did that get fixed?
[12:11:30] <marostegui>	 What's that bug?
[12:11:46] <jynus>	 that's the thing, I cannot find it
[12:11:53] <jynus>	 I remember you upgrading a minor version
[12:12:10] <jynus>	 and something about replication enabling or disabling being replicated
[12:12:14] <jynus>	 an alter maybe?
[12:12:17] <marostegui>	 ah
[12:12:20] <marostegui>	 the optimize
[12:12:23] <marostegui>	 yeah, that's not fixed
[12:12:23] <jynus>	 and that monty was going to see it
[12:12:27] <marostegui>	 I pinged them a week ago I think
[12:12:34] <marostegui>	 https://jira.mariadb.org/browse/MDEV-26618
[12:12:52] <jynus>	 thanks, I couldn't find it
[12:13:09] <volans>	 hey, it's not phabricator, search should work fine :-P
[12:13:30] <jynus>	 I didn't remember if was optimize or alter or something
[12:13:47] <marostegui>	 yeah, it is "limited" to optimize
[12:16:12] <jynus>	 I am going to at least vote for it
[12:16:26] <marostegui>	 pleeease!