[05:04:41] <marostegui>	 Starting to move s5 replicas
[05:37:49] <Amir1>	 marostegui: Thanks
[06:52:20] <jynus>	 "Last dump for s8 at eqiad (db1171) taken on 2022-05-31 00:51:23 is 205 GiB, but the previous one was 217 GiB, a change of -5.6 %"
[06:52:53] <marostegui>	 jynus: that is probably the revision_actor_temp drop :)
[06:58:30] <marostegui>	 Amir1: you can use db1100 I am done (finished schema changes, rebooted it, upgraded it etc)
[06:59:12] <Amir1>	 Thanks
[09:38:23] <marostegui>	 Amir1: should I leave db1100 (old s5 master) depooled?
[09:38:37] <marostegui>	 Or do you want me to pool it and you depool it whenever you need it?
[10:01:25] <Amir1>	 marostegui: let it be depooled for now
[10:01:59] <marostegui>	 sounds good!
[10:03:57] <marostegui>	 Amir1: Enabled notifications, I'll let you adjust the downtime depending on your needs 
[10:35:52] <Amir1>	 marostegui: thanks
[10:55:18] <Emperor>	 blasted eqiad swift hardware. ms-be1066 has a probably-duff disk (it's now twice failed with IO errors)
[11:30:53] <sobanski>	 :(
[12:04:04] <jynus>	 I did a full upgrade on dbprovs to patch for backup version comparson problem and new xtrabackup version
[12:06:59] <jynus>	 I will not upgrade backup sources yet
[12:11:56] <jynus>	 FYI ssh mgmt on db1109 went down a few minutes ago
[12:13:18] <marostegui>	 yep on my radar
[12:32:45] <Amir1>	 marostegui: the schema changes on the old master will take a while :D probably until midnight or so
[12:33:10] <marostegui>	 make sure to downtime it accordingly :)
[12:33:32] <sobanski>	 And hope for no Icinga glitches
[12:33:34] <Amir1>	 the script will do it anyway :P
[12:33:49] <sobanski>	 BTW, does anyone have a task for that Icinga problem handy?
[12:33:56] <marostegui>	 yep
[12:34:11] <marostegui>	 https://phabricator.wikimedia.org/T309447
[12:34:11] <Amir1>	 T309447
[12:34:12] <stashbot>	 T309447: Icinga paged for a host that should have been downtimed - https://phabricator.wikimedia.org/T309447
[12:34:16] <sobanski>	 Thanks
[15:40:35] <Amir1>	 marostegui: the maint is done, slowly repooling db1100 now
[15:40:58] <marostegui>	 cool
[22:51:56] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 61.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[22:54:58] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321