[01:08:51] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1217 is CRITICAL: 10.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321
[01:11:37] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1217 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321
[09:43:20] <btullis>	 jynus: if you're around. I'm about to proceed with T343109 if that's OK. Are you happy for me to restore the snapshot snapshot.s2.2023-10-23--01-34-18 from dbprov1004 to dbstore1007 ?
[09:43:21] <stashbot>	 T343109: Recover dbstore1007:s2 from the database provisioning service - https://phabricator.wikimedia.org/T343109
[09:45:41] <btullis>	 Or anyone, for that matter.
[09:50:28] <jynus>	 you should be able to do it yourself, we gave you the actual command
[09:51:05] <jynus>	 no need to ask for permission
[09:56:02] <btullis>	 OK, thanks. I just thought it best to check. Proceeding.
[09:58:14] <jynus>	 just reach out if the full guide ( https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Provision_a_precompressed_and_prepared_snapshot_(preferred) ) doesn't work for you
[10:01:45] <btullis>	 Will do, thanks.
[10:17:17] <marostegui>	 btullis: did you downtime the host?
[10:17:20] <marostegui>	 we are getting alerts
[10:24:31] <btullis>	 Sorry. I downtimed the s2 services and the systemd alert.
[10:24:56] <marostegui>	 there are alerts for s2
[10:25:09] <btullis>	 Sorry, I missed some. 
[10:25:22] <marostegui>	 np!