[01:10:26] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 13.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:12:16] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[10:07:13] <claime>	 I would need someone to confirm swift and swift-ro can be switched over as part of the service switchover on the 28th? They were excluded from the switchdc.services cookbook because of a rebalance https://phabricator.wikimedia.org/T287539#7339799 but afaik there's no reason to keep them out.
[10:10:39] <paravoid>	 win 26
[10:40:15] <jynus>	 paravoid: https://www.youtube.com/watch?v=9QS0q3mGPGg&t=34s
[10:43:00] <jynus>	 claime: I noted your question for todays meeting, if it happens
[10:43:19] <claime>	 jynus: ack, thank you very much <3
[10:44:00] <jynus>	 big IF, many people off
[10:44:25] <claime>	 Still, thanks :)
[10:46:50] <Amir1>	 dealing with swift feels more like https://youtu.be/gfkts0u-m6w?t=152
[10:47:42] <jynus>	 Amir1: imagine media backups that includes the best parts of swift AND mw file management!
[10:47:51] <Amir1>	 lol
[10:47:52] <Amir1>	 fair
[10:48:23] <jynus>	 big news is errors went down to only 100K
[10:48:47] <Amir1>	 "only" xD
[10:49:45] <jynus>	 to be fair, that's 0.1% of errors
[10:49:57] <jynus>	 and 40K of thoses I think are already accounted for
[13:20:27] <jynus>	 Amir1: there is a section about doing a data check at: https://wikitech.wikimedia.org/wiki/MariaDB#Comparing_tables
[13:23:44] <jynus>	 and this is the open ticket to fully automatize it: https://phabricator.wikimedia.org/T207253
[13:54:11] <Amir1>	 thanks!