[01:09:38] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 8.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:11:24] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[08:34:13] <Amir1>	 marostegui: I updated the docs and uploaded the code somewhere, from now on, password rotation should be doable relatively quickly https://wikitech.wikimedia.org/wiki/MariaDB/Changing_user_passwords
[08:34:22] <Amir1>	 (and wouldn't need me either)
[11:54:31] <Emperor>	 I found code under a rock on the internet to check the Swift expiry queue. It didn't need _much_ work to make it do something useful. Anyhow, we have 276369 objects currently in the queue, range from 2023-01-27 to 2026-01-31, 20 stale entries, 8 pending. So not the cause of our current woe, but it's good we have an OKR to start actually expiring them this quarter before the backlog gets too big
[14:42:14] <marostegui>	 I have depooled the hosts on https://phabricator.wikimedia.org/T327404