[07:38:29] <godog>	 Emperor: FYI I'm going ahead with https://gerrit.wikimedia.org/r/c/operations/puppet/+/956906 and start reimage thanos-fe1001
[07:56:33] <Emperor>	 ack
[10:33:43] <godog>	 Emperor: I see new_rings.tar.bz2 missing right after reimage on thanos-fe1001 (filed as https://phabricator.wikimedia.org/T347964) have you run into this before ?
[10:53:47] <Emperor>	 Ah, yes, you've reimaged the ring manager. I would check that there isn't an outstanding ring change (sudo /usr/local/bin/swift_ring_manager -o /var/cache/swift_rings/ ) and then rebalance the rings (which writes new rings) sudo /usr/local/bin/swift_ring_manager --doit --rebalance -o /var/cache/swift_rings/
[10:58:38] <godog>	 ack, trying that now
[10:59:41] <Emperor>	 (assuming that works, probably worth a note on swift/howto)
[11:03:55] <godog>	 yeah that works, I'll add a note to the docs
[11:42:53] <Emperor>	 TY
[16:14:09] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db1127 is CRITICAL: 76.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1127&var-port=9104
[16:14:33] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2169 is CRITICAL: 23 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2169&var-port=13317
[16:15:35] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2122 is CRITICAL: 31.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2122&var-port=9104
[16:16:03] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2169 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2169&var-port=13317
[16:17:07] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2122 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2122&var-port=9104
[16:17:13] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db1127 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1127&var-port=9104