[08:30:12] Morning folks, can I have a +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1177940 adding some new swift proxy nodes, please? CR has a link to the docs for this step [09:24:51] thanks :) [16:21:28] Amir1: I've done the prep for s8 codfw failover, I'm starting the flip now [17:16:48] FIRING: [8x] MysqlReplicationLagPtHeartbeat: MySQL instance db2152:9104 has too large replication lag (51m 16s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [17:21:48] RESOLVED: [8x] MysqlReplicationLagPtHeartbeat: MySQL instance db2152:9104 has too large replication lag (54m 16s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [17:39:48] FIRING: [7x] MysqlReplicationLagPtHeartbeat: MySQL instance db2152:9104 has too large replication lag (11m 4s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [17:44:48] FIRING: [8x] MysqlReplicationLagPtHeartbeat: MySQL instance db2152:9104 has too large replication lag (15m 39s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [17:49:48] RESOLVED: [9x] MysqlReplicationLagPtHeartbeat: MySQL instance db2152:9104 has too large replication lag (17m 39s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat