[07:17:19] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s4 on db1244 is CRITICAL: 87 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1244&var-port=9104
[07:20:19] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s4 on db1244 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1244&var-port=9104
[10:51:13] <Emperor>	 Hi folks, I'm still hoping for a review on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1054864 please?
[10:56:54] <volans>	 Emperor: +1ed the puppet side, as for the acutal service to be stopped I trust your analisys
[10:57:05] <Emperor>	 TY :)
[12:00:46] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 400 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104
[12:08:40] <jinxer-wm>	 FIRING: [84x] MysqlReplicationLagPtHeartbeat: MySQL instance db2115:9104 has too large replication lag (8m 14s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[12:12:29] <jinxer-wm>	 RESOLVED: [84x] MysqlReplicationLagPtHeartbeat: MySQL instance db2115:9104 has too large replication lag (8m 14s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[12:17:14] <marostegui>	 arnaudb: What's the issue with the long running backup email?
[12:17:51] <arnaudb>	 backup took longer than last time, I was waiting for the next one to run to compare, it finished properly
[12:18:35] <marostegui>	 That's the one from yesterday or today?
[12:18:47] <marostegui>	 There are two
[12:19:30] <arnaudb>	 I missed the one from yesterday, hold on
[12:22:03] <arnaudb>	 don't see the one you're mentionning marostegui is it from backupmon1001?
[12:23:55] <marostegui>	 We had s6 and today's email 
[12:24:59] <arnaudb>	 I don't see s6 email, could you forward it to me?
[13:13:51] <Emperor>	 I've pinged g.odog on T351927 again, will silence the thanos disk space alers for 1w
[13:13:51] <stashbot>	 T351927: Decide and tweak Thanos retention - https://phabricator.wikimedia.org/T351927
[13:15:05] <Emperor>	 ...though that does highlight a bunch of nodes with failed Prometheus-mysqld-exporter for a couple of days.
[14:49:57] <topranks>	 arnaudb: hey!
[14:50:16] <topranks>	 sorry I should have reached out, that last switch upgrade has been postponed until next Tues (Jul 23rd)
[14:50:26] <arnaudb>	 oooh
[14:50:31] <arnaudb>	 even simpler for me
[14:50:36] <arnaudb>	 thanks topranks I forgot to double check the date
[14:51:31] <topranks>	 no probs - we've another big network change today so didn't want to trip over ourselves 
[14:51:47] <topranks>	 last switch upgrade Tuesday so we're almost there!