[06:36:32] <jynus>	 hi there
[07:16:02] <jynus>	 sadly I cannot fix the multiple ongoing backup issues as I don't have production access
[07:21:13] <Emperor>	 oh?
[07:22:01] <jynus>	 Emperor: if you are bold enough, you can +1 this: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338091
[07:23:48] <jinxer-wm>	 FIRING: [2x] MysqlReplicationLagPtHeartbeat: MySQL instance db2196:9104 has too large replication lag (4d 6h 46m 19s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2196&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[07:24:35] <jynus>	 I know we have been without media backups at least since the 4th
[08:23:48] <jinxer-wm>	 RESOLVED: [2x] MysqlReplicationLagPtHeartbeat: MySQL instance db2196:9104 has too large replication lag (1h 14m 23s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2196&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[11:33:25] <Amir1>	 marostegui: FYI https://phabricator.wikimedia.org/T428517#12301261
[11:33:43] <marostegui>	 nice one!