[09:43:31] Hi folks - anyone around today who could review https://gerrit.wikimedia.org/r/c/operations/puppet/+/1059259 please? Marking a failed disk as failed (as there are no spares available, and this hardware is being decom soon). [10:05:17] https://wikitech.wikimedia.org/wiki/Swift/Ring_Management#Removing_a_device ^-- docs for the above [12:06:40] thanks :) [12:07:51] :) [12:55:30] /set weechat.bar.status.color_bg gray [12:55:48] tappof: :) [12:56:11] marostegui: sorry :-P wrong buffer [13:14:48] FIRING: MysqlReplicationLagPtHeartbeat: MySQL instance db1206:9104 has too large replication lag (1m 0s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1206&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [13:15:05] checking [13:15:33] dumps again [13:19:48] RESOLVED: MysqlReplicationLagPtHeartbeat: MySQL instance db1206:9104 has too large replication lag (1m 0s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1206&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [13:20:37] I just commented on the dumps task https://phabricator.wikimedia.org/T368098 [14:19:41] any actions needed right now marostegui or is it mostly informational? xcollazo is on the sec channel if needed (he's typically on by now), and i reckon he'll see the phab comment via email. b.tullis will be back next week. i guess the alerts need to continue existing, is that right? or is there any way to quiet down alerts for these recurring jobs? [15:23:08] dr0ptp4kt: we are good for now yeah, it was mostly for tracking [15:23:12] dr0ptp4kt: thanks though :) [15:47:38] phew! thanks :)