[06:04:57] FIRING: SystemdUnitFailed: prometheus-mysqld-exporter.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:06:08] ^ me [06:06:11] expected [07:42:04] FIRING: MysqlPredictiveFreeDiskSpace: Host db1269:9100 predictive low disk space on /srv - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/goto/Jdz2PnLNg?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMysqlPredictiveFreeDiskSpace [07:42:13] ^ expected [08:48:53] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s5.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:49:27] expected - will silence it there will be some more arriving for the different sections this host will host [09:28:53] FIRING: [6x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s3.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:05:37] Hi folks, could I get a review on https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1328544 to move our ceph images to the latest upstream point release, please? [10:18:26] Thanks :) [13:30:09] FIRING: [3x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s5.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:59:57] FIRING: [4x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s5.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:10:02] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s5.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:25:44] I tweaked https://grafana.wikimedia.org/d/000000303/mysql-replication-lag?orgId=1&from=now-24h&to=now&timezone=utc&var-site=%24__all&var-section=s1&var-section=s2&var-section=s3&var-section=s4&var-section=s5&var-section=s6&var-section=s7&var-section=s8&var-section=x1&var-section=x2 to support Site=All in the top left of the screen [15:33:53] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s5.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:33:53] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s5.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:34:57] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s5.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed