[10:39:09] FIRING: MySQLReplicaNotUsingGTID: MySQL replica db2246:9104 not using GTID - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/0fec1d02-1b0b-44c0-84b0-64894f3ba682/mariadb-gtid - https://alerts.wikimedia.org/?q=alertname%3DMySQLReplicaNotUsingGTID [10:39:20] on it [10:39:22] setting up x4 [10:44:09] RESOLVED: MySQLReplicaNotUsingGTID: MySQL replica db2246:9104 not using GTID - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/0fec1d02-1b0b-44c0-84b0-64894f3ba682/mariadb-gtid - https://alerts.wikimedia.org/?q=alertname%3DMySQLReplicaNotUsingGTID [13:32:24] I guess that timing was 100% predictable [20:34:22] FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:44:22] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed