[03:29:57] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@m5.service on db1217:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:53:54] RESOLVED: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@m5.service on db1217:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:54:05] FIRING: MySQLReplicaNotUsingGTID: MySQL replica db2902:9104 not using GTID - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/0fec1d02-1b0b-44c0-84b0-64894f3ba682/mariadb-gtid - https://alerts.wikimedia.org/?q=alertname%3DMySQLReplicaNotUsingGTID [13:11:39] uh? [13:57:48] federico3: how did you create that instance? [13:59:22] the same process from the wiki page as the others but I'm going to check if something is different [14:00:09] There are 3 log entries from yesterday morning, roughly 30s apart "Previous Using_Gtid=Slave_Pos. New Using_Gtid=No" [14:00:58] Also, on disk I see files left over from the way that the clone occurred, perhaps try the command that I suggested (xtrabackup) and clone it from a sibling? [14:02:26] they all cloned from 2230 [14:03:32] the process first disables gtid then reenables it, but anyhow I set gtid now [14:04:05] RESOLVED: MySQLReplicaNotUsingGTID: MySQL replica db2902:9104 not using GTID - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/0fec1d02-1b0b-44c0-84b0-64894f3ba682/mariadb-gtid - https://alerts.wikimedia.org/?q=alertname%3DMySQLReplicaNotUsingGTID [16:30:02] Emperor: Hi, for when you have time: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1327104 [17:44:37] Binlog diving on pc1018, if you some stuff, ping me