[03:29:43] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1228:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:49:43] RESOLVED: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1228:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:22:18] I want to give a hug to the DBAs for the recent spike in hw errors [09:43:18] also freeing 8Us of space on each datacenter WIP [11:16:16] gerrit backup size has decreased almost 1000x times: https://phabricator.wikimedia.org/T411583#12085288 [11:23:36] it's weird, I got an R status, which is running for a fileset, but it is not running according to the director [11:24:53] and on the other side it says completed on the logs, some metadata issues must have happened, like a db uncontactable or something [11:25:20] I reran it anyway to get rid of the alert [11:30:08] +1 [12:29:49] jynus: db1245 being down for 15h is known/planned? [12:30:02] ? [12:30:06] I wasn't aware [12:30:27] 23:09:39 <+icinga-wm> PROBLEM - Host db1245 is DOWN: PING CRITICAL - Packet loss = 100% [12:30:32] :-( [12:32:10] investigating, serial console unresponsive [12:32:53] jynus: https://phabricator.wikimedia.org/T431115 [12:33:06] :-( [12:33:18] thank you [12:33:42] jynus: you'll take it from there? [12:33:48] yeah [12:33:51] thanks! [12:34:03] I think I may have replacement for that [12:43:05] tried a hardreset and it didn't work, will need dc help to drain and try there [12:51:53] I should not have hugged you :-D [12:52:09] (just kidding, still my strongest hug to you) [13:16:42] thank you a lot for the heads up, at least I was able to workaround it quickly using another host