[01:13:57] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s4.service on db1245:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[03:02:03] <jinxer-wm>	 FIRING: [4x] PuppetFailure: Puppet has failed on backupmon1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[05:15:01] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s4.service on db1245:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[05:20:31] <marostegui>	 ^ that's a backup host that crashed time ago and dcops just brought it back
[05:20:34] <marostegui>	 I will get it started
[05:22:22] <marostegui>	 I will have to reclone it because it's been down way too much so the master rotated all the binlogs already
[06:23:26] <marostegui>	 Amir1:  https://gerrit.wikimedia.org/r/1339509
[08:31:51] <marostegui>	 Amir1: I am done with arbcom_arwiki so it can be created anytime
[08:37:27] <Amir1>	 thanks!
[13:17:17] <elukey>	 Emperor: o/ I lost track of https://phabricator.wikimedia.org/T433635, is it still an issue?
[13:22:29] <Emperor>	 elukey: ms-be2082 is running trixie, so I'm assuming not - I think the underlying issue with the reimage cookbook got fixed (by you?); yes, that fits my notes from 31 July.
[13:23:09] <Emperor>	 elukey: Or at least, I don't know if ipmitool access to SM C-J got fixed, but you made the cookbook not need it for UEFI hosts.
[13:24:04] <Emperor>	 elukey: cf T426180
[13:24:05] <stashbot>	 T426180: Move away from root as user to access our BMCs - https://phabricator.wikimedia.org/T426180
[13:27:17] <elukey>	 Emperor: yep yep but I wasn't sure if they were all UEFI. Perfect, do you mind to review/close the task during the next days if nothing comes up?
[13:27:37] <elukey>	 I am tracking down open things etc.. for wmfroot, and I totally forgot about that task
[13:29:43] <Emperor>	 elukey: we confirmed that all SM C-J are uefi booting (as noted on T433635), is there anything else that should be checked, or shall I just close the task?
[13:29:43] <stashbot>	 T433635: Unable to reimage or reprovision ms-be2082 due to redfish connection errors - https://phabricator.wikimedia.org/T433635
[13:30:04] <elukey>	 Emperor: I don't think so, we can close!
[13:33:02] <Emperor>	 cool
[13:33:34] <Emperor>	 {{done}}
[13:42:42] <elukey>	 thanks!
[19:18:58] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1199:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[23:20:01] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1199:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed