[03:39:59] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:28:55] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:43:55] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: xfs_scrub_all.service on ms-be1090:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:29:43] <Emperor>	 The final sadness is due to T436615
[08:29:43] <stashbot>	 T436615: exim4.conf permissions mean xfs_scrub_all_fail.service cannot run - https://phabricator.wikimedia.org/T436615
[08:30:33] <Emperor>	 actually, let me reset-failed that unit since I've fixed the filesystem and opened a task about the permissions problem that's causing the failure
[08:36:04] <marostegui>	 I've added codfw test-s4 hosts to dbctl
[08:36:13] <cezmunsta>	 +1
[08:58:55] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: prometheus-mysqld-exporter.service on db1270:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:59:08] <marostegui>	 ^fixing, it is being setup
[09:02:57] <marostegui>	 eqiad test-s4 hosts added too
[09:03:55] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: prometheus-mysqld-exporter.service on db1270:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:38:55] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: xfs_scrub_all_fail.service on ms-be1090:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:08:08] <marostegui>	 test-s4 is now fully functional in dbctl: https://phabricator.wikimedia.org/T427059#12274626
[10:08:28] <federico3>	    \o/
[10:09:13] <federico3>	 maybe we can tweak the --skip-safety-checks to do the right thing for test-s4
[10:10:00] <marostegui>	 I'd leave it, so we now it is exactly how production works, but happy to discuss!
[12:21:55] <marostegui>	 federico3 cezmunsta https://gerrit.wikimedia.org/r/c/operations/puppet/+/1333186
[19:28:55] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1228:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[23:28:55] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1228:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed