[10:20:49] FIRING: PuppetFailure: Puppet has failed on ms-be1056:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:21:38] This is (still) T371192 ; I'll extend the downtime [10:21:38] T371192: Disk (sdh) failed on ms-be1056 - https://phabricator.wikimedia.org/T371192 [13:58:33] [cephadm ERROR root] Failed to run cephadm http server: Expected 4 octets in '2620:0:861:102:10:64:16:40' [13:58:41] Really? :sadpanda: [15:27:21] that is not fun :( [15:34:16] after stunt-patching the mgr to actually spit out the exception, I've found it's fixed in 18.2.4 (though the .debs of 18.2.4 have some ... infelicities). Doesn't help with the "only the active mgr actually runs the service discovery endpoint" issue, though...