[17:13:02] Hi observability folks! From a deploy server (e.g. deploy1002.eqiad.wmnet) I can issue queries to logstash via http://logstash1023.eqiad.wmnet:9200/logstash-*/_search . Is there any equivalent URL that can be used for offsite access (assuming I supply an oauth cookie after IDP login)? [17:34:25] Hey 0lly, has there been any work towards EXTERNAL polling via ipmitool and/or redfish API? We've been having some HW issues (usually backplane) that make the host unbootable. Was wondering if any efforts have gone towards this. Re T253810 and T367790 [17:34:26] T253810: Alert on ECC warnings in SEL - https://phabricator.wikimedia.org/T253810 [17:34:27] T367790: Hardware failures: consider alerting via SEL messages - https://phabricator.wikimedia.org/T367790 [18:06:57] dancy: there is/was https://logstash.wikimedia.org/elasticsearch but I think is no longer working (some more info in https://wikitech.wikimedia.org/wiki/Logstash#API) would you mind outlining your use case in a task to have a deeper look? [18:17:34] hey inflatador, offhand I think those tasks are up to date, essentially we can do this via ipmi/prom but aren't yet. fwiw a good portion of the team are ooo this week so I added a note to review at our next meeting too and follow up. I think it'd be good to keep progressing. in the mean time if you have any details/thoughts about the eternal portion please do add on task [19:16:34] Thanks herron , I'll add some suggestions in ticket. As of now, I'm thinking a blackbox check that uses basic auth to scrape redfish API and read the SAL . Will probably need feedback from v-olans/DC Ops [19:21:42] Or maybe something like this https://developer.dell.com/apis/2978/versions/5.xx/docs/1Eventing.md [19:28:03] from a cursory look around this is interesting too https://github.com/sapcc/redfish-exporter [19:38:30] * inflatador wonders if there's some kind of DRAC simulator or test host I could play around with