[02:22:25] (SystemdUnitFailed) firing: opensearch_2@production-elk7-eqiad.service on logstash1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:47:25] (SystemdUnitFailed) resolved: opensearch_2@production-elk7-eqiad.service on logstash1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:50:25] (SystemdUnitFailed) firing: opensearch_2@production-elk7-eqiad.service on logstash1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:15:25] (SystemdUnitFailed) resolved: opensearch_2@production-elk7-eqiad.service on logstash1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:23:25] (SystemdUnitFailed) firing: opensearch_2@production-elk7-eqiad.service on logstash1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:28:25] (SystemdUnitFailed) resolved: opensearch_2@production-elk7-eqiad.service on logstash1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:48:10] Created T359612 for ^^ [04:48:11] T359612: logstash1011 is down - https://phabricator.wikimedia.org/T359612 [15:54:53] in the prometheus apache exporter there is this line that adds arguments '-scrape_uri http://127.0.0.1/server-status/?auto', and that has already been added waaaay back in 2016 [15:55:01] but when I use it on bookwork I get: error: unknown short flag '-s', try --help [15:55:12] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/prometheus/manifests/apache_exporter.pp#11 [15:55:25] guess it changed [15:55:56] cwhite denisse thank you re: prometheus thanos and logstash1011 [15:56:03] mutante: ah yeah totally [15:56:22] will likely need a distro-based switch or sth like that [15:56:46] godog: ACK, let me see what might replace it in bookworm. then happy to make such a change [15:57:05] mutante: cheers! appreciate it [15:58:25] well.. --scrape_uri="http://localhost/server-status/?auto" [15:58:32] just add a - [15:58:44] it seems [15:59:08] like it was odd all this time and now more normal :) [15:59:10] easy enough, they must have changed the argv parsing library [16:01:14] confirmed. on a buster host all options just use -