[00:48:55] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: curator_actions_cluster_wide.service on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[04:48:55] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: curator_actions_cluster_wide.service on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:48:55] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: curator_actions_cluster_wide.service on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:38:55] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: statograph_post.service on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:40:10] <jinxer-wm>	 FIRING: [4x] SystemdUnitFailed: statograph_post.service on alert1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:43:55] <jinxer-wm>	 FIRING: [4x] SystemdUnitFailed: statograph_post.service on alert1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:48:55] <jinxer-wm>	 FIRING: [4x] SystemdUnitFailed: statograph_post.service on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:23:11] <cdanis>	 hi o11y folks and/or [OE]LK stack knowers.  how can I tell OS that I really do want it to do a full table scan for an aggregation query like the following?  presented in their SQL dialect because I'm not even sure how to express this otherwise https://phabricator.wikimedia.org/P69114
[13:42:22] <godog>	 good question, I don't really know off the top of my head
[13:43:45] <cdanis>	 in another question, does anyone know of a good shell one-liner to dump an entire index as json so i can do this with jq and awk instead
[13:48:26] <godog>	 mmhh oneliner I'm not sure about, though either a quick python script using the client comes to mind, or elasticsearch-dump
[14:38:55] <jinxer-wm>	 RESOLVED: [2x] SystemdUnitFailed: curator_actions_cluster_wide.service on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:47:38] <cwhite>	 cdanis: AFAIK, all queries must go through the index and there's no way around that.  To grep through the source data, I think we're forced into dumping it to a text file.
[14:48:13] <cwhite>	 There's also no great one-liner to do it because query responses have to be batched.
[14:49:45] <cwhite>	 I'll look at the query and see if there's anything else we can do.  Do you have a task and/or a description of the end goal?
[14:50:41] <cdanis>	 I was hoping to find traceIds with spans that are minutes or hours apart
[14:51:05] <cdanis>	 the overall context is T331127
[14:51:05] <stashbot>	 T331127: phantom redirects lingering in incategory searches after page moves - https://phabricator.wikimedia.org/T331127
[14:51:21] <cdanis>	 and uh more specifically T374662
[14:51:22] <stashbot>	 T374662: PHP web requests running for multiple hours - https://phabricator.wikimedia.org/T374662
[14:53:02] <cwhite>	 ack, thank you.  I'll circle back sometime today.
[14:54:45] <cdanis>	 thanks!
[17:08:01] <cwhite>	 cdanis: I'm pretty sure I can say that SQL query would yield 0 results in jaeger-span-2024.09.12 +/- a couple days due to filtering for `LOCAL_mw-web`
[17:08:09] <cwhite>	 But I have a query that yields results if you remove that filter.  Here's the queries: https://phabricator.wikimedia.org/P69117
[17:15:56] <cdanis>	 thanks!