[14:31:52] Hi all, I wanted to run some queries on the thanos web interface but get "Error executing query: JSON.parse: unexpected character at line 1 column 1 of the JSON data". Is this expected/related to switchover? Do I have to use another thanos? [14:34:00] jelto: checking [14:36:03] jelto: should be better now, there was thanos-fe2002 pooled for thanos-web, something that doesn't work well with sso [14:36:58] godog: great, thanks for the quick help. Query works now! [14:37:42] np [16:36:05] hi folks! [16:36:50] I am trying to figure out why I don't see some ORES access logs on kibana, and I am wondering if there is any way to see how much traffic is (if any) discarded by the ores filter in logstash [16:45:18] elukey: there should be some indication here https://grafana.wikimedia.org/d/000000561/logstash?orgId=1&refresh=5m&viewPanel=45 [16:46:07] godog: thanks! I don't see ORES so it may be something else.. [16:47:51] elukey: could be yeah, also IIRC we throttle access logs but I can't remember if that's blanket or only some [16:55:18] elukey: mind sharing the link or query you're using? [16:56:47] sure! So I usually check https://logstash.wikimedia.org/app/dashboards#/view/ORES but I noticed from turnilo that some UAs are not displayed, like "WME/2.0 (https://enterprise.wikimedia.com/; wme_mgmt@wikimedia.org)" and others [16:57:07] I do see access logs with those UAs on the ores nodes' access logs [16:57:20] but I can't find them in kibana (and I am really bad with query syntax) [16:59:32] Found some WMEs: https://logstash.wikimedia.org/goto/187726db64b3ee870cfb55a6f6465fb0 [17:00:21] ah ok so all works! It was my bad query then [17:00:37] okok I'll try to see how to better organize my dashboard, thanks! [17:01:18] Not bad query I think. These are unparsed uwsgi logs. There's no user_agent field to aggregate on. [17:02:37] also the dashboard is excluding these logs because the "uri" field must exist per the applied filter. [17:03:32] cwhite: okok so IIUC it may be that the ORES filter in logstash doesn't catch them? [17:04:27] ah I just ran a test (like https://wikitech.wikimedia.org/wiki/Logstash#Writing_&_testing_filters) and indeed it fails [17:04:40] Not logstash the software component. The dashboard is excluding them with the "uri:exists" filter. [17:05:36] yes yes but what I meant is that the uri attribute should be present if the msg was compatible with the ORES filter, no [17:05:39] ? [17:06:05] yes, that grok filter [17:06:15] ack perfect, so something is off in there [17:09:47] one thing that I noticed is that all those logs have two ips serialized with a comma [17:10:34] sure enough, that would trip this grok filter up [17:12:37] ahhhhh [17:12:57] cwhite: what can I use to support both use cases? Like one IP or multiple ones? [17:15:22] Can either parse one out and ignore the other (using regex) or capture them both in a another match rule. [17:17:33] ok I'll try something and I'll add you to the review, I apologize in advance for any horror that I'll send :D [17:22:23] Sounds good! :)