[10:50:08] <dcausse>	 lunch+errand
[13:23:34] <inflatador>	 <o/
[13:35:00] <dcausse>	 o/
[14:07:10] <gehel>	 I was talking to Zbyszko. He mentionned: https://quepid.com/
[14:41:12] <dcausse>	 I thought this was non-free but double-checking it's actually under the apache licence
[16:07:24] <dcausse>	 sigh... running a very bad wildcard query against es7 does not look promising :/
[16:08:03] <ebernhardson>	 dcausse: oh yea, without the thing killing it it can be bad
[16:08:52] <dcausse>	 I thought they optimized it since this bug was initially reported but not enough, I can get it stuck for 10+sec just parsing the wildcard :/
[16:09:04] <ebernhardson>	 oh wow! 
[16:09:43] <ebernhardson>	 what can we do? I guess silly hacks like limit the number of wildcards in the input  to 2 or something
[16:09:47] <dcausse>	 might be a blocker if we don't find a workaround
[16:10:18] <dcausse>	 nick was pretty much againt this kind of hack but if that's all we have then we might have to do it :)
[16:10:49] <ebernhardson>	 i'm sure he was :) But if we can't get any guarantees from elastic, and we know they aren't going to issue a bugfix release to 7.10, i'm not sure what else is viable :)
[16:10:57] <inflatador>	 (╯°□°）╯︵ ┻━┻
[16:14:25] <dcausse>	 hm feels like they delegate leaf queries building to MappedFieldType instead of the lucene query parser base class... and the max determinized state is simply ignored...
[16:16:02] <dcausse>	 other than fixing it and doing a custom build I guess we're stuck with a hack at the cirrus level
[16:16:02] <ebernhardson>	 dcausse: yea thats all i could tell, it's not super obvious what's happening when they delegate to the field type ,but once they do it seems to no longer call the QueryParser stuff
[16:18:50] <ebernhardson>	 also wondering a little what the wildcard field does differently, i wonder if it's just prefix indexing + ngrams or some such under the hood?
[16:19:04] <ebernhardson>	 we probably don't do enough with wildcards for it to matter though
[16:20:19] <dcausse>	 no clue, but might not be applicable for tokenized text
[16:21:54] <dcausse>	 hm.. perhaps not...
[16:23:13] <dcausse>	 seems similar to our optimized regex but only supporting wildcard pattern matching
[16:23:35] <ebernhardson>	 hmm, yea that makes sense
[16:24:06] <inflatador>	 getting a ride home, back in ~45
[16:25:57] <dcausse>	 seems interesting to investigate
[16:27:33] <dcausse>	 but seems difficult to achieve before we get to es7
[16:32:05] <ebernhardson>	 hmm, so seems for now best bet is to hack an extra check in for *'s when making query_string queries i guess?
[16:33:21] <dcausse>	 yes I don't see any other viable options either :/
[16:36:37] <dcausse>	 the new parser has some heuristics IIRC, 3 wildcards max, \CirrusSearch\Parser\QueryStringRegex\NonPhraseParser should have something about this limitation
[16:37:12] <dcausse>	 not sure if anything in it can be reused easily in the old parser tho
[16:38:37] <dcausse>	 dinner
[17:40:15] <ebernhardson>	 hmm, suspect cpjobqueue isn't farming out jobs in beta cluster. But not entirely clear on how to check, `cpjobqueue` isn't found in beta-logs.wmcloud.org. Per docs, "The beta cluster has no working Grafana setup as of May 2022.". Should be fun :P
[18:06:08] <ebernhardson>	 seems like prometheus exists and is collecting data, querying is a bit painful. Can see kafka message counts increasing for some things like rsyslog-* topics, but cpjobqueue is silent.
[18:07:35] <inflatador>	 and back
[18:08:46] <ebernhardson>	 yea, cpjobqueue in beta seems to be in a crash-loop of Error: Local: Broker transport failure
[18:12:18] <inflatador>	 yuck
[18:14:53] <ebernhardson>	 on the upside, that means prod deployment is probably going to be fine, or at least search not getting indexed in beta cluster isn't a new regression we caused
[18:18:28] <ebernhardson>	 and the answer is...some config file somewhere needs to change deployment-kafka-main-5.deployment-prep.eqiad.wmflabs:9092 into deployment-kafka-main-5.deployment-prep.eqiad1.wikimedia.cloud
[18:18:43] <ebernhardson>	 this one exists inside a docker volume on the host, not sure how it got there yet
[18:23:06] <ottomata>	 ebernhardson:  ping hnowlan about this.  
[18:23:15] <ottomata>	 He fixed this for change prop already once
[18:23:23] <ottomata>	 perhaps he just didn't do it for cpjobqueue
[18:23:54] <ebernhardson>	 ottomata: excellent, someone might now how it works :) thanks!
[18:28:18] <ottomata>	 some1, with the emphasis on the number 1
[18:28:19] <ottomata>	 :)
[19:00:40] <inflatador>	 Dr apt, back in ~45
[19:03:10] <gehel>	 ebernhardson: 1:1 at https://meet.google.com/stp-swkd-iho
[19:07:59] <ebernhardson>	 gehel: doh! sec
[19:53:35] <inflatador>	 back
[20:15:29] <inflatador>	 well, my wife can't take me to pick up my car, so I have to get a ride share. back in ~45 
[21:09:43] <inflatador>	 back