[08:58:50] <gehel>	 Errand + lunch
[08:58:55] <gehel>	 And welcome back!
[09:19:40] <dcausse>	 wdqs throttling state seems to be maxing out, wondering if this could make sense to increase that a bit (currently at 2000 per node)
[09:21:12] <dcausse>	 3 months ago wdqs served ~7k requests/min, now it's more around 13k... :/
[09:22:41] <gehel>	 If I remember, the throttling state is an LRU. So it is likely that the evicted entries are for non current clients.
[09:23:44] <gehel>	 I don't think we have a metric telling us how old are the evicted entries. That would be interesting...
[09:41:39] <dcausse>	 yes it's access time based, there's a 15min expiration perhaps we can lower this too as an alternative and see
[09:47:02] <dcausse>	 the bump is kind of sudain too (around april 9), seems like a lot of load moved from codfw to eqiad...
[10:35:43] <dcausse>	 lunch
[12:42:15] <pfischer>	 gehel: Would you want/have time to have a look at the JVM parent POM MRs https://gitlab.wikimedia.org/repos/maven/wmf-jvm-parent-pom/-/merge_requests - if not, I’ll forward that to the JVM stewards.
[12:42:29] <gehel>	 looking
[12:58:10] <gehel>	 pfischer: I merged a few, but I have open comments on https://gitlab.wikimedia.org/repos/maven/wmf-jvm-parent-pom/-/merge_requests/21
[13:09:38] <pfischer>	 gehen: Thanks, looking
[13:58:21] <inflatador>	 <o/
[13:58:44] <dcausse>	 o/
[14:59:51] <ebernhardson>	 \o
[15:01:13] <dcausse>	 o/
[16:05:51] <ebernhardson>	 inflatador: i was wondering if we can do the envoy bits before switching over to migrating the eqiad cluster
[16:22:46] <inflatador>	 ebernhardson I haven't though about it much, but that's a good idea. We have a couple of old patches https://gerrit.wikimedia.org/r/c/operations/puppet/+/838182 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1053789 . Are there any I'm missing?
[16:23:14] <ebernhardson>	 inflatador: not directly related, but also https://gerrit.wikimedia.org/r/c/operations/puppet/+/1132692
[16:23:36] <ebernhardson>	 which needs a .deb build
[16:24:39] <inflatador>	 ebernhardson ACK , will look at that today
[16:39:00] <dcausse>	 inflatador: relatedly we might perhaps want to re-enable completion index rebuild before shifting traffic (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1139518)
[16:44:37] <ebernhardson>	 dcausse: having an odd time with intellij, it want to consider the delomboked classes as duplicates of the real classes. Any ideas? Not sure what changed that caused this to start complaining...
[16:44:57] <ebernhardson>	 I've tried un-marking opensearch-extra-common/target/generated-sources/delombok as a source, so it's exluded, but didn't seem to help
[16:45:53] <ebernhardson>	 also tried deleting .idea and letting it re-import, also no help :S
[16:46:20] <dcausse>	 ebernhardson: looking at my setup
[16:47:06] <ebernhardson>	 maven doesn't have a problem with it and i can do a full build, but it means i can't use the run single test options in idea
[16:47:47] * dcausse opens the extra plugin and sees the duplicates... it used to work well I'm sure
[16:47:50] <ebernhardson>	 i guess i didn't delete all the target dirs, lemme see if rebuilding them makes a difference
[16:47:55] <ebernhardson>	 i'm sure it used to work as well :)
[16:48:43] <ebernhardson>	 dcausse: sigh, i shuld have tried that earlier. Deleting */target made it start working
[16:50:44] <dcausse>	 hm... but they might come back?
[16:51:01] <ebernhardson>	 yea it might
[16:51:03] <ebernhardson>	 well, kinda. now it says unknown char_filter [pattern_replace], which also works when maven runs the tests :S but it's a different error :)
[16:51:27] <dcausse>	 an answer on stackoverflow suggests that the default output for delombok might not be wise (generate-sources) I wonder if we tuned that in the past
[16:51:50] <ebernhardson>	 yea i saw that, but thought that since it's been working for years it must be something else
[16:52:39] <ebernhardson>	 oh, and now it's back...so running maven made the target/ folders and causes the error
[16:59:47] <inflatador>	 dcausse ebernhardson ryankemper (and anyone else), I made a Google doc to list all the things we should do before starting EQIAD. Feel free to edit/add https://docs.google.com/document/d/1JQxqjaS-p_sEkUelFXkXsPl0_Mfj7zYtyRf1GKaz82w/edit?tab=t.0
[17:07:46] <dcausse>	 hm not finding great workarounds... we might simply disable delombok, imo it's mostly useful for lib projects where the javadoc could make sense
[17:08:58] <dcausse>	 whenever intellij attempts to reload the maven project it'll re-add everything under generated-sources
[17:09:20] <ebernhardson>	 so curiously, the reason pattern_replace wasn't available is that i needed to add custom integration to AbstractPluginIntegrationTest which provides the pattern replace char filter. The wierd thing is that it works fine from maven, but intellij is apparently running the tests a different way which starts the embedded elasticsearch in a different way? Unexpected
[17:10:25] <ebernhardson>	 the elastic forum simply says, regarding embedded elasticsearch in tests,  "It's not possible to do so". But apparently we did it anyways :P
[17:10:31] <dcausse>	 weird... it's been a long time but integration tests were a bit fidly even with elastic
[17:10:40] <dcausse>	 :)
[17:11:28] <ebernhardson>	 one oddity i haven't decided on yet...escaping
[17:12:02] <ebernhardson>	 In proper regex, we could have the text content 'Start^Middle$End',  And you could search for that with 'Middle\$'. I figure we should support that somehow?
[17:12:27] <ebernhardson>	 but the thing is, then what about 'Middle\$End$',  
[17:12:54] <ebernhardson>	 do we worry about these oddities? Or do we accept a bastardized regex syntax
[17:16:32] <dcausse>	 ebernhardson: what would you do with 'Start^Middle$End'? replace ^ and $ with unfindable chars?
[17:18:03] <ebernhardson>	 dcausse: I'm not really sure yet, the right solution goes back to parsing it properly, but i spent friday playing with antlr and their xsd-regex...I'm not sure that's going to be a great solution
[17:18:18] <dcausse>	 oh ok
[17:19:10] <ebernhardson>	 i suppose they could be replaced with direct char_filters, worth trying at least
[17:19:37] <ebernhardson>	 but in proper regex they would be escaped as \^, not a bare ^ in the pattern. Might surprise users
[17:19:57] <dcausse>	 yes they should but the second pass will discard the results
[17:21:11] <dcausse>	 to find a title like "Start^Middle$End" you have to write insource:/Start\^Middle\$End/
[17:21:48] <dcausse>	 s/insource/intitle
[17:23:13] <dcausse>	 hm.. it's already broken somehow... https://en.wikipedia.org/w/index.php?search=intitle%3A%2F%24Libra%2F&title=Special%3ASearch&ns0=1
[17:23:23] <dcausse>	 this should not match
[17:24:49] <ebernhardson>	 dcausse: it does because in lucene regex $ and ^ are unparsed, they are treated as the provided char and not a special thing
[17:25:42] <ebernhardson>	 although i guess it at least doesn't care if the user does escape them, '\$Libra` matches too
[17:26:12] <dcausse>	 hm... somehow I thought that the second pass was using the java regex engine...
[17:26:17] <dcausse>	 but no
[17:26:21] <ebernhardson>	 dcausse: nope, it's the automata too
[17:26:54] <ebernhardson>	 i suppose as long as lucene treats $ and \$ the same, it wont be quite as surprising to users
[17:29:09] <ebernhardson>	 Or we could go for craziness, add a variety of test cases and see if something like `(?<!\\)(?<=^|\()(\^)(?![^\[]*\])` will appropriately replace the ^ anchor, but not when escaped or inside a character class
[17:29:24] <dcausse>	 :)
[17:29:42] <ebernhardson>	 (thats what chatgpt thinks will work, i have no clue :P)
[17:30:58] <dcausse>	 to double check you could ask it in another context "what can of problem this regex <insert regex here> is trying to solve?" :)
[17:31:11] <dcausse>	 s/can/kind/
[17:33:11] <ebernhardson>	 Per claude, In plain language, this regex matches a caret symbol that:
[17:33:37] <ebernhardson>	 Is not escaped (not preceded by a backslash), appears either at the start of a line or immediately after an opening parenthesis, is not inside square brackets
[17:33:39] <ebernhardson>	 so, not terrible
[17:55:14] <ebernhardson>	 actually poking at maven output...i suspect SourceRegexQueryIntegrationTest isn't being run by maven
[18:00:52] <dcausse>	 :/
[18:12:35] <dcausse>	 dinner
[18:14:46] <inflatador>	 lunch, back in ~40-50
[18:30:00] <ryankemper>	 inflatador: patch to make query.wikidata.org requests route to `wdqs-main` up here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1139531
[18:30:10] <ryankemper>	 this is probably the simplest way to do the cutover
[18:30:42] <ryankemper>	 I also fixed up this old unmerged data-xfer cookbook patch of mine, hopefully it passes CI https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1097552
[18:59:16] <ryankemper>	 and one more patch, this will hopefully set the cluster to `wdqs-legacy-full` so that we can view it separately in the wdqs dashboard https://gerrit.wikimedia.org/r/c/operations/puppet/+/1139537
[19:11:11] <inflatador>	 back
[19:18:35] <inflatador>	 ryankemper I just +1'd the first patch. I have to go pick up my wife but will look at others when I get back. Hope your leg is feeling OK!
[19:20:53] <ryankemper>	 inflatador: cool. i've got an ortho appointment during pairing time so no pairing for me today
[19:44:45] <ebernhardson>	 may have something working here, using simple looping over chars and remembering some state instead of a crazy regex
[19:45:42] <ebernhardson>	 have a test case with queries like `(^|qqq)abc` and `Middle\\$` where we run both the java Pattern class and the regex query and verify same results from both
[19:46:24] <ebernhardson>	 probably someone can find edge cases that fail, but it's closer :P
[19:46:45] <ebernhardson>	 In theory the same bit of code could perhaps add support for character classes like \s or \S
[19:53:29] <ebernhardson>	 one awkward bit is that there are now some exceptions that will throw with the mutated string instead of the user-provided string (TooComplexToDeterminize, maybe others)...but probably fine
[20:41:30] <inflatador>	 we should probably make a cumin alias for master-eligibles at some point
[21:17:23] <inflatador>	 2 hosts left in CODFW!
[22:07:35] <inflatador>	 hmm, somehow didn't get the `opensearch-madvise-dbgsym_0.2_amd64.deb` file created when I built the madvise deb...checking
[22:11:44] <inflatador>	 ah, my playbook isn't matching its filename
[22:17:01] <inflatador>	 OK, that's published: https://apt-browser.toolforge.org/bullseye-wikimedia/component/opensearch13/ . Puppet's next
[22:24:07] <ebernhardson>	 nice! thanks
[22:24:52] <ebernhardson>	 inflatador: it should be fine since we dont use `ensure => latest`, but note that the puppet code only updates the opensearch call to provide the extra argument, and not the elasticsearch ones
[22:25:15] <ebernhardson>	 but unless we reimage an elasticsearch instance to elasticsearch, they should never install the updated .deb
[22:26:00] <ebernhardson>	 also i think i've worked out a thing to support char class shorthands in the on-wiki regex...but needs more testing
[22:26:23] <ebernhardson>	 (wasn't part of the ticket, but it seemed semi-obvious how to extend the anchor support to do this too)
[22:26:30] <inflatador>	 It's a separate package for opensearch, so I don't think elastic hosts would get it
[22:26:37] <ebernhardson>	 oh right, that works too
[22:28:14] <inflatador>	 I installed the new package on cloudelastic1009 and replaced the script by hand so we can test
[22:28:28] <inflatador>	 you probably did that already, but humor me ;)
[22:28:37] <ebernhardson>	 it's always worth verifying :)
[22:29:53] <inflatador>	 andd....works, at least as far as I can tell ;)
[22:30:44] <inflatador>	 ebernhardson can you remove your -1 from https://gerrit.wikimedia.org/r/c/operations/puppet/+/1132692 or is it too late? Probably should've asked before I merged ;(
[22:31:18] <ebernhardson>	 indeed it's hard to know 100% sure, it doesn't print on a per-file basis. if you have `strace` you can verify that it's actually doing something to the files by seeing what strace says
[22:32:25] <ebernhardson>	 oh, it does print
[22:32:34] <inflatador>	 I ran it from the cmd line and I got file output
[22:32:38] <ebernhardson>	 yea, as long as it prints a bunch of files that it adjusted, it should be working
[22:32:50] <ebernhardson>	 (really it's memory maps and not files it's adjusting, but same difference in context)
[22:36:04] <ebernhardson>	 some day we can get rid of it, i didn't check which version of 2.x has it, but the newest lucene has some support for settings the memory maps to random access
[22:39:35] <inflatador>	 hmm, relforge puppet is not happy after the update. I'm going to roll back and we can kick it again tomorrow
[22:39:50] <inflatador>	 error here: https://phabricator.wikimedia.org/T390592#10774892
[22:40:35] <ebernhardson>	 oh, silly me...thats something i would have expected a very generic test case (or i guess PCC?) to catch. I'll update it, but i'm also hitting the end of my day. Will do it in the morning
[22:41:08] <inflatador>	 yeah, no worries. It's harmless either way, I'm only reverting to prevent Puppet alerts