[08:51:34] weekly update published: https://wikitech.wikimedia.org/wiki/Search_Platform/Weekly_Updates/2023-06-30 [09:32:55] re: lucene regex and character classes, I bet the reason is that the regex engine has to materialize all the codepoints of the character class in order to determinize it, something that a non-determinizing regex engine does not have to do can simply access its unicode info (e.g. java would call Character.isLowerCase()) while applying the automata [09:36:05] for us I believe that if a state has too many transitions we simply ignore it we could perhaps support "some" well-known and small classes at the optimization level and consider the others as a ".", then the second pass with the java regex engine should work as expected, but that seems a non-negligible work. [10:11:16] Lunch [10:14:16] lunch 2 [12:51:44] o/ [13:43:16] dropping off kids, back in ~20 [15:01:02] \o [15:16:16] o/ [15:43:23] Time to start vacations! Enjoy while I'm away, and see you in 2 weeks! [15:52:04] Enjoy while you're away? How could we ;P [16:19:23] workout, back in ~40 [17:53:54] sorry, been back [17:54:31] ryankemper et al reformatting PR up for review if anyone has time to look: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/934595/ [17:59:29] inflatador: hmm, I don't really like its preference of `"` over `'`. Is that part of our style guide? and along the same lines is there a list of what arguments or config values it was given [18:01:52] ryankemper I didn't add any arguments to black, nor did I consult our style guide (we have a style guide?). But I'm happy to have a larger discussion if need be. My main goal is to minimize linting errors that constantly seem to crop out. Maybe I should address that in my IDE instead? [18:02:22] err...crop up that is [18:43:40] inflatador: We definitely do for puppet and the like, not sure if we have an explicit one for cookbooks per se. We at least have an implicit one via the linter. If there's a flag to make it not switch the `'`' to `"` can you run it again w/ that and push the patch? That'll make it easier to see the "actual" changes it's making wrt spacing or whatever [18:44:05] tl;dr: I'm in favor of auto-formatting as long as it does the right thing [18:44:31] ryankemper sure, will give it a shot. I'm not 100% committed to this approach, just trying to think about how to get the dev tools (IDE, CI) to work with me instead of against me ;) [18:47:48] ebernhardson: dcausse: how do we reference super_detect_noop in API calls to ES? Just by lang oder do we have to pass either an ID (stored script) or source? [19:01:27] pfischer: we always pass a script definition in the update, which references the registered script. Sec lemme find the cirrus bits [19:02:29] pfischer: essentially thats this bit: https://github.com/wikimedia/mediawiki-extensions-CirrusSearch/blob/d24013cd8667914e290598843b4400a1cf1b7811/includes/DataSender.php#L168-L173 [19:03:26] thats for tags, regular updates all flow through this function which does similar: https://github.com/wikimedia/mediawiki-extensions-CirrusSearch/blob/d24013cd8667914e290598843b4400a1cf1b7811/includes/DataSender.php#L646 [19:31:08] check out the funky whitespace in pybal! https://config-master.wikimedia.org/pybal/codfw/wdqs . We might want to revisit https://gerrit.wikimedia.org/r/c/operations/puppet/+/934403 as that's probably where the whitespace came from [19:32:25] I'm running a transfer from 2012-2020 to test https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/934602 ....probably should've picked a host that didn't already have data, but too late now ;( [19:36:19] ebernhardson: Thanks, I just got confused because super_detect_noop shows up twice when instantiating the Script. I think my PR is ready now. [19:49:23] * inflatador just heard a loud popping noise coming from somewhere in my room [19:49:52] I don't smell any ozone or see any smoke yet [19:50:13] pfischer: yea it's a little odd, due to the way the script is registered it is both the script engine, and the script to run inside the script engine iirc [20:13:14] wdqs2021 services are flapping...git-fat is still not being properly installed on Bullseye. Not sure if this is puppet's all-or-nothing approach (prom exporters are failing, which is failing puppet runs) or that we have to manually scap deploy on new hosts [20:13:22] Probably should add docs for this process [20:14:05] inflatador: memory's a bit rusty, what did we do to get git-fat working last time? did we manually install it? [20:16:18] ryankemper this should allow it https://gerrit.wikimedia.org/r/c/operations/puppet/+/920365 [20:16:47] It isn't installing...but puppet doesn't seem to be removing it on 2021. But Puppet is also failing due to the prom exporters [20:20:05] modules/prometheus/templates/initscripts/prometheus-blazegraph-exporter.systemd.erb is the unitfile template for the blazegraph exporters...will try and get a PR up for that 'conditionpathexists' logic we talked about earlier [20:22:25] inflatador: grabbing lunch rn but wrt your pybal question from earlier, we just need to set inactive=true and weight=0 on the new hosts (until they’re rdy ofc) [20:24:00] understood...we can look it when you get back. I just wondered why there's a bunch of blank lines at https://config-master.wikimedia.org/pybal/codfw/wdqs but it's not a huge deal overall [21:26:32] inflatador: the blank lines are there because of those hosts [21:38:26] ie there’s an invisible line for each host which has had the conftool-data set by puppet but hasn’t had the manual conftool command ran yet [21:53:38] ryankemper interesting, thanks for checking that out [22:14:08] wdqs2012 -> wdqs2021 xfer is still running in a tmux under bking@cumin1001 , should be done in ~30m or so. I used the new 'pool_action' flag https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/934602/1/cookbooks/sre/wdqs/data-transfer.py#48 [22:17:03] * ebernhardson should figure out how to build env's from an analytics machine, copying 500MB envs from my laptop is a pain :P [22:18:37] Left some notes up on my troubleshooting, happy weekend all! https://phabricator.wikimedia.org/T331300#8983005 [22:21:46] Ran the conftool commands. On second thought I think I just needed to set the one host (think it was 2022) that was appearing to inactive, rather than all of 2013-2022. oh well