[13:39:34] <inflatador>	 greetings
[13:52:24] <gehel>	 o/
[14:05:11] <inflatador>	 <o/
[14:10:10] <inflatador>	 Sorry for being late, my cat's veternarian is apparently more popular than the Beatles. There were 15 people in front of me(!!!)
[15:11:35] <ebernhardson>	 \o
[15:13:41] <inflatador>	 <o/
[15:22:00] <inflatador>	 any objections to me getting rid of the stretch nodes in deployment-prep? I'm not deleting anything until next week, but the cluster is healthy with 2 stretch and 3 bullseye nodes ATM.
[15:24:49] <gehel>	 inflatador: since we don't have a load balancer in WMCS, the ES nodes are configured directly in the Mediawiki config. That config needs to be updated before switching off all nodes.
[15:24:57] <gehel>	 ebernhardson: can probably help
[15:25:42] <inflatador>	 Thanks! No big rush, just need to know stuff like that before we move ahed
[15:25:45] <inflatador>	 or ahead
[15:32:34] <ebernhardson>	 inflatador: in the operations/mediawiki-config repository there is a file, wmf-config/LabsServices.php. There are search-* keys in there to be changed
[15:33:15] <ebernhardson>	 the annoying part will be deploy, that repo is shared between beta cluster (previously called labs) and production, labs will auto-deploy but it also has to be deployed to prod so future deployers don't wonder what the undeployed patch does
[15:34:11] <inflatador>	 sounds about right ;(
[15:34:14] <ebernhardson>	 inflatador: you can plug the gerrit id into this url and get deployment commands: https://deploy-commands.toolforge.org/bacc/788773
[15:35:28] <ebernhardson>	 usually no-deploy fridays, but no-ops to the LabsServices.php file will be fine
[15:35:40] <ebernhardson>	 err, noop's, no-ops sounds like something else :P
[15:40:19] <inflatador>	 No-ops is how much ops I typically do on a Friday ;P
[16:08:46] <inflatador>	 workout , back in ~35
[17:01:23] <inflatador>	 back
[17:46:20] <inflatador>	 lunch, back in ~30
[17:47:35] <ebernhardson>	 signs of a good day :P An exception has occurred in the compiler (14.0.2). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program, the following diagnostic, and the parameters passed to the Java compiler in your report. Thank you.
[17:51:10] * ebernhardson sets everything to 13 and hopes it works better :P
[17:53:59] <ebernhardson>	 nope, apparently somewhere between 7.6.2 and 7.7.0 need to switch from jvm 11 to 14
[17:56:42] <ebernhardson>	 the way the tests ran changed between those as well, and the location of integration test files.  Elastic changes too many things to make this convenient :S
[18:03:49] <ebernhardson>	 oh wow never mind, i'm an idiot and checked out 6.6.2 instead of 7.6.2 :P Of course so much moved
[18:13:55] <razzi>	 Hi ebernhardson inflatador an-airflow1001 is on my list of hosts to reboot for a kernel update - is there a good time to take it offline for a few minutes or can I reboot it whenever?
[18:49:24] <inflatador>	 razzi I'll defer to e-bernhardson on that one, not sure myself
[18:49:49] <razzi>	 Cool, I'll let it simmer until next week inflatador 
[18:50:12] <inflatador>	 np. Also, good to hear from ya! Have a great weekend
[18:50:25] <ebernhardson>	 razzi: annoyingly it's not obvious, basically we can restart it any time the only jobs running are wait_for_* jobs
[18:50:35] <ebernhardson>	 checking if there is a cli way to see
[18:54:03] <ebernhardson>	 razzi: no, not seeing anything accessible from CLI :S  With the UI forwarded to localhost:8778 can use: http://localhost:8778/admin/taskinstance/?flt0_state_equals=running&flt3_task_id_not_contains=wait_for_ 
[18:55:50] <ebernhardson>	 razzi: it says its running hyperparam for the mjolnir dag right now, it's fine to kill that. It will start over but no big deal
[18:56:15] <razzi>	 ebernhardson: ok I'm going to go for it, it'll only be a couple minutes of downtime
[19:07:32] <razzi>	 Reboot is done, looks like airflow is back up and running just fine
[19:08:49] <ebernhardson>	 razzi: sounds good, thanks
[20:06:29] <inflatador>	 OK, here's the mediawiki-config PR as discussed earlier https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/789877
[20:34:38] <ebernhardson>	 inflatador: lgtm
[20:35:41] <inflatador>	 Thanks for taking a look. Will hold up on merging though, call me superstitious
[20:37:50] * inflatador bookmarks deploy-commands page
[21:02:14] <RhinosF1>	 inflatador: if you haven't deployed before, I'm sure one of the lovely B&C team will help during the week
[21:02:36] <RhinosF1>	 There's a calendar on wikitech.wikimedia.org/wiki/Deployments
[21:05:14] <inflatador>	 RhinosF1  thanks for the advice , will read up and get in touch next wk
[21:05:25] <RhinosF1>	 inflatador: no problem
[21:05:55] <RhinosF1>	 The deployers are very friendly people, they definitely won't mind
[21:06:08] <RhinosF1>	 They are some training sessions on Thursdays
[21:07:47] <inflatador>	 That might be good. The only delpoys we really do regularly is WDQS
[21:39:27] <inflatador>	 OK, sent an email to releng . See ya next week, folks
[21:57:40] <ebernhardson>	 and the unsatisfying answer, "The Query string parser was not delegating the construction of wildcard/regex queries to the underlying field type." from https://github.com/elastic/elasticsearch/commit/633e00ae505b7dd06d34f173a9a566a68d3fb4e6 is the proximate start of d*e*a*d*l*y*w*i*l*d*c*a*r*d  not being killed as expected
[21:57:56] <ebernhardson>	 so, somehow delegating to the correct field type made it stop failing this check...needs more investigation :P
[22:16:41] <ebernhardson>	 there was another unrelated change that mixed up debugging, in 7.6.2 it would fail even if the referenced field doesn't exist, starting in 7.7.0 they optimize away the query to non-existent field before checking