[11:24:00] <dcausse>	 lunch
[13:21:02] <inflatador>	 <o/
[13:28:44] <dcausse>	 o/
[13:41:47] <ebernhardson>	 \o
[13:43:02] <dcausse>	 o/
[13:51:07] <inflatador>	 .o/
[13:59:05] <ebernhardson>	 forcemerge took forever...but turns out a full optimize to single segment for frwiki brings it from ~78qps to 123qps
[13:59:31] <ebernhardson>	 latencies from ~190ms down to ~120ms as well.  But that wont play well with weekly batch updates
[14:01:43] <dcausse>	 ebernhardson: available to test cirrus' perfs? 
[14:01:47] <ebernhardson>	 dcausse: yup
[14:02:16] <dcausse>	 we're in https://meet.google.com/szi-shwu-xhc?authuser=0
[16:41:17] <ebernhardson>	 one potential problem with opensearch plugin adding the search header, it implements ActionPlugin.getRestHandlerWrapper, but the cluster refuses to start if more than one plugin implements that
[16:41:39] <ebernhardson>	 (not sure we have one, haven't gotten that far)
[17:59:23] <ebernhardson>	 finished the attempt to reproduce the "one hot node" in a local docker env with groups to limit cpus,  moving completion suggester away doesn't help much of anything.  In this test having dedicated coordinator nodes helped (because the queries coordinated by the overloaded node are slow)
[17:59:42] <ebernhardson>	 in prod though if one node is behind, thats <2% of the cluster
[18:02:56] <inflatador>	 I wonder how to gauge system requirements for coordinator nodes. I assume they could be weaker than the other nodes
[18:03:58] <ebernhardson>	 it should be relatively low resource requirements, but i'm not really sure how much.  
[18:04:45] <ebernhardson>	 It doesn't need a disk cache at all, but runtime cpu and memory i'm not really sure
[18:08:01] <inflatador>	 I'm guessing the coordinator code keeps track of latency of different hosts? Like they'd be able to steer away from the overloaded node assuming it had replicas elsewhere?
[18:08:58] <ebernhardson>	 The coordinator is basically the one that receives requests from mediawiki, asks data nodes to run the shard requests, then aggregates together the answers from the shard nodes into a response. It's basically a query coordinator.
[20:21:09] <ebernhardson>	 meh, turns out we can't change the hits array in the request logging schema. Has to be a new array: https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines#Complex_array_element_and_map_value_type_evolution_is_not_well_supported
[20:21:26] <ebernhardson>	 (to log highlights)
[21:03:38] <pfischer>	 ebernhardson: Ah, you wanted to augment the hits array items with the highlight info?
[21:04:11] <ebernhardson>	 pfischer: yea, we can still do it just there will be a second array and it carries some positional indexes: https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/merge_requests/87