[07:23:30] <ryankemper>	 inflatador: data transfers complete. i'll leave the adding to etcd and pooling for tomorrow
[08:41:00] <gehel>	 ryankemper: Thanks for the work on the Graph Split! It's good to see that we're not serving the full graph anymore!
[13:52:02] <ebernhardson>	 \o
[13:52:27] <inflatador>	 ryankemper NICE! I just checked the logs and the categories reload takes ~90m...just leaving that as a benchmark 
[13:52:30] <inflatador>	 .o/
[16:29:19] * ebernhardson wonders if we should be caching deepcat filters,  would improve latency of scrolling on commons mediasearch
[16:40:58] <inflatador>	 Apologies if this is a dumb question, but how would we implement deepcat caching?
[16:43:15] <ebernhardson>	 inflatador: well, one way would be a http cache between mediawiki and blazegraph, another would be to use the application cache in mediawiki 
[16:43:40] <ebernhardson>	 it's essentially going from a category name to a set of categories contained, with an expensive call in the middle
[16:44:43] <inflatador>	 quick CR for adding the new cirrussearch names to site.pp: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1143865
[16:46:24] <inflatador>	 Another dumb question: do we cache WDQS queries now?
[16:46:54] <ebernhardson>	 inflatador: i don't think so
[16:49:51] <inflatador>	 I was just thinking whether or not a cache for deepcat would have broader utility. I also have no idea how you'd implement that anyway
[16:52:04] <ebernhardson>	 it would probably just be some cache wrapping code in the mediawiki bits, i'm only really thinking of it because i'm looking into a bug report about deepcat timing out
[16:52:21] <ebernhardson>	 curiously, i'm actually having a difficult time getting it to regularly time out. One category will sometimes get a timeout, but not reliably
[17:08:38] <inflatador>	 we've had 2 alerts for eqiad omega today. They cleared almost immediately, but it's a little weird. Let me check their pybal config
[17:09:38] <inflatador>	 can't tell from https://config-master.wikimedia.org/pybal/eqiad/search-omega-https , but my best guess is that I accidentally put a non-omega host in rotation
[17:16:19] <ebernhardson>	 lets see, omega is 9400. in a quick test requesting 9400 from all the hosts in the config-master list repond back with cluster name of omega
[17:23:08] <inflatador>	 hmm, maybe not then
[17:36:27] <inflatador>	 yeah, I'm not seeing any problems there either...back to the drawing board, I guess
[17:48:33] <inflatador>	 ebernhardson I changed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1143633 pretty significantly. One thing I'm not sure about it why relforge had`search.svc.eqiad.wmnet` as one of its cert domains, would it break anything if we get rid of that? ref https://puppet-compiler.wmflabs.org/output/1143633/3773/relforge1008.eqiad.wmnet/index.html
[17:49:03] <ebernhardson>	 inflatador: hmm, no sounds like a mistake from when it was created 
[17:49:52] <ebernhardson>	 :q
[17:51:17] <inflatador>	 all good, just wanted to make sure we weren't actually using it
[17:51:55] <ebernhardson>	 inflatador: i don't think we will have search-<name>.svc.<dc>.wmnet,  should we?
[17:52:04] <ebernhardson>	 only search-<name>.discovery.wmnet
[17:57:33] <inflatador>	 ebernhardson there is a `search.svc.codfw.wmnet` SAN on all the CODFW hosts' certs currently. Are you saying it won't be necessary anymore once we're using discovery records?
[17:59:08] <inflatador>	 I think discovery works by pointing `x.discovery.wmnet -> x.svc.${dc}.wmnet`
[17:59:38] <inflatador>	 Looking at wdqs1022 I see `wdqs-main.svc.eqiad.wmnet` and `wdqs-main.discovery.wmnet` on its cert
[18:01:38] <inflatador>	 lunch, back in ~40
[18:41:58] <ebernhardson>	 i think you're right, we do need them
[18:48:52] <inflatador>	 back
[18:51:23] <ebernhardson>	 it does make it a bigger task, we essentially need to https://wikitech.wikimedia.org/wiki/LVS#Add_a_new_load_balanced_service, but some parts are already in place
[18:56:55] <inflatador>	 Oh boy, LVS ;P
[19:11:34] <ebernhardson>	 i wonder how dumb it would be to make search-chi.svc.<dc>.wmnet a CNAME for search.svc.<dc>.wmnet and using the current lvs setup
[19:11:48] <ebernhardson>	 probably asking for trouble :P
[19:11:49] <inflatador>	 I had to redo that site.pp patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/1143888 
[19:14:21] <inflatador>	 Anything that avoids touching LVS is a win ;) . I'm trying to find the presentation about PyBal's replacement (https://wikitech.wikimedia.org/wiki/Liberica ) . I vaguely remember something about opportunities to control it thru etcd, similar to k8s service discovery/autoconfig  
[19:15:53] <inflatador>	 I'll hit up Traffic next wk and see where they're at
[19:16:02] <ebernhardson>	 awesome
[19:21:57] <inflatador>	 I'd like to use envoy, but if the choice is between an mwconfig patch or multiple patch sets and multiple turnup calls with traffic for LVS, well...
[19:23:05] <inflatador>	 on the other hand, we just had to do this for WDQS so maybe we should strike now before we forget everything again ;)
[19:55:05] <ebernhardson>	 pondering things, i suspect a cname will work fine.
[20:10:15] <inflatador>	 yeah, maybe it depends on how the gdnsd DYNA records work? https://man.archlinux.org/man/gdnsd.zonefile.5.en#DYNA ?
[20:10:57] <inflatador>	 or maybe we use DYNC
[20:11:15] <inflatador>	 probably worth asking b-black, but we'll probably be OK
[20:19:02] <inflatador>	 I wonder if those omega alerts are related to some kind of rate-limiting or blocking. my shard checking script craps out after a few `GET _cat/shards` to search.svc.eqiad.wmnet from CODFW
[20:47:59] <inflatador>	 ryankemper just updated the description on T388610 , but FYI 1053 and 1054 are having problems reimaging. I would've liked to use their capacity during the migration but we don't really need them, so we can decom
[20:48:00] <stashbot>	 T388610: Migrate production Elastic clusters to Opensearch - https://phabricator.wikimedia.org/T388610
[20:49:35] <inflatador>	 I downtimed them both 'till next week
[21:05:59] <inflatador>	 ryankemper I'm heading out a bit early. Feel free to work on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1143897 if you have time. If not, have a great weekend!