[13:32:24] <ebernhardson>	 \o
[13:57:23] <inflatador>	 <o/
[14:21:49] <inflatador>	 Low priority, but I created a ticket for the apifeatureusage stuff T434906
[14:21:50] <stashbot>	 T434906: Check the health of apifeatureusage indices - https://phabricator.wikimedia.org/T434906
[14:22:16] <ebernhardson>	 thanks! the trackpad distraction meant i didn't get to it yesterday, but still intending to look into it
[14:22:48] <inflatador>	 yeah, no hurry. If I can help let me know
[15:02:36] <ebernhardson>	 meh, i can't login to apifeatureusage machines :P  Might not matter
[15:02:45] <ebernhardson>	 also not sure what i was looking for, was just poking around
[15:04:58] <inflatador>	 I can make a PR to get you added to whatever group if you like. Or I can post the logstash config. From what I can tell, it's just running logstash to filter out PII from the apifeatureusage extension logs and format for elastic
[15:05:44] <ebernhardson>	 inflatador: i'll let you know, i suppose i'll start from the other end on mediawiki side and verify how the data is getting out.  It could be as simple as a logging channel misconfiguration or some such (IIRC mediawiki writes those as application logs to a special channel)
[15:06:20] <inflatador>	 ebernhardson ACK np, hit me up if you want a peek at anything
[15:20:08] <ebernhardson>	 oh curious, i can't log into centrallog either :S  I thought that would be like the old mwlog hosts where deployers can log in.  mwlog1002 is still in site.pp, but doesn't exist afaict
[15:36:16] <gehel>	 Hey! T349481 has been opened for quite a long time. I'm not entirely clear if this will be moved forward by Search Platform or by DPE SRE. In case this is on us, could you add some context to the ticket as to what you need from us?
[15:36:17] <stashbot>	 T349481: Create SLI/SLO for search index inconsistencies - https://phabricator.wikimedia.org/T349481
[15:37:28] <ebernhardson>	 gehel: hmm, i'll have to ponder. clearly it's been awhile :)  Thanks for the heads up
[15:47:03] <gehel>	 it doesn't look to be urgent, but I'm trying to do some cleanup.
[15:59:34] <ebernhardson>	 inflatador: when you have a chance can you grab a tcpdump on apifeatureusage2001 for requests to search.svc.codfw.wmnet:9200/_bulk? It looks to be logging an http 400 (bad request) several times a minute but no hints about what the request/response is
[15:59:47] <ebernhardson>	 oh, actually that is probably https :S 
[16:00:17] <ebernhardson>	 oh no, if it's 9200 it should be plaintext
[16:05:56] <inflatador>	 Sure, taking a look now
[16:06:16] <ebernhardson>	 if you can copy it to a deploy host or elastic host where i can download it would be awesome
[16:06:23] <ebernhardson>	 s/elastic/cirrus
[16:07:03] <inflatador>	 sure, will put it up on `deploy1003` after I grab a few minutes...lemme know how long you need it to run
[16:07:18] <ebernhardson>	 inflatador: a minute or three should be plenty, logstash shows it logging multiple errors a minute
[16:15:01] <inflatador>	 ebernhardson OK, it says 60 packets captured, hope that's enough. It's in your homedir on `deploy1003`
[16:15:54] <ebernhardson>	 inflatador: thanks! Yup it has the 400 bad request's
[16:16:14] <ebernhardson>	 yup, it's the 2.x upgrade: https://phabricator.wikimedia.org/T384335⏎
[16:16:17] <ebernhardson>	 wrong paste...
[16:16:23] <ebernhardson>	 {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
[16:16:30] <ebernhardson>	 will figure out how to remove that
[16:18:56] <inflatador>	 ebernhardson ACK, it might be https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/files/apifeatureusage/templates/apifeatureusage_7.0-1.json ?
[16:19:13] <inflatador>	 hmm, maybe not
[16:19:46] <inflatador>	 I know that with vector, you had to explicitly tell it to use the elasticsearch 8 API or it would automatically send that field...wonder if Logstash does the same
[16:20:50] <inflatador>	 `conf.d/95-output-opensearch-apifeatureusage-search-svc-codfw-wmnet.conf` does have `document_type   => "_doc"` in it
[16:20:59] <inflatador>	 I think that comes from a Puppet module, 1 sec
[16:22:13] <inflatador>	 yeah, might be a matter of removing https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/apifeatureusage/logstash.pp#143
[16:22:30] <inflatador>	 worth a try, anyway...I'll get a patch started
[16:22:50] <ebernhardson>	 yea can't hurt to try, not going to break it anymore than it is :)
[16:26:59] <ebernhardson>	 looks like it will default to undef which the template checks and will skip defining document_type.  I'm not 100% clear though, because that potentially already gets ignored and imputed as _doc (opensearch 1.x never accepted arbitrary values there).  The test suite still allows passing through depending on maximum_seen_major_version
[16:27:19] <inflatador>	 OK, CR up here. I'm heading to lunch in a few min but can merge after I get back https://gerrit.wikimedia.org/r/c/operations/puppet/+/1325928
[16:30:38] <ebernhardson>	 based on my reading of the current (master) branch of the logstash-output-opensearch plugin, i think that will work.  But i dunno what version we have :P
[16:34:04] <ebernhardson>	 i suppose a separate question is if we should be swapping this over to https and 9243.  While it's very convenient that this was plain http and we could just read the responses, i think our current best practices say we should be eliminating plain http
[16:36:28] <ebernhardson>	 in theory we can use eBPF, but no clue how much of that we can actually use with the way prod hosts are setup
[16:36:33] <ebernhardson>	 (and i've never used it :P)
[17:29:11] <inflatador>	 back, merging ^^
[17:38:32] <inflatador>	 OK, merged/puppet-merged/logstash restarted. I'll try another pcap and see what we get
[17:46:36] <ebernhardson>	 inflatador: can watch logstash, i found the other logs looking for host.name:apifeatureusage* in ECS logs
[17:47:56] <inflatador>	 ebernhardson cool. The sheer number of packets in the new pcap compared to before leads me to believe it's working
[17:48:02] <ebernhardson>	 although right now it looks like the restart takes so much volume it's less clear,  but i think they've gone away
[17:48:44] <ebernhardson>	 yea `host.name:apifeatureusage2001 AND log.level:ERROR` is probably a better query, can see them fall off at 17:34
[17:49:26] <ebernhardson>	 i suppose the big question...yup i see an apifeatureusage-2026.08.14 in eqiad and codfw
[17:49:33] <inflatador>	 {◕ ◡ ◕}
[17:51:40] <inflatador>	 Nice! I'll update the ticket and close it out unless you have any objections
[17:51:47] <ebernhardson>	 nope seems good to me
[18:01:59] <ebernhardson>	 a few mjolnir train tasks failed this week due to too much memory usage, went ahead and bumped up the resource requests for the train takss
[18:02:04] <ebernhardson>	 *tasks
[18:04:19] <inflatador>	 ACK, if I need to review any values.yaml changes or anything LMK
[18:10:26] * ebernhardson finds it amusing that when asking claude code to use an LLM to keyword-ify the multilingual dataset...it says "No API key, but the claude CLI is available and scriptable"
[18:10:31] <ebernhardson>	 like no dude...you are the claude CLI
[23:23:13] <ebernhardson>	 hmm, ran more analysis with better keyword-ified queries...i hadn't realized how much mdeberta abstains when we switch to keyword form
[23:23:39] <ebernhardson>	 and in opensearch we have to choose one highlighter per field, so will have to do something silly like copy_to the passage into a second field to allow both semantic and lexical highlights
[23:24:04] <ebernhardson>	 or maybe instead of copy_to just two sub fields (similar to text vs text.plain)