[13:19:53] <ebernhardson>	 \o
[13:39:23] * ebernhardson is almost getting tired of reading and adjusting this report on highlighting by now...
[13:39:36] <ebernhardson>	 i was intending to finish yesterday, but i only mad eit about 60% through
[13:41:08] <inflatador>	 <o/
[13:42:54] <inflatador>	 starting the reimage of apifeatureusage2001 shortly
[13:43:55] <ebernhardson>	 current draft if anyone wants to take a look, suppose hoping to talk about it a bit at wed meeting: https://phabricator.wikimedia.org/P96020
[13:53:52] <inflatador>	 do y'all happen to remember where we turn on journald logging for opensearch? I know it's not on by default and the normal way of turning it on (ref https://github.com/opensearch-project/OpenSearch/blob/main/distribution/packages/src/common/systemd/opensearch.service#L39 ) isn't it
[13:54:09] <ebernhardson>	 hmm, not off the top of my head but can look around
[13:55:47] <inflatador>	 No worries, I can keep digging. It might be implicit somewhere outside of our config. 0lly pinged us about log spam, looks like we have a lot of entries related to 'safepoint'
[13:56:07] <ebernhardson>	 safepoint means garbage collection usually
[13:57:22] <inflatador>	 yeah, it's def GC-related, looks like trace level gc logging is making it into the journal somehow
[13:58:20] <inflatador>	 OK, that's in jvm options`-Xlog:gc+age=trace` but not sure how it gets to the journal yet
[13:58:25] <ebernhardson>	 inflatador: maybe Xlog:safepoint in modules/opensearch/manifests/instance.pp, gated by $gc_log flag
[13:58:33] <ebernhardson>	 with no file specified i think that goes to stdout
[14:00:05] <inflatador>	 it's fine to keep it on, we just need to prevent it from making it into the rsyslog hosts
[14:00:19] <ebernhardson>	 i think you could just point it to a file like Xlog:gc*
[14:00:27] <ebernhardson>	 but i don't know the details, would have to look it up
[14:05:30] <inflatador>	 No worries, I see we already have gc logs so I can try adding that option
[14:13:21] <inflatador>	 https://www.javacodegeeks.com/2026/05/reading-jvm-safepoint-logs-without-going-mad-a-practical-stop-the-world-diagnosis-guide.html has some good info
[16:07:09] <inflatador>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324763 CR for the jvm options change, going to test it on relforge/cloudelastic before I remove WIP
[17:35:08] <inflatador>	 ^^ OK, the above is ready for review. I tested it on relforge and it the logs are on longer going to the journal. The only concern I have is whether or not the other gc entries are getting clobbered and never making it into /var/log. safepoint logs about 300x as much as `trace` so we may have to wait and see
[17:35:36] <inflatador>	 300x as often, that is
[17:48:13] <ebernhardson>	 i don't know how much we need the safepoint logs. I don't think i've ever looked at them. But maybe they are occasionally relevant
[19:43:16] <inflatador>	 ACK, I'll update the patch to get rid of them then
[20:32:31] <inflatador>	 in unrelated news, apifeatureusage looks to be working on bookworm with a little massaging
[20:33:55] <inflatador>	 logstash is showing some errors posting the bulk API, but that seems true on the bullseye apifeatureusage host as well
[21:14:34] <inflatador>	 I updated https://wikitech.wikimedia.org/wiki/Search_Platform/Accountability to remove references to WDQS and change Elastic to cirrussearch, feel free to have a look and change anything
[21:37:28] <ebernhardson>	 nice!
[21:49:00] <inflatador>	 yeah, I dunno how much longer apifeatureusage can limp along. we'll probably end up making a logstash image and moving to k8s at some point