[07:01:19] <pfischer>	 o/
[07:14:43] <dcausse>	 o/
[10:06:40] <dcausse>	 lunch
[13:17:58] <inflatador>	 <o/
[13:23:02] <dcausse>	 o/
[13:37:13] <pfischer>	 o/
[13:56:08] <ebernhardson>	 \o
[14:02:21] <ebernhardson>	 curiously all three reindexers died with a helmfile problem last night, but came back on restart this morning.  I wonder if they will need some sort of retries around that
[14:03:30] * ebernhardson could also make it not print the 4k events it already processed on startup
[14:07:40] <dcausse>	 o/
[14:07:55] <dcausse>	 :/
[14:08:43] <inflatador>	 .o/
[14:09:25] <ebernhardson>	 i wonder if i mucked something up with logging, because i can't find from the backfill logs why it failed :P  just the exit code 1 from all three separate helmfile invocations
[14:15:11] <dcausse>	 it's a flink deployment causing issues?
[14:17:57] <ebernhardson>	 sadly i dunno, my best guess is i overwrote the log :S
[14:18:23] <dcausse>	 :/
[14:18:25] <ebernhardson>	 the log filename comes from the backfill count, so if we start a backfill but fail to add a state event the next run overwrites it
[14:19:40] <ebernhardson>	 hmm, no i open it in 'at' mode, so it always appens
[14:19:54] <ebernhardson>	 hmm, thats more odd then that there is nothing interesting in the backfill output
[14:19:55] <dcausse>	 unsurure if this helps but I remember seeing dangling -backfill deployments last wednesday when deploying the flink-op with balthazer
[14:20:29] <ebernhardson>	 hmm, not sure. In theory the backfill shouldn't be able to finish without cleaning those up
[14:20:46] <ebernhardson>	 thats actually one of the weak points of this orchestration, if anything else cleans up it's stuff it doesn't know what to do
[15:06:35] <ebernhardson>	 hmm, curious...running a build localy with java 17 i can't use the regex Pattern class to match half a surrogate pair. but CI runs java 8 that matches the half pair (also works in 9, 10, 11, 12, and 14, maybe more. But not 17 :P)
[15:06:45] <ebernhardson>	 doesn't really matter, it was just an equivalence test, but curious
[15:14:24] * ebernhardson also notes Runtime.version() doesn't yet exist in java 8 either :S whatever...cant drop the test
[15:19:34] <inflatador>	 hmm, chatGPT 5 doesn't like reading my plantUML files for some reason
[15:19:45] <inflatador>	 I mean, besides the fact that they're plantUML ;P
[15:20:31] <pfischer>	 inflatador: how, does it display its dislike?
[15:21:43] <inflatador>	 It says "file may not be available" and then confabulates its own version of the diagram ;)
[15:22:14] <inflatador>	 https://chatgpt.com/s/m_68b1c59de9ec8191b9b948c5619c876e
[15:22:29] <pfischer>	 I’ve seen upload hicups in the past
[15:23:35] <pfischer>	 Did that work in the past? Maybe you need more of an agent to execute the plant UML compiler?
[15:23:56] <inflatador>	 No, I've never tried it before. I'm not too optimistic, but I thought it was worth a try
[15:37:24] <inflatador>	 hmm, I did eventually get it to write valid plantUML, but nothing coherent. Oh well
[15:39:18] <inflatador>	 I'll try the Miro GPT
[15:43:19] <inflatador>	 dentist apt, back in ~90m
[16:05:27] <Trey314159>	 Just got back from the eye doctor. got my eyes dilated, so i can't see very well. I'll be out for a couple of hours until they get back to normal and I can read again
[16:13:21] <dcausse>	 heading out, have a nice week-end
[16:51:49] <inflatador>	 back
[17:54:24] <inflatador>	 lunch, back in ~40
[19:25:43] <inflatador>	 Bah, I just realized we don't have a story for prometheus metrics on the Opensearch on K8s project. That means we'll either need to install https://github.com/opensearch-project/opensearch-prometheus-exporter or make some modifications to the chart :(
[19:31:21] <Trey314159>	 Ugh.. I still can't read well, though light no longer hurts. I'm going to throw in the towel and call it a day.
[19:31:38] <inflatador>	 Enjoy your Labor Day!
[19:34:30] <Trey314159>	 Thanks!
[19:35:45] <inflatador>	 seems easy enough to use the prometheus exporter, I'm gonna give that a try. Of course that means our metrics won't align exactly, but their exporter also has a dashboard: https://github.com/opensearch-project/opensearch-prometheus-exporter/tree/main/mixin
[19:52:58] <inflatador>	 ryankemper ^^ just a heads-up if you wanna do pairing later, we could look at the java-plugin-based prometheus exporter
[20:41:41] <inflatador>	 Also created T403301 to talk about releasing OpenSearch 3/helping 0lly with OpenSearch deb pkgs
[20:41:42] <stashbot>	 T403301: Discuss OpenSearch 3 roadmap/future improvements - https://phabricator.wikimedia.org/T403301
[20:55:29] <ryankemper>	 inflatador: might not make it back from hike in time
[20:55:40] <ryankemper>	 working on some wdqs puppet cleanup when I get back
[21:16:38] <inflatador>	 ryankemper no worries, nothing urgent
[21:19:02] <inflatador>	 created T403300 for the java plugin stuff. It's supported by the chart, I'm thinking it will be easy enough to integrate
[21:19:07] <stashbot>	 T403300: OpenSearch on K8s: enable prometheus metrics - https://phabricator.wikimedia.org/T403300
[21:33:20] <inflatador>	 OK, I'm out. See ya Tuesday!