[10:07:56] <dcausse>	 lunch
[13:16:31] <inflatador>	 <o/
[13:20:46] <dcausse>	 o/
[13:21:09] <dcausse>	 inflatador, ryankemper https://docs.google.com/spreadsheets/d/1YhtKWffXpgAWwk9CJ0LJR-hQK-qXethA9o6SDyvLFg4/edit?gid=0#gid=0 (some details about the plugins for opensearch2)
[13:26:30] <inflatador>	 dcausse  Thanks! Cole White from Observability shared that the OpenSearch dashboard plugin management is equally difficult, apparently they are using custom puppet code for this: https://gerrit.wikimedia.org/r/c/operations/puppet/+/907838
[13:29:13] <dcausse>	 nice! not sure I understand this puppet code tho :)
[13:29:13] <inflatador>	 I was thinking of setting `/usr/share/opensearch/plugins` as a ceph mount or something equally crazy ;)
[13:29:42] <dcausse>	 plugins used to be deployed via scap :)
[13:30:18] <dcausse>	 but that was slightly difficult to operate
[13:32:17] <dcausse>	 debian package solved this by ensuring proper version dependencies such that you're less likely to be in a situation where opensearch runs version A but plugins got bumped to version B
[14:05:00] <inflatador>	 Yeah, I don't think I want to stop using Deb pkgs either
[15:10:30] <dcausse>	 trying to change the cirrus opensearch to use the dpe opensearch base image but seems like it's using opensearch 2.19.4, tried to bump it but getting E: Version '2.19.5' for 'opensearch' was not found
[15:12:44] <dcausse>	 might be in trixie not bookworm
[15:13:41] <dcausse>	 I guess this all depends on what docker-registry.wikimedia.org/openjdk-17-jre:latest is using as it's the base image for the opensearch image
[15:33:43] <inflatador>	 dcausse the current image is on 2.19.4, I should've already updated it but let me do it now
[15:34:38] <dcausse>	 inflatador: I have a MR almost ready
[15:36:32] <dcausse>	 https://gitlab.wikimedia.org/repos/data-engineering/opensearch/-/merge_requests/17
[15:45:16] <inflatador>	 dcausse approved, I can merge but was curious if you have perms to do so?
[15:50:17] <dcausse>	 sure trying
[15:50:19] <inflatador>	 oops, we need a minor change
[15:50:22] <inflatador>	 Don't merge yet
[15:50:26] <dcausse>	 ok :)
[15:50:32] <inflatador>	 Trixie has Java 21
[15:50:54] <dcausse>	 should I update the base image?
[15:51:58] <inflatador>	 dcausse yeah. I'm still checking but I bet the OpenSearch 3 variant uses trixie
[15:52:59] <inflatador>	 yes, will need to update the base image https://gitlab.wikimedia.org/repos/data-engineering/opensearch/-/blob/main/3/blubber.yaml?ref_type=heads#L41
[15:53:00] <dcausse>	 I think it still uses the java17 from the base image but pulling the opensearch package from trixie, feels like a hack but seems to work locally
[15:53:25] <dcausse>	 ok
[15:54:03] <inflatador>	 I think it will work OK with OpenSearch 2 on trixie, that's what I've been testing on cloudelastic. Although to be fair it hasn't started yet, that is more config related ;)
[15:59:39] <dcausse>	 ok MR updated
[16:02:49] <dcausse>	 inflatador: also https://gitlab.wikimedia.org/repos/search-platform/opensearch-plugins-deb/-/merge_requests/21 to fix the stconvert typo
[16:07:22] <inflatador>	 dcausse 1st MR approved if you wanna try and merge
[16:08:41] <inflatador>	 merged the package MR, will roll out shortly
[16:16:32] <dcausse>	 I could merge, does it publish automatically?
[16:16:41] <dcausse>	 seems like it
[16:16:55] <inflatador>	 Cool
[16:17:31] <inflatador>	 LMK if you are still having issues using our base image, happy to help
[16:17:56] <dcausse>	 sure
[16:25:11] <inflatador>	 I'm also going to play around with https://www.thelinuxvault.net/blog/how-to-run-podman-containers-under-systemd-with-quadlet/ , if we could run OpenSearch in containers in prod with systemd we could probably simplify the provisioning quite a bit
[16:35:35] <dcausse>	 hm.. the base image uses "copie" from the pkg image and thus lose the info that opensearch2 is installed from apt, and cannot use apt again to install the plugins because it wants to re-install opensearch2...
[16:36:27] <dcausse>	 it's no big deal we can use the old technique by unpacking the deb manually
[16:38:41] <inflatador>	 cool, or if you want to create your own variant in the same repo that's fine too
[16:45:55] <dcausse>	 yes possibly, I guess the question is how close we want to be from prod, the current image is more targetted at k8s I think?
[16:46:44] <dcausse>	 I think I'll get a first version working and see if we want to improve and make apt happy
[17:00:47] <inflatador>	 lunch, back in ~1h
[17:16:15] <dcausse>	 ok uploaded https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-opensearch-image/-/merge_requests/24 (still failing waiting for the new plugin version to be uploaded)
[17:16:17] <dcausse>	 dinner