[10:28:35] lunch [13:04:49] o/ [13:34:32] \o [13:34:56] o/ [13:42:57] just feeling out of it this morning, poor sleep, decided to take a sick day. Be back tomorrow. [13:46:03] sorry to hear this, hope you'll get better soon, take care! [13:49:35] +1, hope you feel better soon [14:19:43] CR up for enabling ECS logs in the non-production cirrus environments if anyone has time to look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1319497 [14:21:25] ^^ nm, got my +1 [14:24:25] thanks! [14:38:20] Hey folks, we're working on security plugin support in opensearch and it's unclear to me how relforge has the security plugin enabled, yet does not set the required certificate parameters. Could someone reach out to me to chat about it? [14:38:56] cwhite: looking [14:40:35] cwhite the security plugin is disabled, we use systemd and hieradata to allowlist plugins [14:41:06] ref https://gerrit.wikimedia.org/r/c/operations/puppet/+/1273887 [14:41:23] cwhite: yes, the sec plugin is visible on disk but not available to opensearch, (absent from GET /_cat/plugins) [14:43:41] hmm, this is problematic. [14:44:15] https://puppet-compiler.wmflabs.org/output/1318792/9091/relforge1008.eqiad.wmnet/index.html [14:45:22] TL;DR: we're amending opensearch_2.yml.erb to add the required configuration if disable_security_plugin=false [14:46:26] Ah, I think we had some issues with that var when we implemented the filtering [14:46:40] since it adds a bunch of extra configuration and opensearch has been sensitive (at least in the past) to dangling config options, we'll need to validate how to gate this configuration [14:46:59] Agreed, I'll start thinking about it, if you have suggestions LMK [14:47:14] or, would relforge tolerate the config? [14:48:09] relforge can be broken if this helps testing, but if this config affects cirrussearch hosts I'd be cautious [14:48:23] I doubt it would be able to start with security config active but no security plugin [14:49:08] Let me think back about why we used `disable_security_plugin=false` in the first place. Maybe we don't need to do that [14:49:59] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1294308 here's past me's explanation anyway [14:51:39] Thanks! I CC'd you on the patch to collaborate about options and blocked it for now. [14:56:08] Cool, will take a look. I think maybe we can add a gate to the template that checks for the existence of the `plugins_mandatory` var , since it doesn't look like y'all are using that var [15:06:01] actually it's probably easier to set `disable_security_plugin=true` and add the security plugin to the list of `plugins_mandatory`. I'll try that first [15:33:13] * cwhite crosses fingers for good luck [15:33:48] cwhite it worked! I left a comment on your change. Basically I just need to apply these changes to cloudelastic/production and you should be unblocked [15:34:13] Fantastic! Thank you so much! [15:35:38] np! I might have to bug you on T324335 at some point in the near future, but I need to fix the logging pipeline on the cirrus servers first. [15:35:39] T324335: Index the OpenSearch server application logs in the central observability cluster, using ECS format - https://phabricator.wikimedia.org/T324335 [15:38:57] I've been keeping an eye on that project and from here it looks like it's going in a good direction. Please do to reach out if you encounter mysterious behavior! [15:41:12] Yeah, I'm trying to make it more gradual than 0 servers shipping logs -> ~130 shipping logs, if that is excessively cautious LMK. [15:52:04] It's definitely worth getting an idea of volume. We're still struggling to store it all for the 90d retention period. [17:15:52] inflatador: filed T433624, unsure how much work this is on your end but please let me know if difficult [17:15:52] T433624: Upload opensearch 3.7.0 to apt.wikimedia.org (trixie/thirdparty) - https://phabricator.wikimedia.org/T433624 [17:16:41] dinner [17:20:09] dcausse ACK, will take a look [17:30:12] OK, that should be ready [18:16:02] inflatador: oh awesome that was quick, thanks! will upload a MR tomorrow for the docker-image