[15:38:06] Hi team! As part of https://phabricator.wikimedia.org/T344798, I'd need to allow the cumin hosts to reach the logstash-{eqiad,codfw} hosts on port 9200, to hit the opensearch API from a cookbook. Could any of you walk me through the steps I'd need to follow to enable this? Thank you! [15:42:18] hi, the firewall rules are managed via ferm, which is a wrapper around iptables; you can create a ferm::service define for port 9200 along with an "srange" for which you can retrieve the host list from the "cumin_masters" Hiera variable [15:43:00] feel free to ad me as reviewer in Puppet, happy to have a closer look [15:43:52] we have sth like that in place already for other host-based access to opensearch, I'm looking it up on puppet [15:45:14] thanks to both of you [15:46:50] brouberol: would you need access to each and every host or just the general api would be fine regardless of host? [15:47:06] I'm assuming the former, and I thought I'd ask [15:47:33] is seems to me that we have such a rule for datahubsearch (https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/manifests/opensearch/datahubsearch.pp#11) but no such rule for logstash servers [15:47:33] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/manifests/opensearch/logstash.pp [15:49:25] godog: the way the cookbook works is by calling http://{host_ip}}:9200/_cluster/health. That being said, I could flip that on its head and have a mapping of cluster alias -> "general" cluster API URL that I could hit, thus eliminating the need to have the cumin host be able to hit port 9200 on each server in the opensearch cluster [15:50:39] brouberol: oh ok I assumed wrong! yeah for general cluster health totally [15:52:28] Speaking of general API url: is there a way I can infer it from the cumin alias, or the cluster name itself? (sorry if the question seems naive/uninformed, I joined last week) [15:53:16] heheh not uninformed question at all, there is not that I'm aware of [15:54:20] (thinking out loud) as of today we have two ways of accessing the logs opensearch cluster, either via ferm rules on specific hosts from profile::logstash::production, that's direct access on 9200 [15:54:50] or https + basic auth via logs-api.svc.{eqiad,codfw}.wmnet [15:55:19] gotcha. So in that case, how can I know what URL to hit to talk to the opensearch API via its "general" DNS? Something like logstash.svc.eqiad.wmnet ? [15:56:51] Oh, I'm guessing there are defined in service.yaml, which goes into LVS via pybal? [15:56:53] yeah we have that set up for logs-api.svc i.e. https+basic auth and not for direct 9200 access IIRC [15:57:12] that's right yeah [15:58:13] hmm, so in that case, I *think* cumin -> host_ip:9200 might be what we want there. We're only interested in hitting the /_cluster/health API [16:00:03] and if that is the case, would defining something like https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/manifests/opensearch/datahubsearch.pp#11 but for the logstash cluster work? [16:01:34] something like that but restricted to cumin hosts would work yes, check out profile::logstash::production [16:02:42] cc cwhite ^ [16:04:25] mmhh though that would give you access to the logstash hosts only, not the whole cluster [16:05:04] in general the logs opensearch cluster api isn't accessible from everywhere in the network for privacy and security reasons [16:05:53] having said that the cumin hosts are already quite privileged (I think?) in terms of network access, so we could add those to the list [16:06:58] personally I'd prefer going for logs-api.svc + https and basic auth, I can see how that complicates things a little in the cookbook [16:07:40] I'd allow cumin hosts access to :9200. I don't know the specifics within spicerack, but I'd *guess* that a cookbook running against a host will want to check that host's health directly. [16:08:17] If not, general cluster health can be queried against any other host in the cluster, including logs-api. [16:09:22] indeed [16:10:12] so yeah ferm access to 9200 from cumin hosts seems the easiest way forward [16:11:05] brouberol: ^ [16:12:50] thank you, I'll have a look at that! [16:13:34] sure np [16:13:54] and one note: we're currently in the process of refactoring the firewall Puppet code to allow a gradual migration towards nftables, most of the existing ferm::service definitions still use Ferm-specific settings, but when you add a new service you can directly use the new compatible method: [16:14:27] most of the existing ferm services pass the values to a resolve() function which is specific to ferm [16:14:57] insted the ferm::service API has been extended to instead resolve interally, as such you can simply retrieve an array of hostnames from Hiera [16:15:24] and then pass this to the srange argument of ferm::service [16:15:40] https://gerrit.wikimedia.org/r/c/operations/puppet/+/956367 is an example change which moves from the legacy syntax to the new format [16:15:45] nftables is great,I love how clear and concise their syntax is as compared to iptables... [16:16:39] and if you make an error it will show you to the exact line/character where the parsing error is [16:36:24] I _think_ that should work: https://gerrit.wikimedia.org/r/c/operations/puppet/+/957324. I need to log out (family time), but in the meantime, thank you again for your time and help! [16:59:37] that's the right idea but will work only for a part of the cluster, not the whole cluster [16:59:45] I'll followup on the review