[12:01:50] replied on the task [15:33:28] wikikube question: is there an accepted pattern for handling services that use discovery for read traffic, but require a DC-specific ingress for writes? We are just now starting dse-k8s-codfw and I'm looking at this in T410956 [15:33:29] T410956: OpenSearch on K8s: Allow applications to target specific DC for writes - https://phabricator.wikimedia.org/T410956 [15:34:45] inflatador: look at the discovery DNS RRs suffixed with -ro and -rw in the the DNS repo. [15:35:32] that is, yes, there is a pattern. [15:36:56] akosiaris thanks! Will check out https://gerrit.wikimedia.org/r/plugins/gitiles/operations/dns/+/refs/heads/master/templates/wmnet#949 and friends [16:08:37] <_joe_> inflatador: you need a dnsdisc record that is active/active (that we suffix with -ro) and one active/passive (that we suffix -rw) [16:09:55] It's not necessarily that I want to run active/passive, I just need a way to send writes to a specific endpoint from outside the cluster...would that concept still work? [16:19:25] yes [16:22:00] ACK, thanks again [16:25:05] <_joe_> inflatador: A/P works for writes if you want to make sure they go to a specific DC and not any other [16:38:31] _joe_ it's not exactly what I'm going for...more like production cirrussearch or WDQS where we have independent "clusters" per DC and we write to them independently from each other. So we don't necessarily want to stop writes to all DCs, just the ability to target per DC for write traffic. The design is not set in stone, so if would be easier to say, create different ns per DC we could do that [16:40:06] <_joe_> can you be more explicit about what you want to do? [16:40:16] inflatador: do you just want to write into the local DC? [16:40:38] <_joe_> you want for say the X WDQS cluster to be able to write to one DC, but read from both? [16:41:13] I think it's the other way around, he wants to be able to target both DCs independently to implement a mechanism to replicate the stream of incoming writes from the indexer? [16:41:32] in which case I would just use the dns records for the cluster-local svc ip [16:42:11] We need a way for clients to target a specific OpenSearch cluster (CODFW or EQIAD) for writes from outside the cluster [16:43:48] the different opensearch deployments won't share the same dataset? [16:46:52] understanding the intent of these different clusters at the application layer would be helpful imo :) [16:52:40] cdanis ACK, I'm just mimicking what we do for prod cirrussearch, which is technically 2 independent clusters. The data source is the same (mediawiki DB) and they stay more or less in sync via kafka/streaming updater. [16:53:12] If the cluster-local DNS records can be accessed from outside the cluster then that'll work just fine [16:57:40] DNS discovery records are generally CNAMEs to one of those addresses [17:08:36] inflatador: erik suggested investigating in opensearch cross cluster replication in which case a -rw discovery record could make sense to point at the current primary "opensearch" cluster [17:11:14] if cross-DC replication is done at the app layer like cirrus/wdqs then the app needs to control what DC they write to [17:11:40] It sounds like we'd need: 1)DC-specific record CNAMEd to DC-specific k8s ingress, 2)same record entered into envoy and cert manager via deployment charts [17:13:17] dcausse I wouldn't rule out cross-cluster replication but I would prefer to start by mimicking the current design since it's battle-tested [17:15:53] sure, built-in cross cluster replication will put some burden on dp-sres, iirc you'll have max 12hours of allowed downtime before the "replication link" breaks