[09:52:30] qq: if we theoretically wanted to define a CNAME record pointing to multiple A records in the wmnet zone, would we do this directly in https://gerrit.wikimedia.org/r/admin/repos/operations/dns, or do we have a way to do this in the puppet repo? [09:55:01] brouberol: CNAMES are only in the dns repo afaik, what's the usecase? maybe a discovery record would fit? [09:56:30] I'd like to stop maintaining lists of individual kafka broker DNS all over as a way of doing service discovery (ex: https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/master/gobblin/common/kafka_to_hdfs_hourly.properties#5) [09:57:14] can those be populated by puppet with a puppetdb query? [09:57:26] kafka is a bit "special" as it does not support SRV records last time I checked, _but_ it's happy to be given a CNAME record that points to individual broker DNS records, pick the first that answers, and move on [09:58:04] does it do some form of balancing/round robin between the hosts in the list? [09:58:08] does it change with a single cname? [09:58:20] SRV records would have been my next question :D [09:58:28] > can those be populated by puppet with a puppetdb query? volans: possibly yes, but we maintain these lists all over the place, in at least 4 repos that I know of [09:58:46] > does it do some form of balancing/round robin between the hosts in the list? [09:59:14] In case it helps, there was some previous discussion on this topic here: https://phabricator.wikimedia.org/T213561 [09:59:14] I think it picks the first one, see if it answers, if not, moves on to the 2nd one, etc. so if you want round robin, you have to do this at the DNS level [10:00:58] my thinking was: say we define a kafka-..wmnet CNAME that resolves to individual broker DNS, then we could at least maintain that in a single place, and use that DNS everywhere else without having to patch all the connection strings every time we add/remove a broker [10:05:12] it also looks like that prior to kafka 2.1, we'd need to rely on RR DNS for CNAME (should we go that way) as the client always takes the first IP if the CNAME resolves to multiple ones: https://cwiki.apache.org/confluence/display/KAFKA/KIP-302+-+Enable+Kafka+clients+to+use+all+DNS+resolved+IP+addresses [10:05:48] anyway, this is just food for thought atm, I'm trying to figure out the best way forward, but I'm not planning to take action right now [10:08:02] isn't that against the RFC? I think maybe bind allowed that but IIRC CNAMEs are 1:1 mappings [10:09:30] sorry, you mean the way kafka behaves goes against the RFC or my suggestion? To make sure I'm following [10:09:43] (or both x) [10:10:01] (brb, lunch) [10:10:04] in the end both :D [10:12:15]  ?J#?Jhttps://datatracker.ietf.org/doc/html/rfc2181#section-10.1 [10:12:25] ops, bad pasting [10:12:27] https://datatracker.ietf.org/doc/html/rfc2181#section-10.1 [10:12:56] [cit.] "There may be only one such canonical name for any one alias." [10:27:41] so knowing bran.don I guess that gdnsd would not allow you to do that in the first place :D [10:30:56] Ack [10:58:58] i suspect that was what informed this coment https://phabricator.wikimedia.org/T213561#4882509 [11:22:18] <_joe_> brouberol: I strongly suspect that the lists of kafka brokers are also used for stuff that has little to do with connecting a kafka client to the servers [11:22:37] <_joe_> like to open firewall ports for instance, or egress rules in k8s [11:23:06] <_joe_> so for those we'd need to do quite a bit of work [11:23:38] <_joe_> my suggestion is to collect all places where those are defined separately in a task and we can try to understand what's the best way to unify them [11:54:04] _joe_: indeed! But i think there are 2 different things at play here. A) listing all brokers with their associated IP, rack, etc, in configuration, to then generate network policies & such, and B) _maintaining_ a list of brokers kafka-x1001:9092,kafka-x1002:9092,... in each application's configuration to access kafka itself [11:55:30] I'm not suggesting in any way we need to stop doing A). My thoughts were, we could simplify B by exposing a DNS record that would indirectly resolve to the kafka broker IPs in a given cluster. If brokers are added or removed, the service DNS stays the same, and gets updated with the new IPs [11:56:05] so that we don't need to update application configuration all around (and redeploy each app) every time we add/remove a broker [16:04:05] !log sudo cumin -b1 -s300 'A:dns-rec and A:edges' 'systemctl restart ntp.service' [16:04:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:04:20] er, wrong channel but ok [18:39:03] https://neal.fun/internet-artifacts/ might be of interest