[10:30:51] * Emperor neither [10:55:51] PROBLEM - MariaDB sustained replica lag on s7 on db2122 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2122&var-port=9104 [10:56:51] RECOVERY - MariaDB sustained replica lag on s7 on db2122 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2122&var-port=9104 [15:27:56] Emperor, urandom - o/ if you have time could you review https://gerrit.wikimedia.org/r/c/operations/puppet/+/1013571 and let me know if the approach is good or not? [15:28:25] I'd like to start with some base aqs config changes to prep for the PKI upgrades during next q if you are ok [15:28:33] (not immediately, even next week) [16:00:48] \o [16:01:41] I have a question about the Puppet cassandra roles. As I understand it, the usual way at WMF is to have separate IPs to bind to for e.g. cassandra-a, cassandra-b etc. [16:02:55] Now our own cass clusters (ml-cacheX00Y) are unlikely to ever have more than one node per machine. What's the easiest way to just use the host IP? Just not declaring the listen_address? setting it to the hosts IP? [16:04:02] I ask because I recently added some stuff to utilize Balthazar's framework for k8s-external services so LW can talk to our Cass clusters. I used the machine role in puppet for that, but naturally, it only returns the host IPs, not the additional ones for Cassandra. [16:19:23] elukey: sorry, I've meant to, but I'm slammed. This is why I was hinting at making it a KR for the coming quarter, because as soon as this week is over, I should be in good shape [16:20:44] ahh yes yes thanks np :) [16:21:37] klausman: the short answer is that you can't/shouldn't [16:22:11] I mean, anything is possible, but we've been trying to reduce the number of conditions/exceptions [16:22:27] sessionstore is also only one instance, too [16:24:56] klausman: there is work pending (future KR) to refactor things to so that we don't need additional IPs, even with multiple instances (all instances would bind to the host IP on different ports), but the biggest obstacle there isn't that configuration, but doing it in such a way that we can migrate existing clusters. [16:25:07] and tooling [16:25:21] so TL;DR it's still a bit down the road [16:26:03] Alright. I'll try and find a different way to get the IPs out of puppet and into depl-charts