[05:46:21] urandom, mutante, yeah it goes switch port->SFP->cable->server nic, the SFP-T we use here allows to connect a copper RJ-45 cable to a switch port. There are inedded multiple types of SFPs for fiber, copper, etc) [10:20:12] hello folks, the Cassandra TLS expiration alerts for ml-cache clusters are due to some work that I am doing, they are now running with PKI certs so I'll need to figure out how to best alarm [10:20:24] I'll do it after lunch [10:21:08] in theory cassandra >= 4.x should auto-reload keystores with new TLS certs, so in the future we shouldn't take any action when a new cert is rolled out [10:22:06] my idea is to add an alarm 3/4 days before the expiration, since puppet by that time should have already rotated the cert [10:22:19] lemme know if you have other suggestions/ideas [10:22:42] (basically an alert to catch issues with puppet/cassandra reloads) [12:49:15] mmm so check_ssl_on_host_port doesn't offer le possibility to override warning/critical threshold afaics [13:13:00] elukey: check out prometheus::blackbox::check::http [13:13:55] jbond: ah lovely! [13:14:43] in this case I think I'd need ::tcp since cassandra has it own protocol [13:14:54] ahh yes probably [13:37:55] elukey: in https://gerrit.wikimedia.org/r/c/operations/puppet/+/932413/ I think the blackbox tcp check will also replace the -cql Icinga check, since it's just testing a TCP connection [13:38:38] created https://gerrit.wikimedia.org/r/c/operations/puppet/+/932413 [13:39:14] taavi: yes yes definitely, I'll also follow up on that one, but I am not 100% sure what is the plan for cassandra [13:56:47] if no one objects, I'm going to repool sessionstore/codfw [15:44:28] Hi folks! There are still some ml-cache-related cassandra alerts flowing, they should clear soon-ish. I filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/932427 to move everything to Prometheus, but I'll merge on monday if all agree :)