[11:02:44] marostegui: vgutierrez: I'm gonna roll restart kafka-main so it picks up new certs, no disruption anticipated but we never know [11:02:53] thanks for the heads up claime [11:02:55] 🍿 [11:03:05] (with coffee sauce) [11:27:27] all done [11:28:06] May or may not do the much-needed rebalance now, depends if brouberol is available or not [14:17:24] claime: I'm around [14:17:35] send me a meet link, so we can have a look? [14:19:35] qq: does anyone have prior experience with rebalancing kafka main? I know the hosts have a 1G NIC, so I"m particularly interested in knowing whether someone has a "safe" replication throttle value in mind [14:35:34] Sorry bud, meeting rn [14:41:01] np [14:41:17] elukey: do you happen to know about a safe throttle for kafka-main? [14:43:04] I'd be tempted to start with something akin to 30MB/s given https://grafana.wikimedia.org/d/000000027/kafka?var-kafka_cluster=main-eqiad&orgId=1&from=now-30m&to=now&timezone=utc&var-datasource=000000026&var-cluster=kafka_main&var-kafka_broker=%24__all&var-disk_device=%24__all&viewPanel=panel-56 [14:43:10] network wise, top 30d history maxes at 25MB/s [14:43:20] so 50MB/s would probably be quite fine [14:43:47] ah yeah, your 30MB/s would also work, slightly more conservative, but fine. [14:44:10] I am wondering whether there is some IO layer "limit" we might not want to hit [14:44:21] but again, at the age of SSD, 50MB/s should be nothing [14:44:27] 30 even [14:45:06] ok cool, thanks [14:45:27] when it comes to main, I tend to be more conservative and cautious than jumbo [14:46:13] sounds sane to me [15:31:47] I'll be depooling DNS for DRMRS in a few minutes. This is part of: https://phabricator.wikimedia.org/T390813 [16:29:32] jasmine_: please LMK when done with puppet-merge [16:30:38] ack o> [16:37:02] godog: ran a second time to pick up an additional change, done for sure now :) [16:38:46] jasmine_: \o/ sounds good [17:26:38] https://releases.wikimedia.org and https://releases-jenkins.wikimedia.org are now served by codfw and on bookworm. reimaging eqiad backend to also become bookworm right now. one of those bullseye remnants gone. [17:27:51] \o/ [17:39:04] nice! [18:10:56] {◕ ◡ ◕}