[09:24:17] 06serviceops, 07Kubernetes: k8s staging seems to be out of IP addresses - https://phabricator.wikimedia.org/T386107#10556224 (10JMeybohm) There where two alerts still firing for which I guess this (plus an SSH disconnect?) was the root cause: - Helm release eventgate-analytics/canary on k8s-staging@eqiad i... [10:13:28] 06serviceops, 07Kubernetes: Add pod ip address blocks to staging-eqiad - https://phabricator.wikimedia.org/T386232#10556463 (10JMeybohm) We have a free /21 in the Prefix containers used for wikikube staging in codfw and eqiad that we could probably use instead of adding another /24: - [[ https://netbox.wikimed... [10:15:09] 06serviceops, 07Kubernetes: Add pod ip address blocks to staging-eqiad - https://phabricator.wikimedia.org/T386232#10556469 (10JMeybohm) [10:15:11] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10556470 (10JMeybohm) [12:51:47] hey folks :) [12:52:01] I just pooled more k8s nodes to kartotherian's lvs [12:52:04] (1 -> 3) [14:16:19] and I have also removed 2 bare metal nodes from eqiad/codfw [14:28:01] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar): Migrate CentralAuth maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385866#10557190 (10larissagaulia) [14:44:00] also if anybody has time: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1120186 [14:46:06] elukey: {{done}}, but kartotherian is paging in -ops [14:46:22] yeah I am working on t [14:46:25] thanks [14:46:25] not sure what's wrong [14:46:32] need any help? [14:48:13] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10557253 (10akosiaris) [14:48:58] in theory no, it seems eqiad-related, I am going to verify what's wrong and report back :) [15:00:48] 06serviceops, 07Kubernetes: Add pod ip address blocks to staging-eqiad - https://phabricator.wikimedia.org/T386232#10557269 (10akosiaris) > Not sure if there is anything speaking against that, but I think it's more future proof than adding another /24 (cc @akosiaris). There is unfortunately. WikiKube's servic... [16:33:58] 06serviceops, 06Abstract Wikipedia team, 10function-evaluator: Have SRE provide a production-ready Rust image upstream - https://phabricator.wikimedia.org/T380807#10557452 (10akosiaris) 05Open→03Resolved a:03akosiaris The image is published and hopefully the name makes it clear that it shouldn't be... [16:52:15] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Review maps outage happened on Feb 17th - https://phabricator.wikimedia.org/T386648 (10elukey) 03NEW [18:15:17] 06serviceops, 06Content-Transform-Team, 07Epic, 10Maps (Kartotherian), 13Patch-For-Review: Move Kartotherian to Kubernetes - https://phabricator.wikimedia.org/T216826#10557613 (10WMDE-Fisch)