[11:38:09] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c70b0979-84e8-4fe7-8682-45d50615a587) set by cmooney@cumin1002 f... [13:15:42] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10cmooney) Ok I have made the Netbox changes and pushed the resulting config to lsw1-b8-codfw now, and the port it up (note the por... [13:32:45] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10cmooney) [14:11:26] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10akosiaris) >>! In T352883#9445709, @cmooney wrote: > Ok I have made the Netbox changes and pushed the resulting config to lsw1-b8... [14:14:42] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10ayounsi) > This is the thing we need to get fixed, I see Yep, that's {T352893} and its 2 CRs. [14:34:51] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10Clement_Goubert) ` # kubectl describe nodes kubestage2002.codfw.wmnet | grep -A3 Addresses Addresses: InternalIP: 10.192.22.13... [14:35:04] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10akosiaris) >>! In T352883#9445907, @ayounsi wrote: >> This is the thing we need to get fixed, I see > Yep, that's {T352893} and i... [14:37:18] XioNoX: So, I commented in the change. The kubelet will update topology labels from the config file and restart automatically after such a change to the config file and calico will react too, bringing down BGP peerings and trying new ones if applicable [14:37:36] so, we probably want something more static than LLDP in there. [14:45:22] 10Traffic, 10Patch-For-Review: tcp-mss-clamper doesn't work on bullseye / kernel 5.10 - https://phabricator.wikimedia.org/T353657 (10CodeReviewBot) vgutierrez merged https://gitlab.wikimedia.org/repos/sre/tcp-mss-clamper/-/merge_requests/12 clamper: support bullseye kernels [14:52:19] 10Traffic, 10Patch-For-Review: tcp-mss-clamper doesn't work on bullseye / kernel 5.10 - https://phabricator.wikimedia.org/T353657 (10CodeReviewBot) vgutierrez opened https://gitlab.wikimedia.org/repos/sre/tcp-mss-clamper/-/merge_requests/13 Test IPv6 MSS clamping [15:29:56] XioNoX: looking into how we could export the device's primary interface connection through the sync-netbox-hiera cookbook and match on that for the labels [15:30:23] Going blind into graphql >_> [15:40:35] claime: yeah, that's why I didn't want to go that way for the scope of this project [15:49:39] 10Traffic, 10Release-Engineering-Team, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10dancy) Looking at https://gitlab.wikimedia.org/repos/sre/dnsdist/-/blame/bookworm-wikimedia/.gitlab-ci.yml#L14 it seems you identified the problem with running mult... [16:12:27] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ayounsi) [16:43:28] 10Traffic, 10Release-Engineering-Team, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10BCornwall) Indeed, this was my mistake. Sorry for that! Thanks for your help, @dancy [17:49:02] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10RobH) [18:22:20] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw, 10Patch-For-Review: Migrate mr1-codfw from asw-a1-codfw to lsw1-a1-codfw - https://phabricator.wikimedia.org/T348164 (10cmooney) @papaul hey the link from mr1-codfw ge-0/0/3 to lsw1-a2-codfw ge-0/0/47 is now configured, but it's down both sides.... [18:57:35] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ayounsi) > I was surprised we don't have the same need for public IPs there. We will at some point, the thought process was that I assigned both public/private for codf... [20:49:12] 10Traffic, 10Infrastructure-Foundations, 10SRE: NetworkProbeLimit cookie should set samesite attribute - https://phabricator.wikimedia.org/T342624 (10Tgr) [23:11:54] 10Traffic: purged package cannot be built due to failing test - https://phabricator.wikimedia.org/T354712 (10Fabfur) [23:15:29] 10Traffic, 10Patch-For-Review: purged package cannot be built due to failing test - https://phabricator.wikimedia.org/T354712 (10CodeReviewBot) fabfur opened https://gitlab.wikimedia.org/repos/sre/purged/-/merge_requests/4 Skip go tests during package build