[07:58:48] <wikibugs>	 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Traceback with IRC notifications on Trixie - https://phabricator.wikimedia.org/T429681#12039356 (10MoritzMuehlenhoff) 05Open→03Invalid No code change was needed, this was solely triggered by a missing ACL which got fixed with https://gerrit.wikimedia....
[07:59:34] <wikibugs>	 10netbox, 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Manage FR-tech 1:1 NAT rules using Netbox - https://phabricator.wikimedia.org/T429756 (10ayounsi) 03NEW p:05Triage→03Low
[09:52:16] <wikibugs>	 10CAS-SSO, 10WMF-General-or-Unknown, 07Epic, 07Security: All Wikimedia developer services should use single sign-on - https://phabricator.wikimedia.org/T189531#12039905 (10LSobanski) p:05High→03Medium
[09:58:15] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations, 10Phabricator, 06SRE: Add logout.d script for Phabricator - https://phabricator.wikimedia.org/T286904#12039921 (10LSobanski) @brennen @Aklapper https://phabricator.wikimedia.org/T406495 is now complete, how would you like to proceed from here?
[10:01:35] <wikibugs>	 10CAS-SSO, 06collaboration-services, 06Infrastructure-Foundations, 10GitLab (Auth & Access): gitlab account maps to two different developer accounts - https://phabricator.wikimedia.org/T384025#12039937 (10LSobanski) 05Open→03Resolved a:03LSobanski From the information above, this looks like expec...
[11:08:31] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773 (10BTullis) 03NEW
[11:14:08] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12040123 (10BTullis) p:05Triage→03High
[11:28:30] <wikibugs>	 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 06tools-infrastructure-team, 06cloud-services-team (FY2025/2026-Q3-Q4): Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451#12040218 (10fgiunchedi) Ack, thank you both! I've added a meeting note...
[11:37:18] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations, 06SRE: WebAuthn FIDO2 support in CAS - https://phabricator.wikimedia.org/T277841#12040275 (10SLyngshede-WMF) 05Open→03In progress Currently targeting CAS 7.3.X series.
[11:37:27] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations, 06SRE: WebAuthn FIDO2 support in CAS - https://phabricator.wikimedia.org/T277841#12040277 (10SLyngshede-WMF) a:03SLyngshede-WMF
[13:13:56] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12040643 (10JMeybohm) /link {T375845}
[13:24:04] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026) - https://phabricator.wikimedia.org/T429386#12040676 (10Papaul) @cmooney thank you for the update
[13:28:19] <wikibugs>	 10netops, 10Cloud-VPS, 06Data-Platform-SRE, 10Data-Services, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12040707 (10fgiunchedi) @brouberol @BTullis please let me know what you think of the load balancer plan and implementation in t...
[13:32:49] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12040751 (10BTullis)
[14:11:45] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12040868 (10BTullis) Thanks @JMeybohm - So I see now from a bit more digging...
[14:29:55] <wikibugs>	 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 06tools-infrastructure-team, 06cloud-services-team (FY2025/2026-Q3-Q4): Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451#12040954 (10LSobanski) p:05Triage→03Medium
[15:21:55] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations, 10Phabricator, 06SRE: Add logout.d script for Phabricator - https://phabricator.wikimedia.org/T286904#12041211 (10Aklapper) Hi, I myself am not sure what to add apart from T286904#11091482. Please elaborate if anything is needed from our side - thanks!
[15:42:25] <wikibugs>	 10netops, 06Data-Persistence, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: codfw: rack A6 maintenance - https://phabricator.wikimedia.org/T429812 (10ayounsi) 03NEW p:05Triage→03High
[15:42:48] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12041420 (10ayounsi)
[15:52:34] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12041474 (10BTullis) a:03BTullis
[15:54:18] <wikibugs>	 10netops, 06Data-Persistence, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: codfw: rack A6 maintenance - https://phabricator.wikimedia.org/T429812#12041495 (10jcrespo) ms-backup2003 -> I will take care of stopping vital network services before the window.
[16:15:23] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12041649 (10ayounsi)
[16:25:25] <wikibugs>	 10netops, 06Data-Persistence, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: codfw: rack A6 maintenance - https://phabricator.wikimedia.org/T429812#12041701 (10Eevans) No action is needed for the aqs nodes themselves.   I //think// that the nightly ETL jobs that load data happen at ~2am UTC, which s...
[17:13:57] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12041865 (10bking) Forgive the drive-by comment, but what do y'all think abou...
[18:43:33] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12042127 (10Papaul)
[21:59:34] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12042867 (10cmooney) >>! In T429773#12041865, @bking wrote: > Forgive the dri...
[22:27:44] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12042984 (10cmooney) > The IPv6 pool 2620:0:861:302::/64 is in the same state...
[22:43:57] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12043055 (10BTullis) >>! In T429773#12042984, @cmooney wrote: >> The IPv6 poo...
[23:01:22] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12043077 (10BTullis) >>! In T429773#12042867, @cmooney wrote: > A new /20 sti...