[06:16:17] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Upgrade new codfw switches to Juniper recommended - https://phabricator.wikimedia.org/T341670 (10ayounsi) [06:16:25] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: TLS certificates for network devices - https://phabricator.wikimedia.org/T334594 (10ayounsi) [06:32:28] sukhe: looks like there is another case of "Not accepting/receiving prefixes from anycast BGP peer" from doh1002 to cr2-eqiad [06:37:35] I did a "bird> reload bgp2" where bgp2 is the v4 session to cr2, and it's back to normal [07:31:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10ayounsi) The last host connected to asw-b1-codfw (the prod switch) is cloudweb2002-dev (https://netbox.wikimedia.org/dci... [09:32:44] hi, there's a lvs diff alert for lvs1020 since 17h, known? [09:33:04] https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DPyBal%20IPVS%20diff%20check [09:34:41] thanks godog , I'll check in the channels history if someone mentioned something about [09:36:08] cheers fabfur [09:38:41] I think could be related to some message sent yesterday by andrew.bogott [10:24:21] fabfur: congrats on the "Silent dropping of abusive traffic on port 80" ! [10:28:34] well, first we check that works (as intended) and then congrats :) [10:29:27] (thanks anyway but vgutierrez is the one that really pointed me in the right direction and fixed all the bugs) [10:42:09] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [10:42:29] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Direct 1% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341463 (10Clement_Goubert) 05In progress→03Resolved Done, all looks ok. We'll now start preparing for 5% [10:43:58] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Direct 1% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341463 (10Ladsgroup) >>! In T341463#9012011, @Clement_Goubert wrote: > Done, all looks ok. We'll now start preparing for 5% {meme, src=itshappening} [10:53:17] 10netops, 10Infrastructure-Foundations, 10SRE: Add Dell switches support to Homer/Cookbooks - https://phabricator.wikimedia.org/T320638 (10ayounsi) [10:53:25] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Users management on SONiC - https://phabricator.wikimedia.org/T338028 (10ayounsi) [10:57:41] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Direct 5% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341780 (10Clement_Goubert) [10:58:41] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Direct 5% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341780 (10Clement_Goubert) p:05Triage→03High [11:04:07] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [11:04:36] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Migrate group1 to Kubernetes - https://phabricator.wikimedia.org/T340549 (10Clement_Goubert) 05In progress→03Resolved After moving a couple group1 wikis, we have decided to go with a global traffic percentage to roll forward. Marking Resolved. [11:06:15] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [11:12:44] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [11:48:15] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Direct 1% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341463 (10Clement_Goubert) 05Resolved→03In progress I'm wondering if in the vein of >>! In T290536#8466377, @Ladsgroup wrote: > This is not really user-impacting, spec... [11:48:27] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [16:13:10] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10cmooney) >>! In T327919#9011559, @ayounsi wrote: > The last host connected to asw-b1-codfw (the prod switch) is cloudweb... [18:08:29] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10Papaul) @cmooney I am ok moving the server when it is ready. We can move it to B5/U20 ge-5/0/15. Also I see that asw-b1... [19:13:42] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Reedy) [21:29:44] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Direct 1% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341463 (10Quiddity) Thanks for the draft, appreciated! I've [[https://meta.wikimedia.org/wiki/Tech/News/2023/29#Tech_News:_2023-29|added this to Tech News]]. (The only major... [22:37:51] 10Traffic, 10Collection, 10Release-Engineering-Team, 10SRE, and 3 others: Strange error pattern noticed on viwiki during unrelated deploy - https://phabricator.wikimedia.org/T340850 (10Krinkle) 05Open→03Resolved a:03Reedy No entries in WMF Logstash for this after July 1st. {F37138383 height=200} [22:38:48] 10Traffic, 10Collection, 10Release-Engineering-Team, 10SRE, and 3 others: Strange error pattern noticed on viwiki during unrelated deploy - https://phabricator.wikimedia.org/T340850 (10Krinkle)