[02:45:26] 10Domains, 10Traffic, 10SRE: Acquire enwp.org - https://phabricator.wikimedia.org/T332220 (10Legoktm) >>! In T332220#8700530, @violetwtf wrote: > I think this spans a bit outside of "literally everything" -- enwp.org is widely used by Wikipedia editors in Wikipedia-adjacent channels to refer to Wikipedia. I... [09:26:31] 10Traffic, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10Marostegui) [09:43:34] 10Traffic, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10fgiunchedi) [09:55:13] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops: Find a sensible way to redirect traffic to mw-on-k8s - https://phabricator.wikimedia.org/T331318 (10Joe) After some thought, I think the most maintainable way to do this is to add an additional lua module to the maps for api/appservers. Specifically, this would... [10:04:01] 10Traffic, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10fgiunchedi) [10:09:52] 10Traffic, 10MW-on-K8s, 10SRE, 10Wikidata, and 2 others: Migrate testwikidata to Kubernetes - https://phabricator.wikimedia.org/T331268 (10Clement_Goubert) [10:10:04] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [10:25:51] 10netops, 10Infrastructure-Foundations, 10SRE: Invesitgate requirement for 'session-mode auatomatic' on EVPN iBGP peerings - https://phabricator.wikimedia.org/T332295 (10cmooney) p:05Triage→03Low [11:02:47] vgutierrez: when you have a moment, https://gerrit.wikimedia.org/r/c/operations/alerts/+/900241 [11:03:03] ack [11:10:17] cheers [11:30:44] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (FY2022/2023-Q3): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10aborrero) >>! In T327919#8699523, @cmooney wrote: > > In terms of the move we need to work with @aborr... [13:23:34] 10Traffic, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10fgiunchedi) [13:52:07] 10netops, 10Infrastructure-Foundations, 10SRE, 10observability, 10Patch-For-Review: BFD Status Check Fails when device is unavailable - https://phabricator.wikimedia.org/T332080 (10cmooney) 05Open→03Resolved a:03cmooney [13:55:48] 10netops, 10Infrastructure-Foundations, 10SRE, 10observability, 10Patch-For-Review: SNMP Network Checks throw exception when device is unreachable - https://phabricator.wikimedia.org/T332080 (10cmooney) [13:56:14] 10netops, 10Infrastructure-Foundations, 10SRE, 10observability, 10Patch-For-Review: SNMP Network Checks throw exception when device is unreachable - https://phabricator.wikimedia.org/T332080 (10cmooney) 05Resolved→03Open [14:09:48] vgutierrez: bblack: can I steal about 10? minutes of time today from at least one of you? want to go over my vague plans for our intern (and soon we should introduce both of you to him) [14:50:22] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (FY2022/2023-Q3): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10Papaul) @cmooney thank you for getting the table ready for the cloud nodes move. As you can see on asw... [14:51:36] tomorrow would also be ok :) [14:52:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10observability, 10Patch-For-Review: SNMP Network Checks throw exception when device is unreachable - https://phabricator.wikimedia.org/T332080 (10cmooney) Looks like our OSPF check already handles any exceptions: ` cmooney@alert1001:~$ ./check_ospf.py --hos... [15:00:13] cdanis: I suspect bblack would be more useful than me, given his broader context but happy to step in if needed [15:06:25] cdanis: if you schedule something for tomorrow I should be able to make it! [15:07:00] cool :) [15:07:36] thanks! [15:16:53] cdanis: can you add me as optional if it's during my working hours? [15:17:51] yes! [15:36:56] 10Traffic, 10SRE: Clean up and refactor the dnsrecursor module - https://phabricator.wikimedia.org/T332083 (10ssingh) 05Open→03Resolved a:03ssingh This has been resolved with the https://gerrit.wikimedia.org/r/898957 and all `R:Class = dnsrecursor` hosts running bullseye: ` sukhe@cumin2002:~$ sudo cumin... [16:03:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (FY2022/2023-Q3): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10cmooney) >>! In T327919#8702610, @Papaul wrote: > I know the first 4 ports on cloudsw1-b1 are set up as... [16:50:06] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (FY2022/2023-Q3): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10Papaul) cloudservices2004-dev = U37 cloudservices2005-dev = U38 cloudweb2002-dev = 39 yes we already o... [17:19:44] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (FY2022/2023-Q3): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10cmooney) Ok cool. So I'd propose we take it like this: **1. Move sretest2001 from port xe-0/0/1 to xe... [18:29:23] 10Domains, 10SRE, 10Traffic-Icebox: Register wiki(m|p)edia.ro - https://phabricator.wikimedia.org/T222080 (10BCornwall) Hi, @CRoslof! Have you been able to look into these registrations? Thanks! [18:36:23] 10Traffic, 10PyBal, 10Traffic-Icebox, 10Sustainability (Incident Followup): Pybal should reject a confctl configuration that indicates only one cp-text is pooled - https://phabricator.wikimedia.org/T245060 (10BCornwall) Possible duplicate of T184715 [18:41:32] 10Traffic, 10SRE, 10VPS-project-Codesearch, 10serviceops, 10Patch-For-Review: Consider using BindsTo instead of Requires to declare dependencies between systemd unit - https://phabricator.wikimedia.org/T284555 (10BCornwall) [18:44:01] 10netops, 10Infrastructure-Foundations, 10SRE: Automate EVPN switch underlay BGP neighbor peerings - https://phabricator.wikimedia.org/T327934 (10cmooney) 05Open→03Resolved Changes merged and feature now fully controlled from automation / homer. [18:58:37] 10netops, 10Infrastructure-Foundations, 10SRE, 10observability, 10Patch-For-Review: SNMP Network Checks throw exception when device is unreachable - https://phabricator.wikimedia.org/T332080 (10cmooney) 05Open→03Resolved Closing this one, all our checks now deal with the scenario gracefully. The exa... [19:10:47] 10Traffic, 10Observability-Logging, 10SRE: varnish-frontend-fetcherr sets incorrect level in logstash - https://phabricator.wikimedia.org/T330267 (10colewhite) 05Open→03Resolved a:03colewhite Fixes rolled out for SEVERITY_LABEL and levelName field. [20:26:35] 10Traffic, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: Integrate In-App Internet censorship circumvention by domain fronting - https://phabricator.wikimedia.org/T327286 (10LGoto)