[00:00:07] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) 05Open→03Resolved @akosiaris this is complete [08:12:56] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: K8s etcd on bullseye show TLS errors in logs - https://phabricator.wikimedia.org/T329556 (10elukey) The new SAN has been rolled out to all the clusters using PKI, so the ones left to fix are the... [09:14:47] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update wikikube codfw to k8s 1.23 - https://phabricator.wikimedia.org/T329664 (10JMeybohm) [09:17:00] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update wikikube codfw to k8s 1.23 - https://phabricator.wikimedia.org/T329664 (10JMeybohm) [09:30:16] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: What should happen to Toolhub during the 2023 DC switch? - https://phabricator.wikimedia.org/T329319 (10Clement_Goubert) >>! In T329319#8616304, @Ladsgroup wrote: > I think there is a misunderstanding, I'm not suggesting to make eqiad RW... [09:43:12] 10serviceops, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 10 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10MoritzMuehlenhoff) [10:09:08] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: What should happen to Toolhub during the 2023 DC switch? - https://phabricator.wikimedia.org/T329319 (10Marostegui) >>! In T329319#8616304, @Ladsgroup wrote: > I think there is a misunderstanding, I'm not suggesting to make eqiad RW in m... [10:18:17] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [10:23:59] 10serviceops, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: Switch wikikube-staging (codfw and eqiad) etcd clusters to use PKI - https://phabricator.wikimedia.org/T329717 (10JMeybohm) [10:26:42] 10serviceops, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: Switch wikikube-staging (codfw and eqiad) etcd clusters to use PKI - https://phabricator.wikimedia.org/T329717 (10JMeybohm) [10:30:55] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update wikikube codfw to k8s 1.23 - https://phabricator.wikimedia.org/T329664 (10JMeybohm) [10:38:41] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: Post Kubernetes v1.23 cleanup - https://phabricator.wikimedia.org/T328291 (10JMeybohm) [10:50:25] taavi: should be good, only thing left is a lingering alarm from an instance I just deleted, it should go away [10:50:34] Just did. [10:51:39] great, thanks! [11:44:46] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10Brycehughes) > From your perspective that would mean either browser cache or some forward proxy yo... [13:24:46] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: What should happen to Toolhub during the 2023 DC switch? - https://phabricator.wikimedia.org/T329319 (10Ladsgroup) >>! In T329319#8617660, @Marostegui wrote: > mX sections aren't switched over to codfw, they have never been switched to c... [14:58:38] 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Agree strategy for Kubernetes BGP peering to top-of-rack switches - https://phabricator.wikimedia.org/T306649 (10ayounsi) Circling back on the network side config now that there are a few patches out to improve the server s... [15:31:50] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover Excluded services - https://phabricator.wikimedia.org/T329193 (10Clement_Goubert) [15:32:23] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover Excluded services - https://phabricator.wikimedia.org/T329193 (10Clement_Goubert) [15:33:35] I've compiled the list of the services that are currently skipped by the sre.datacenter.discovery (which should end up being used for the switchover iiuc) in the task above. akosiaris _joe_, does that look right? [15:34:04] <_joe_> claime: I can't take a look rn :/ [15:35:00] <_joe_> a cursory look looks great [15:35:02] ack, that's ok, I just need someone to get eyes on it at some point [16:46:42] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jan-Mar-2023), 10Datacenter-Switchover: CommRel support for March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T328287 (10RZamora-WMF) [16:54:34] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10Brycehughes) @akosiaris would a simple text file of page names by line that need to be updated wor... [16:54:38] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: What should happen to Toolhub during the 2023 DC switch? - https://phabricator.wikimedia.org/T329319 (10bd808) Based on the various responses to my questions about specific dependencies (thanks to everyone who participated!), I think the... [17:20:49] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10Brycehughes) @akosiaris someone suggested that just dummy/null editing really commonly used templa... [17:30:27] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jan-Mar-2023), 10Datacenter-Switchover: CommRel support for March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T328287 (10Trizek-WMF) Central Notice * Banner updated ** [[ https://meta.wikimedia.org/w/index.php?title=MediaWiki%3ACentral... [17:59:38] 10serviceops, 10Wikimedia Enterprise, 10Performance-Team (Radar), 10affects-Kiwix-and-openZIM: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866 (10RBrounley_WMF) [18:02:05] 10serviceops, 10Wikimedia Enterprise, 10Performance-Team (Radar), 10affects-Kiwix-and-openZIM: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866 (10RBrounley_WMF) Hello - good news here! @Kelson and I met last week and discussed the opportunity to m... [18:40:59] !log dummy entry [18:43:40] no stashbot here [18:45:43] It would be possible to have stashbot here and also to make !log work from this channel. -analytics and -releng both have custom !log setups. [18:47:28] -analytics logs to https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log and -releng logs to https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:48:25] this channel could !log to a new page somewhere, or it could be treated the same as a !log in -opertations.