[01:54:00] I also just switched https://releases.wikimedia.org to codfw to go with the flow. [01:54:30] it pulls mw-security patches from deploy2002 (not deploy1002 anymore because you switched that) [01:54:51] and syncs between each other. double checked those automatic rsyncs and that files were the same on both [01:55:05] will add to the docs "how to switch" page tomorrow [02:06:27] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Dzahn) https://releases.wikimedia.org switched to codfw (T330960) [09:04:08] 10serviceops, 10Infrastructure-Foundations, 10Scap, 10Patch-For-Review, 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸): Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10jnuche) [12:26:46] hi all we have loads of alerts for "WARNING: opcache cache-hit ratio is below 99.99% on php 7.4." in eqiad. im gussing this is harmless and related to the dc switch but not sure how best to deal with the alert (cc claime ) [12:37:38] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10cmooney) a:03cmooney [12:43:36] jbond: well eqiad seems 0 traffic until Wednesday, maybe we can just silence them until then [12:46:04] sgtm [13:33:28] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update wikikube eqiad to k8s 1.23 - https://phabricator.wikimedia.org/T331126 (10akosiaris) [13:33:46] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update wikikube eqiad to k8s 1.23 - https://phabricator.wikimedia.org/T331126 (10akosiaris) p:05Triage→03High [13:53:56] 10serviceops, 10DC-Ops, 10Prod-Kubernetes, 10SRE, 10ops-codfw: kubernetes202[34] implementation tracking - https://phabricator.wikimedia.org/T313871 (10JMeybohm) 05Open→03Resolved Was was done as part of {T326340} [13:53:59] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes202[34] - https://phabricator.wikimedia.org/T313870 (10JMeybohm) [13:54:30] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update wikikube eqiad to k8s 1.23 - https://phabricator.wikimedia.org/T331126 (10JMeybohm) [13:54:34] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [14:15:06] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10MoritzMuehlenhoff) [14:17:42] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10jbond) [14:18:04] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ayounsi) [14:18:30] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10MoritzMuehlenhoff) [14:19:04] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10MoritzMuehlenhoff) [14:25:05] hi all i see the failed jobs for mediawiki_job_pagetriage_cleanup_{en,testwiki,test2wiki} on mwmain2002. can these sjust be re-run or shuld i do a reset-failed and see what happens on the next run [14:44:58] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10lbowmaker) [14:45:31] 10serviceops, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10lbowmaker) [14:55:22] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10elukey) [14:58:00] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10fgiunchedi) [15:10:33] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10herron) [15:12:45] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Dzahn) [15:17:41] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [15:22:18] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10hnowlan) [17:25:48] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10colewhite)