[00:53:32] switched today from eqiad to codfw: annual.wikimedia.org, 15.wikipedia.org, bienvenida.wikimedia.org, security.wikimedia.org, sitemaps.wikimedia.org, tendril.wikimedia.org, dbtree.wikimedia.org, os-reports.wikimedia.org, research.wikimedia.org [08:25:47] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Joe) >>! In T330651#8648604, @Joe wrote: > We should probably test that both scap works and a scap3 deployment also works (e.g. `docker-pkg`) when w... [09:17:08] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10JMeybohm) @Ottomata could you please add an estimation of the compute resources this will require from... [09:33:02] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) >>! In T330651#8651629, @Joe wrote: >>>! In T330651#8648604, @Joe wrote: >> We should probably test that both scap works and a scap... [10:16:02] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [10:17:45] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [10:54:41] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [11:00:41] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [11:33:06] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [11:55:01] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10akosiaris) >>! In T327920#8647481, @Tgr wrote: > MwHttpRequest (that is, Guzzle/php-curl) and the URLs from https://wikitech.wikimedia.org/wiki/Ana... [12:03:00] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: etcd cluster reimage strategies to use with the K8s upgrade cookbook - https://phabricator.wikimedia.org/T330060 (10JMeybohm) >>! In T330060#8644396, @akosiaris wrote: > The experience above m... [13:03:44] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [13:06:52] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [13:12:54] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10Ottomata) [14:22:12] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover - T330651 s... [14:29:30] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10Ottomata) @gmodena you had some estimations somewhere, right? [14:42:17] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover - T330651 f... [14:42:40] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover - T330651 s... [14:44:16] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover - T330651 c... [14:45:48] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [14:49:06] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10gmodena) @JMeybohm @Ottomata this page contains metrics re compute and memory resources for the applica... [15:13:43] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [15:14:38] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: Kubernetes v1.23 use PKI for service-account signing (instead of cergen) - https://phabricator.wikimedia.org/T329826 (10JMeybohm) p:05High→03Medium [15:20:53] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [15:28:21] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, 10User-jijiki: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10jnuche) Scap updates on Thumbor hosts are currently failing since Scap requires Python 3.7. I would like to disable any Scap deployments until this ticket is comple... [15:29:22] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [15:55:49] 10serviceops, 10SRE, 10Traffic, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Traffic Switchover checklist - https://phabricator.wikimedia.org/T330650 (10Clement_Goubert) [15:56:00] 10serviceops, 10SRE, 10Datacenter-Switchover: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [16:10:07] 10serviceops, 10SRE, 10Datacenter-Switchover: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Volans) Netbox was switched too to codfw as part of the discovery services switch and appears to be quite slow. This setup has not been tested properly (and the DB was n... [16:20:23] 10serviceops, 10SRE, 10Datacenter-Switchover: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [16:37:30] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Refactor common_templates/0.2/default-network-policy-conf.yaml into a GlobalNetworkPolicy - https://phabricator.wikimedia.org/T275035 (10JMeybohm) a:03JMeybohm [16:39:40] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Refactor common_templates/0.2/default-network-policy-conf.yaml into a GlobalNetworkPolicy - https://phabricator.wikimedia.org/T275035 (10JMeybohm) I took the lazy approach to just convert the common default-network-policy-conf.yaml into Glo... [16:42:03] 10serviceops, 10SRE, 10Traffic, 10Datacenter-Switchover: 28 February 2023 Traffic Switchover checklist - https://phabricator.wikimedia.org/T330650 (10Clement_Goubert) 05Open→03Resolved [16:42:14] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [16:43:22] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [16:43:34] 10serviceops, 10SRE, 10Datacenter-Switchover: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) 05Open→03Resolved Marking as Resolved for now, will reopen in a week (or whenever `restbase-async` wants us to switch it back).