[01:24:01] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Tgr) Just as a heads-up: we recently increased AQS traffic from MediaWiki PHP code (T324675) which seems to work fine (it's causing some timeouts:... [06:30:00] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Joe) >>! In T327920#8647335, @Tgr wrote: > Just as a heads-up: we recently increased AQS traffic from MediaWiki PHP code (T324675) which seems to w... [07:18:57] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Tgr) MwHttpRequest (that is, Guzzle/php-curl) and the URLs from https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews. I don't know if RESTBa... [09:08:06] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: etcd cluster reimage strategies to use with the K8s upgrade cookbook - https://phabricator.wikimedia.org/T330060 (10elukey) @akosiaris I added https://wikitech.wikimedia.org/wiki/Etcd#Reimage_... [09:34:10] 10serviceops, 10Performance-Team, 10Patch-For-Review: Rewrite mw-warmup.js in Python - https://phabricator.wikimedia.org/T288867 (10Volans) >>! In T288867#8642058, @RLazarus wrote: > Sure, we could look at adding a warmup step to the server repool process. Historically we haven't worried about it, because th... [10:26:33] 10serviceops, 10Performance-Team: Rewrite mw-warmup.js in Python - https://phabricator.wikimedia.org/T288867 (10Clement_Goubert) Live-tested, codfw warmup takes around 3 minutes. [10:49:41] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Marostegui) [11:04:25] 10serviceops, 10Data-Persistence (work done), 10SRE, 10Datacenter-Switchover: sre.switchdc.mediawiki.03-set-db-readonly fails in live-test mode - https://phabricator.wikimedia.org/T330302 (10Marostegui) [11:04:31] 10serviceops, 10Data-Persistence (work done), 10SRE, 10Datacenter-Switchover: sre.switchdc.mediawiki.03-set-db-readonly fails in live-test mode - https://phabricator.wikimedia.org/T330302 (10Marostegui) Circular replication is now enabled (T330619) everywhere where it is supposed to be. It is one of our pr... [11:05:51] 10serviceops, 10Data-Persistence (work done), 10SRE, 10Datacenter-Switchover: sre.switchdc.mediawiki.03-set-db-readonly fails in live-test mode - https://phabricator.wikimedia.org/T330302 (10Clement_Goubert) >>! In T330302#8648213, @Marostegui wrote: > Circular replication is now enabled (T330619) everywhe... [11:08:37] 10serviceops, 10Data-Persistence (work done), 10SRE, 10Datacenter-Switchover: sre.switchdc.mediawiki.03-set-db-readonly fails in live-test mode - https://phabricator.wikimedia.org/T330302 (10Marostegui) It is probably something we still need to test before the switch anyways, as it is key, especially for t... [11:16:46] 10serviceops, 10Data-Persistence (work done), 10SRE, 10Datacenter-Switchover: sre.switchdc.mediawiki.03-set-db-readonly fails in live-test mode - https://phabricator.wikimedia.org/T330302 (10Clement_Goubert) We can probably just run `sre.switchdc.mediawiki.03-set-db-readonly` and `sre.switchdc.mediawiki.06... [11:17:26] 10serviceops, 10Data-Persistence (work done), 10SRE, 10Datacenter-Switchover: sre.switchdc.mediawiki.03-set-db-readonly fails in live-test mode - https://phabricator.wikimedia.org/T330302 (10Marostegui) That works for me :) We might need to make a not that having circular replication is a hard dependency [11:23:28] 10serviceops, 10Data-Persistence (work done), 10SRE, 10Datacenter-Switchover: sre.switchdc.mediawiki.03-set-db-readonly fails in live-test mode - https://phabricator.wikimedia.org/T330302 (10Clement_Goubert) Added https://wikitech.wikimedia.org/wiki/Switch_Datacenter#03-set-db-readonly as well as a note in... [12:04:13] 10serviceops, 10SRE, 10Traffic, 10Datacenter-Switchover: March 2023 Traffic Switchover checklist - https://phabricator.wikimedia.org/T330650 (10Clement_Goubert) [12:08:24] 10serviceops, 10SRE, 10Datacenter-Switchover: March 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [12:08:41] 10serviceops, 10SRE, 10Datacenter-Switchover: March 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [12:08:49] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [12:09:01] 10serviceops, 10SRE, 10Datacenter-Switchover: March 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) p:05Triage→03High [12:09:52] 10serviceops, 10SRE, 10Datacenter-Switchover: March 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [12:10:59] 10serviceops, 10SRE, 10Traffic, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Traffic Switchover checklist - https://phabricator.wikimedia.org/T330650 (10Clement_Goubert) [12:11:19] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [12:12:20] 10serviceops, 10SRE, 10Traffic, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Traffic Switchover checklist - https://phabricator.wikimedia.org/T330650 (10Clement_Goubert) p:05Triage→03High [12:22:58] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [12:23:21] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [12:56:00] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Joe) We should probably test that both scap works and a scap3 deployment also works (e.g. `docker-pkg`) when we've migrated the deployment server.... [12:56:29] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), 10User-notice: Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022) - https://phabricator.wikimedia.org/T308932 (10Ladsgroup) Wrong ticket :facepalm: [13:39:51] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Volans) [nit] the `enable-puppet` + `run-puppe-agent` can be simplified with `run-puppet-agent --enable "reason"`. [14:14:52] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [14:16:21] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [14:18:30] My scap/scap3 is a bit weak, if someone can give examples of how I can test deployments after the switchover I'll take it. [14:18:52] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [14:37:21] 10serviceops, 10Data-Persistence (work done), 10SRE, 10Datacenter-Switchover: sre.switchdc.mediawiki.03-set-db-readonly fails in live-test mode - https://phabricator.wikimedia.org/T330302 (10Clement_Goubert) 05Open→03Resolved Looks good, resolving. [14:37:24] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: March 2023 Datacenter Switchover live test - https://phabricator.wikimedia.org/T330271 (10Clement_Goubert) [14:37:59] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [14:38:23] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: March 2023 Datacenter Switchover live test - https://phabricator.wikimedia.org/T330271 (10Clement_Goubert) 05In progress→03Resolved All code paths exercised and fixes applied and tested. Resolving. [14:38:33] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: March 2023 Datacenter Switchover Blockers - https://phabricator.wikimedia.org/T328770 (10Clement_Goubert) [14:38:49] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [14:38:54] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: Ensure sre.switchdc.mediawiki live test multi-DC compatibility - https://phabricator.wikimedia.org/T329065 (10Clement_Goubert) 05In progress→03Resolved All code paths exercised for multi-DC, fixes applied and working. Resolving. [14:39:13] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: March 2023 Datacenter Switchover Blockers - https://phabricator.wikimedia.org/T328770 (10Clement_Goubert) 05Open→03Resolved All blockers resolved. [14:43:24] 10serviceops, 10SRE, 10Datacenter-Switchover: March 2023 Datacenter Switchover SRE-side Communication - https://phabricator.wikimedia.org/T329042 (10Clement_Goubert) [15:14:42] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [16:13:36] 10serviceops, 10Wikimedia-Apache-configuration, 10Wikimedia-Site-requests, 10Patch-For-Review: Temporarily redirect sgs.wikipedia.org to bat-smg.wikipedia.org until bat-smg->sgs move can be done - https://phabricator.wikimedia.org/T204830 (10JMeybohm) [16:19:55] 10serviceops, 10Prod-Kubernetes: More flexible updating of default network policy for kubernetes - https://phabricator.wikimedia.org/T330674 (10MoritzMuehlenhoff) [16:26:25] 10serviceops, 10Data-Engineering, 10Data-Persistence, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10MPhamWMF) [17:01:06] 10serviceops, 10Prod-Kubernetes: More flexible updating of default network policy for kubernetes - https://phabricator.wikimedia.org/T330674 (10JMeybohm) [17:02:05] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Refactor common_templates/0.2/default-network-policy-conf.yaml into a GlobalNetworkPolicy - https://phabricator.wikimedia.org/T275035 (10JMeybohm) [18:13:03] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Reset management module of mc1039 - https://phabricator.wikimedia.org/T330072 (10wiki_willy) a:03Jclark-ctr [21:22:55] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jan-Mar-2023), 10Datacenter-Switchover: CommRel support for March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T328287 (10Trizek-WMF) After multiple reviews, fixes, and the last translations being done, the message has been sent to 832 c...