[00:24:43] 06serviceops, 10Dumps-Generation: Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736#10649873 (10Novem_Linguae) [00:35:53] 06serviceops: Align mw-on-k8s alerts with capacity pools - https://phabricator.wikimedia.org/T389224#10649886 (10Scott_French) I spent some time fiddling with this today, and after writing a bunch of heavily duplicated YAML that manages "which releases should alert at what aggregation level" individually I reali... [09:47:01] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 10Data-Platform-SRE (2025.03.01 - 2025.03.21), and 2 others: Update wikikube-staging-eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T389045#10650883 (10Gehel) [10:44:42] 06serviceops, 06SRE, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 10Wikimedia-Site-requests: Increase entityAccessLimit for WikibaseClient wikis - https://phabricator.wikimedia.org/T384455#10651140 (10thiemowmde) The #wikidata_integration_in_wikimedia_projects team essentially does have two... [14:03:41] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652008 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-disable-puppet for datacenter switchover from codfw to eqiad -... [14:04:04] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652012 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from cod... [14:12:03] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652052 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from codfw to eqiad - fin... [14:13:14] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652053 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from codfw to eqiad... [14:14:43] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652055 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from codfw to eqiad... [14:15:43] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652058 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from codfw to eqiad - M... [14:15:58] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652060 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from codfw to eqiad - f... [14:16:37] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652064 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from codfw to eqiad... [14:17:37] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652078 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from codfw to eqiad... [14:17:45] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652090 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from codfw to eqiad... [14:18:10] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652096 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from codfw to eqiad -... [14:18:16] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652097 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from codfw to eqiad -... [14:18:53] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652098 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from codfw to e... [14:21:57] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652136 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from codfw to eqia... [14:23:56] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652143 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from codfw to eqiad - fi... [14:34:22] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10652197 (10Trizek-WMF) Confirmed as done, with 2 minutes, 24 seconds of read-only time. @hnowlan and I agreed that someone from SRE - but not the person... [14:40:47] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10652229 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from codfw... [14:59:50] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10652468 (10Trizek-WMF) ==== Debrief how it went, and document * As suggested by @Tacsipacsi on the message talk page, I changed the title of the message... [15:30:46] 06serviceops, 06SRE, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 10Wikimedia-Site-requests: Increase entityAccessLimit for WikibaseClient wikis - https://phabricator.wikimedia.org/T384455#10652638 (10Ladsgroup) Hii, from DBA point of view, I ask that please hold off bumping the number in co... [15:50:43] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10652818 (10Trizek-WMF) [18:36:30] 06serviceops, 07Datacenter-Switchover: Determine switchover changes for migration of video scaling to k8s - https://phabricator.wikimedia.org/T372849#10653971 (10Scott_French) So, this worked out quite similar to T372849#10592983, with an interesting twist that's obvious in retrospect: When we depooled `eventg... [19:33:24] 06serviceops, 10Cassandra, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#10654178 (10Eevans) @AUgolnikova-WMF, @Cparle Any update on this? [19:59:29] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10654250 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2248.codfw.wmnet with... [20:35:18] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10654353 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2248.codfw.wmnet with OS... [23:47:38] 06serviceops, 10Arc-Lamp: Gather PHP 8.1 profiling data - https://phabricator.wikimedia.org/T385199#10655080 (10Krinkle) FYI: {T389243}, \cc @ori >>! In T389243#10648713, @Krinkle wrote: >> This is causing performance.wikimedia.org flamegraphs to show bogus data [on PHP 8.1]. > > Specifically, it means we h...