[01:18:53] 06serviceops: Move conf2005 within the same rack - https://phabricator.wikimedia.org/T387416 (10Scott_French) 03NEW [01:19:14] 06serviceops: Move conf2005 within the same rack - https://phabricator.wikimedia.org/T387416#10585573 (10Scott_French) p:05Triage→03Medium [02:46:16] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10585743 (10Scott_French) It has now been over 4h since reverting shellbox-media to 7.4, and no further errors of the kind in T377038#10584869 have been observed. I'll return to this t... [08:45:53] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 07Epic: Add page namespace information on resource change events - https://phabricator.wikimedia.org/T387435 (10Jgiannelos) 03NEW [08:46:51] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 07Epic: Adapt changeprop rules to only pregenerate content on main namespace - https://phabricator.wikimedia.org/T387436 (10Jgiannelos) 03NEW [08:47:45] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 07Epic: Adapt changeprop rules to purge content on resource changes for non main namespace events - https://phabricator.wikimedia.org/T387437 (10Jgiannelos) 03NEW [08:48:32] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 07Epic: Add cache purge support on nodejs cassandra storage middleware - https://phabricator.wikimedia.org/T387438 (10Jgiannelos) 03NEW [09:40:37] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10586315 (10Jelto) I was able to build `helm3` version `3.17` on the build host. However there is a issue with installing the proper `... [10:50:11] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10586474 (10cmooney) I don't think it should matter to have the same setting for all interfaces on the box. As I understand it we can... [11:00:16] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10586504 (10Lucas_Werkmeister_WMDE) FWIW, “Shellbox server returned incorrect Content-Type” can mean a lot of different errors – @AudreyPenven_WMDE and I ran into it once, and it turned... [11:28:52] 06serviceops, 07Datacenter-Switchover: SRE comms for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T385157#10586543 (10Trizek-WMF) > Create a CommRel Phabricator task (see Switch Datacenter/Coordination#Notes) I started the process off-record, the Phab task would be welcomed as we need 3... [11:42:31] 06serviceops, 07Datacenter-Switchover: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444 (10hnowlan) 03NEW [11:42:55] 06serviceops, 07Datacenter-Switchover: SRE comms for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T385157#10586580 (10hnowlan) >>! In T385157#10586543, @Trizek-WMF wrote: >> Create a CommRel Phabricator task (see Switch Datacenter/Coordination#Notes) > I started the process off-record,... [11:43:08] 06serviceops, 07Datacenter-Switchover: SRE comms for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T385157#10586581 (10hnowlan) [11:44:11] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10586593 (10akosiaris) > Needs to have rp_filter off (0) or in "loose" mode (2) as pods want to send packets from the service VIP, whi... [11:57:33] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10586605 (10cmooney) >>! In T352956#10586593, @akosiaris wrote: > This isn't true. Pods do not see the service VIP ever. Traffic reach... [13:41:15] 06serviceops: Move conf2005 within the same rack - https://phabricator.wikimedia.org/T387416#10586863 (10JMeybohm) I would say that you only need to restart all the confd's after conf2005 is back up. The issue with confd is, IIRC, that it stops checking etcd servers that have been unreachable at some point (so i... [13:53:32] 06serviceops, 10MediaWiki-Uploading, 06SRE: Reproducible blocking error using the basic upload form, no upload possible - https://phabricator.wikimedia.org/T387007#10586915 (10Vgutierrez) Thanks for reporting the issue @Grand-Duc, from what I'm seeing your request to `https://commons.wikimedia.org/wiki/Speci... [13:59:40] 06serviceops, 06SRE, 10Wikimedia-Apache-configuration, 10Wikimedia-Portals, and 2 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10586951 (10Gehel) [14:48:43] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10587179 (10Jelto) @JMeybohm Helm should be available now in version 3.11 and 3.17 as dedicated packages `helm311` and `helm317`: ` s... [15:12:10] 06serviceops, 06Infrastructure-Foundations, 10Maps (Kartotherian), 13Patch-For-Review: Scale up Kartotherian on Wikikube and move live traffic to it - https://phabricator.wikimedia.org/T386926#10587241 (10SLopes-WMF) [15:33:45] 06serviceops, 06SRE, 10Wikimedia-Apache-configuration, 10Wikimedia-Portals, and 2 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10587407 (10Pcoombe) 05Open→03Resolved a:03simon04 `search` is working a... [15:35:12] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 07Epic: Add time jitter on TTL when invalidating caches on PCS - https://phabricator.wikimedia.org/T387472 (10Jgiannelos) 03NEW [15:40:57] 06serviceops, 10Page Content Service, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Rollout more wikis after week 1 of testing with production traffic - https://phabricator.wikimedia.org/T387277#10587452 (10Jgiannelos) [15:41:22] 06serviceops, 10Page Content Service, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Rollout more wikis after week 1 of testing with production traffic - https://phabricator.wikimedia.org/T387277#10587458 (10Jgiannelos) a:03Jgiannelos [16:02:31] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10587631 (10Jhancock.wm) @Scott_French honestly, since everything else went so well, we don't need to move i... [16:06:09] 06serviceops, 07Datacenter-Switchover: SRE comms for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T385157#10587655 (10Trizek-WMF) Thank you! [16:11:36] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Ensure tls-proxy container is started before launching main container - https://phabricator.wikimedia.org/T387208#10587678 (10Clement_Goubert) Had to revert the mediawiki change as `scap` uses `MWScript.php` in a few places and this breaks it since there's no me... [16:15:09] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Ensure tls-proxy container is started before launching main container - https://phabricator.wikimedia.org/T387208#10587688 (10Joe) >>! In T387208#10587678, @Clement_Goubert wrote: > Had to revert the mediawiki change as `scap` uses `MWScript.php` in a few places... [16:47:15] 06serviceops, 06Fundraising-Backlog, 10fundraising-tech-ops: Update applepay verification code for donate wiki - https://phabricator.wikimedia.org/T387496 (10Dwisehaupt) 03NEW [16:51:05] 06serviceops, 06Fundraising-Backlog, 10fundraising-tech-ops, 13Patch-For-Review: Update applepay verification code for donate wiki - https://phabricator.wikimedia.org/T387496#10588036 (10Dwisehaupt) p:05Triage→03High Changeset is available. going to have fr-tech folks review shortly. [17:09:47] 06serviceops, 07Datacenter-Switchover: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10588174 (10Trizek-WMF) 05Open→03In progress a:05hnowlan→03Trizek-WMF @hnowlan, any notable changes since the last switchover? [17:11:32] 06serviceops, 07Datacenter-Switchover: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10588184 (10Trizek-WMF) [17:19:08] 06serviceops, 07Datacenter-Switchover: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10588200 (10Trizek-WMF) [17:19:18] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10588202 (10Trizek-WMF) [17:25:38] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588224 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-disable-puppet for datacenter switchover from eqiad to codfw - finished with status:... [17:26:16] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588230 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from eqiad to codfw - finished... [17:26:54] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588232 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw - finished with... [17:30:16] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10588259 (10Scott_French) As of ~ 15:40 UTC, traffic on the mw-api-ext / mw-web next releases has stabilized at the 50% enrollment mark. As before, this seems to correspond to ~ 15% of... [17:30:47] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10588264 (10Scott_French) [17:32:44] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588278 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw - finished with status: SUC... [17:33:14] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588279 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw - finished with statu... [17:34:17] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588281 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw - [DRY-RUN] MediaWiki rea... [17:34:29] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588282 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw - finished with status: S... [17:35:34] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588285 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from eqiad to codfw - finished with status... [17:36:17] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588287 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from eqiad to codfw - finished with statu... [17:36:25] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588291 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from eqiad to codfw - finished with statu... [17:36:47] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588292 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw - [DRY-RUN] MediaWiki re... [17:36:50] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588293 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw - finished with status:... [17:42:07] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588318 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from eqiad to codfw - finished with s... [17:45:13] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588341 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from eqiad to codfw - finished with stat... [17:46:23] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588343 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw - finished with status: SU... [17:57:56] 06serviceops, 07Datacenter-Switchover: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10588390 (10ops-monitoring-bot) hnowlan@cumin2002 - Cookbook cookbooks.sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from eqiad to codfw - finished wi... [21:36:16] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10589041 (10Scott_French) Thanks, Lucas! Agreed, yeah, given the way the client error handling works, there's a lot that's potentially masked by "Shellbox server returned incorrect Con...