[12:25:43] 06Traffic, 10RESTBase, 10RESTBase Sunsetting, 06serviceops: Block traffic to RESTBase /page/related endpoint before it's deprecated - https://phabricator.wikimedia.org/T376297#10528510 (10MSantos) @akosiaris this is ready for work and communication has been sent Today. [12:30:26] 06Traffic, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages - https://phabricator.wikimedia.org/T385395#10528517 (10Ahonc) Are there any news about this case? I still can edit such pages only using API on remote server. It is difficult and uncomfortable. [12:38:47] 06Traffic, 10RESTBase, 10RESTBase Sunsetting, 06serviceops: Block traffic to RESTBase /page/related endpoint before it's deprecated - https://phabricator.wikimedia.org/T376297#10528536 (10Seddon) [12:58:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Dec 2024: cr3-ulsfo errors on et-0/0/0 link from cr4 - https://phabricator.wikimedia.org/T384288#10528624 (10cmooney) 05Open→03Resolved This link has had a reasonable amount of traffic since the move and still error free so I a... [12:58:51] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Homer trying to delete BGP peerings for VMs on new Eqiad ganeti nodes - https://phabricator.wikimedia.org/T381175#10528627 (10cmooney) p:05High→03Low [13:06:39] 06Traffic, 10RESTBase, 10RESTBase Sunsetting, 06serviceops: Block traffic to RESTBase /page/related endpoint before it's deprecated - https://phabricator.wikimedia.org/T376297#10528672 (10akosiaris) @msantos. And it's done. Thanks to [requestctl](https://wikitech.wikimedia.org/wiki/Requestctl) web interfac... [13:54:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10528719 (10cmooney) >>! In T385217#10527341, @Jhancock.wm wrote: > we received two each of these. > > SFP-GIG BASE-T RJ45 > S... [14:01:54] 06Traffic, 06Content-Transform-Team-WIP, 06Data-Persistence, 10iOS-app-feature-Performance, and 7 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365#10528739 (10Jgiannelos) Currently we have testwiki and hewiki to send changeprop events to PCS... [14:08:53] 06Traffic, 06Content-Transform-Team-WIP, 06Data-Persistence, 10iOS-app-feature-Performance, and 7 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365#10528765 (10Jgiannelos) More specifically here is the events produced from my test: resource c... [14:21:36] 06Traffic, 10RESTBase, 10RESTBase Sunsetting, 06serviceops-radar: Block traffic to RESTBase /page/related endpoint before it's deprecated - https://phabricator.wikimedia.org/T376297#10528800 (10akosiaris) Moving to #serviceops-radar as there probably isn't an actionable for Service Ops here anymore. [14:28:24] 06Traffic, 10RESTBase, 10RESTBase Sunsetting, 06serviceops-radar: Block traffic to RESTBase /page/related endpoint before it's deprecated - https://phabricator.wikimedia.org/T376297#10528818 (10Jgiannelos) Thanks all, i will go ahead and deprecate the relevant codebase. [14:50:03] 10netops, 06Infrastructure-Foundations, 10ops-magru: Jan 2025 - Magru core router connectivity blips - https://phabricator.wikimedia.org/T384774#10528885 (10cmooney) Case 2025-0206-449740 opened [15:09:01] 06Traffic, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages - https://phabricator.wikimedia.org/T385395#10528963 (10Vgutierrez) >>! In T385395#10528517, @Ahonc wrote: > Are there any news about this case? I still can edit such pages only using API on remote server. It is difficult an... [15:10:09] 06Traffic, 06collaboration-services, 10Phabricator, 06SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228#10528968 (10LSobanski) [15:30:01] 06Traffic, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages - https://phabricator.wikimedia.org/T385395#10529107 (10Ahonc) I do not know. I cannot use traceroute there. But I also can edit from my smartphone (in Samsung Browser) from the same network as my laptop. [15:39:48] 06Traffic, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages - https://phabricator.wikimedia.org/T385395#10529133 (10Vgutierrez) that's interesting, I've seen in our logs for your 503s that your browser has a `PHP-ENGINE=8.1` cookie, instructing our CDN to send your request to mediawi... [15:45:45] 06Traffic, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages - https://phabricator.wikimedia.org/T385395#10529191 (10Ahonc) Yes. it works if change PHP-ENGINE cookie. [15:48:00] 06Traffic, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages - https://phabricator.wikimedia.org/T385395#10529203 (10Vgutierrez) Adding @Scott_French to the task as he is taking care of the PHP 8.1 migration [15:58:15] 06Traffic, 06collaboration-services, 10Phabricator, 06SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228#10529263 (10DLynch) Incidentally, Kosta and I wound up making [an API proxy](https://toolhub.wikimedia.org/tools/toolforge-phabroxy) t... [16:11:03] 06Traffic, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages - https://phabricator.wikimedia.org/T385395#10529366 (10Scott_French) Thanks @Vgutierrez and @Ahonc! @Ahonc - If edits to the large pages you linked were consistently failing until you altered the `PHP_ENGINE` cookie value... [16:11:21] 06Traffic, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages - https://phabricator.wikimedia.org/T385395#10529367 (10Scott_French) [16:22:07] 06Traffic, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages - https://phabricator.wikimedia.org/T385395#10529426 (10Scott_French) Pulling together some items: @CDanis was able to identify a possible correlate of this searching for the URL-encoded form of one of the repro URLs: https... [16:43:24] 06Traffic, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10529520 (10Scott_French) [16:51:59] 06Traffic, 06collaboration-services, 10Phabricator, 06SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228#10529569 (10taavi) >>! In T274228#10503971, @Aklapper wrote: > Does anyone have sufficient understanding to outline the next potential... [17:27:18] 06Traffic, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10529715 (10Scott_French) To summarize, the user is consistently receiving a 503 response synthesized by varnish after 125s when attempting to edit large pages like the... [17:49:16] 06Traffic, 06MediaWiki-Engineering, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10529767 (10Scott_French) [17:59:44] 06Traffic, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 3 others: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10529813 (10EAkinloose) [18:47:49] 06Traffic, 06MediaWiki-Engineering, 06SRE, 13Patch-For-Review, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10529999 (10mszabo) >>! In T385395#10529425, @Scott_French wrote: > Pulling together some items: > > @CDanis was able t... [18:51:58] 06Traffic, 06MediaWiki-Engineering, 06SRE, 13Patch-For-Review, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530008 (10ssingh) >>! In T385395#10529715, @Scott_French wrote: > To summarize, the user is consistently receiving a 5... [18:59:42] 06Traffic, 06MediaWiki-Engineering, 06SRE, 13Patch-For-Review, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530019 (10mszabo) I'm boldly thinking of pasting all that wikitext into an user subpage and trying to save that on mwd... [19:24:23] 06Traffic, 06MediaWiki-Engineering, 06SRE, 13Patch-For-Review, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530073 (10Scott_French) Thank you both! @ssingh - Interesting, so it sounds like the expected result of hitting `idle... [19:59:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10530122 (10Jhancock.wm) all of the **External fibre patches** are completed. all of the **In rack mgmt connections** are comple... [20:05:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10530131 (10cmooney) Thanks Jenn!! Yep once things are in Netbox I'll be able to try connecting from here and will let you know... [20:28:20] 06Traffic, 06MediaWiki-Engineering, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530161 (10Ahonc) @ssingh there was 125 seconds before 503: https://phabricator.wikimedia.org/T385395#10522170 [20:30:18] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10530163 (10Jhancock.wm) if we can get the juniper DAC to work while testing, we won't need to. However if it doesn't work we'll... [20:37:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10530189 (10cmooney) >>! In T385217#10530163, @Jhancock.wm wrote: > if we can get the juniper DAC to work while testing, we won'... [21:39:09] 06Traffic, 06MediaWiki-Engineering, 06SRE, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530334 (10mszabo) We've debugged this with @Scott_French a bit. Attempting to edit https://uk.wikipedia.org/w/index.php?title=%D0%A1%D0%BF%D... [21:43:54] 06Traffic, 06MediaWiki-Engineering, 06SRE, 13Patch-For-Review, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530335 (10ssingh) Thanks for the update @mszabo and @Scott_French! At least this is something. I don't have good answe... [22:27:19] 06Traffic, 06MediaWiki-Engineering, 06SRE, 13Patch-For-Review, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530488 (10BCornwall) Possibly related: https://phabricator.wikimedia.org/T383013 [22:27:36] 06Traffic, 06MediaWiki-Engineering, 06serviceops, 13Patch-For-Review, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530496 (10Scott_French) p:05Triage→03High Many thanks again, all, for the investigation today. So, we're a... [22:29:29] 06Traffic, 06MediaWiki-Engineering, 06serviceops, 13Patch-For-Review, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530540 (10Scott_French) @BCornwall - Likely not related to the 8.1 migration, since we didn't start ramping tra... [22:55:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Multiple host down alerts from rack C2 - https://phabricator.wikimedia.org/T279457#10530701 (10Papaul)