[07:19:40] 06serviceops, 10envoy, 06SRE, 13Patch-For-Review: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11166132 (10elukey) Ack! Upgraded staging, and pinged the DSE SREs as well on slack to gather their opinion about ownership etc.. [10:06:37] 06serviceops, 10envoy, 06SRE, 13Patch-For-Review: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11166565 (10MoritzMuehlenhoff) All baremetal installations of Envoy have been upgraded [10:17:41] 06serviceops, 06MediaWiki-Platform-Team: Set up a rest-gateway deployment for rate limiting testing - https://phabricator.wikimedia.org/T402914#11166606 (10Clement_Goubert) [10:18:38] 06serviceops, 06MediaWiki-Platform-Team: Set up a rest-gateway deployment for rate limiting testing - https://phabricator.wikimedia.org/T402914#11166613 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High [10:18:41] 06serviceops, 06MediaWiki-Platform-Team: Set up a rest-gateway deployment for rate limiting testing - https://phabricator.wikimedia.org/T402914#11166616 (10Clement_Goubert) a:03Clement_Goubert [11:38:59] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job initsitestats failed - https://phabricator.wikimedia.org/T404026#11166907 (10A_smart_kitten) It's not worth rerunning this job given that (AFAICS) [[https://gerrit.wikimedia.org/g/operations/puppet/+/6fc1e6e5b8f4ffc... [11:54:14] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job initsitestats failed - https://phabricator.wikimedia.org/T404026#11166945 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert New run already deleted the failed job, recovered stack trace from logsta... [14:13:00] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work, 13Patch-For-Review: Improved API rerouting strategy for REST gateway - https://phabricator.wikimedia.org/T400131#11167659 (10ssingh) >>! In T400131#11164901, @Scott_French wrote: > There's been some ongoing discussion (e.g., [[ https://gerrit.wikimedia.org/r/... [14:22:40] 06serviceops, 13Patch-For-Review: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11167706 (10Lucas_Werkmeister_WMDE) >>! In T403284#11164940, @Scott_French wrote: > In terms of workload diversity, focusing on shellbox-constraints for the next pilot and migration wou... [14:51:11] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work, 13Patch-For-Review: Improved API rerouting strategy for REST gateway - https://phabricator.wikimedia.org/T400131#11167908 (10Scott_French) Many thanks for the review, @ssingh! I'll aim to make these changes today, and will reach out to Traffic with a heads-up... [15:39:33] 06serviceops, 06Traffic, 10WE4.2 Bot detection (WE4.2 hCaptcha account creation trial): Investigate options for per-wiki, percentage-based rollout of hCaptcha - https://phabricator.wikimedia.org/T404184#11168134 (10jijiki) [15:40:57] 06serviceops, 06Traffic, 10WE4.2 Bot detection (WE4.2 hCaptcha account creation trial): Investigate options for per-wiki, percentage-based rollout of hCaptcha - https://phabricator.wikimedia.org/T404184#11168146 (10jijiki) [16:33:28] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work, 13Patch-For-Review: Improved API rerouting strategy for REST gateway - https://phabricator.wikimedia.org/T400131#11168540 (10Scott_French) On the second attempt after scaling up proton 1.5x (not yet locked in, which I'll do in a bit), depooling codfw seems to... [16:40:22] 06serviceops, 13Patch-For-Review: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11168574 (10Scott_French) Thank you very much, @Lucas_Werkmeister_WMDE. I'll get the pilot started today. [17:34:54] 06serviceops, 10envoy, 06SRE: Envoy config updates from v1.29 - https://phabricator.wikimedia.org/T404036#11168927 (10RLazarus) And by request from @CDanis, adding to this config update cycle: If `mesh.tracing.service_name` is set in values, pass it through in the `service_name` field of the `OpenTelemetryC... [17:36:22] 06serviceops, 10envoy, 06SRE: Envoy config updates from v1.29 - https://phabricator.wikimedia.org/T404036#11168928 (10RLazarus) [17:37:37] 06serviceops, 10envoy, 06SRE: Envoy config updates from v1.29 - https://phabricator.wikimedia.org/T404036#11168934 (10CDanis) Oh, one other thing we might want to do, if mesh.tracing.service_name is unset, default it to `.Release.Namespace`. This is effectively what happens anyway in the big mess between En... [20:18:55] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work, 13Patch-For-Review: Improved API rerouting strategy for REST gateway - https://phabricator.wikimedia.org/T400131#11169787 (10Scott_French) Alright, it's been about 4h in "soft" single-DC serving mode (i.e., where it's easily revertible by simply repooling cod... [20:45:36] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work, 13Patch-For-Review: Improved API rerouting strategy for REST gateway - https://phabricator.wikimedia.org/T400131#11169897 (10Scott_French) Well, that's it! We're now serving traffic via `rest-gateway.discovery.wmnet` as an A/P service and the `rest-gateway-ro... [21:24:58] 06serviceops, 10envoy, 06SRE: Envoy config updates from v1.29 - https://phabricator.wikimedia.org/T404036#11170002 (10RLazarus) [22:08:56] 06serviceops: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11170193 (10Scott_French) A bit more than 5 hours into the pilot, no issues observed so far with a fraction of shellbox-constraints traffic serving on 8.3. This is based on the same set of monitoring signals... [23:49:40] 06serviceops, 13Patch-For-Review: Configure mw-next-routing for the PHP 8.3 migration - https://phabricator.wikimedia.org/T403655#11170519 (10Scott_French)