[00:48:01] 06serviceops, 10envoy, 06SRE, 13Patch-For-Review: Envoy config updates from v1.26 - https://phabricator.wikimedia.org/T403101#11160842 (10RLazarus) [01:10:13] 06serviceops, 10envoy, 06SRE: Envoy config updates from v1.29 - https://phabricator.wikimedia.org/T404036 (10RLazarus) 03NEW [01:18:15] 06serviceops, 10envoy, 06SRE: Envoy config updates from v1.29 - https://phabricator.wikimedia.org/T404036#11160892 (10RLazarus) Note the `event_log_path` comes up in `mesh.configuration._tcp_cluster`, which pulls in the entire `health_checks` field from values.yaml, so that's where the `event_log_path`s are... [01:39:20] 06serviceops, 10envoy, 06SRE: Envoy config updates from v1.29 - https://phabricator.wikimedia.org/T404036#11160928 (10RLazarus) The global_downstream_max_connections was deprecated in the 1.28 release notes, but as of 1.29, the downstream connections resource_monitor was still a work in progress. So we won't... [01:39:50] 06serviceops, 10envoy, 06SRE: Envoy config updates from v1.29 - https://phabricator.wikimedia.org/T404036#11160929 (10RLazarus) [07:30:48] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10GitLab (Integrations), 10Release-Engineering-Team (Radar): Container image reports in debmonitor are broken - https://phabricator.wikimedia.org/T348876#11161195 (10elukey) 05Open→03Resolved Getting back to this after all the... [08:54:34] 06serviceops, 06Release-Engineering-Team, 06SRE: docker-registry "Last updated at" text hiding under scrollbar - https://phabricator.wikimedia.org/T404008#11161533 (10LSobanski) [08:54:43] 06serviceops, 06Release-Engineering-Team, 06SRE: docker-registry "Last updated at" time should specify TZ - https://phabricator.wikimedia.org/T404010#11161536 (10LSobanski) [08:54:51] 06serviceops, 06Release-Engineering-Team, 06SRE: docker-registry will show different last updated time as you refresh the page... - https://phabricator.wikimedia.org/T404011#11161537 (10LSobanski) [10:04:06] 06serviceops, 10envoy, 06SRE, 13Patch-For-Review: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11161791 (10elukey) @RLazarus Hi! I found this task while trying to remove all the occurrences of `1.23.10-2-s4-20231203` still running on the k8s clusters. Are you going t... [10:11:47] 06serviceops, 10envoy, 06SRE, 13Patch-For-Review: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11161822 (10elukey) I noticed the following occurrences of Buster images: dse-k8s-eqiad-152-namespace: datasets-config dse-k8s-eqiad-157-namespace: datasets-config-next e... [10:40:10] 06serviceops, 07sre-alert-triage: Alert in need of triage: KubernetesWorkerUnschedulable - https://phabricator.wikimedia.org/T400969#11161973 (10Clement_Goubert) Silenced for 30 days again [10:56:50] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission mwmaint1002.eqiad.wmnet - https://phabricator.wikimedia.org/T400442#11162050 (10VRiley-WMF) 05Open→03Resolved [15:13:43] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-18 (2025-09-09 to 2025-09-23)), 07OKR-Work: [SPIKE] Test plan for rest.php routes in REST gateway - https://phabricator.wikimedia.org/T400152#11163287 (10OWresch-WMF) [15:14:22] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 10RESTBase Sunsetting, and 2 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#11163293 (10OWresch-WMF) [15:50:00] 06serviceops, 10envoy, 06SRE: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11163532 (10RLazarus) @elukey Thank you! Looks like an ownership issue, and yes please if you're comfortable deploying those, I'll take you up on it. (We were just talking in serviceops about th... [16:24:33] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11163850 (10Krinkle) [16:25:57] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11163864 (10Krinkle) [17:27:01] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-18 (2025-09-09 to 2025-09-23)), 07OKR-Work: [SPIKE] Test plan for rest.php routes in REST gateway - https://phabricator.wikimedia.org/T400152#11164160 (10aaron) FYI, here are the supported verb types for the current APIs: ` /campaignevents/v0/event_registration... [17:59:01] 06serviceops: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11164363 (10Scott_French) As of a bit before 17:40 UTC, shellbox-syntaxhighlight has been migrated to PHP 8.3 in both DCs. So far, everything is looking good in terms of: * General service health ([[ https:/... [18:06:45] 06serviceops: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11164393 (10Scott_French) [18:22:27] 06serviceops: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11164495 (10Scott_French) 05Open→03In progress [19:27:36] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work, 13Patch-For-Review: Improved API rerouting strategy for REST gateway - https://phabricator.wikimedia.org/T400131#11164901 (10Scott_French) There's been some ongoing discussion (e.g., [[ https://gerrit.wikimedia.org/r/c/operations/dns/+/1182853/comments/059052... [19:37:57] 06serviceops, 13Patch-For-Review: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11164940 (10Scott_French) In terms of workload diversity, focusing on shellbox-constraints for the next pilot and migration would be ideal - i.e., a service for which clients exercise t... [20:59:17] 06serviceops, 06Release-Engineering-Team, 06SRE: docker-registry will show different last updated time as you refresh the page... - https://phabricator.wikimedia.org/T404011#11165251 (10bd808) `lang=shell-session bd808@deploy1003:~$ curl -s https://docker-registry.discovery.wmnet | grep 'Last updated at'... [21:29:35] 06serviceops, 06Release-Engineering-Team, 06SRE: docker-registry will show different last updated time as you refresh the page... - https://phabricator.wikimedia.org/T404011#11165359 (10bd808) Those content pages are generated by [[https://gerrit.wikimedia.org/g/operations/puppet/+/refs/heads/production/modu... [21:31:21] 06serviceops, 06Release-Engineering-Team, 06SRE: docker-registry will show different last updated time as you refresh the page... - https://phabricator.wikimedia.org/T404011#11165361 (10bd808) 05Open→03Invalid Closing as "works as designed". I guess reopen if the timer skew feels problematic and not...