[05:42:08] 06serviceops, 10Dumps-Generation, 10MW-on-K8s, 06Release-Engineering-Team: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#9959610 (10Joe) p:05Medium→03High The priority of this task has become high, as doing this is currently a blocker... [08:29:30] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review, 07Security: Upgrade K8s docker images to running in production on Buster with either Bullseye or Bookworm - https://phabricator.wikimedia.org/T368366#9959882 (10elukey) To keep archives happy: Alex removed the fluend image and configs since no... [08:29:48] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review, 07Security: Upgrade K8s docker images to running in production on Buster with either Bullseye or Bookworm - https://phabricator.wikimedia.org/T368366#9959891 (10elukey) [08:35:02] o/ folks [08:35:20] if you don't have anything against it, I'd deploy thumbor and api/rest gateway today in prod [08:35:46] thumbor (haproxy on bookworm) and api/rest gateway (envoy on bookworm) have been tested by Hugh last week in staging, all good IIRC [08:48:12] elukey: please go ahead :) [09:39:19] ack! [09:39:28] also deployed wikifeeds (envoy on bookworm canary) [10:22:23] 06serviceops, 10Prod-Kubernetes, 13Patch-For-Review: PodSecurityPolicies will be deprecated with Kubernetes 1.21 - https://phabricator.wikimedia.org/T273507#9960265 (10JMeybohm) [10:23:31] 06serviceops, 10Prod-Kubernetes, 13Patch-For-Review: PodSecurityPolicies will be deprecated with Kubernetes 1.21 - https://phabricator.wikimedia.org/T273507#9960279 (10JMeybohm) 05Open→03Stalled [10:54:04] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9960396 (10Lucas_Werkmeister_WMDE) {T355292} should probably be a subtask of this (or maybe a subtask of T321899)? At least I’ve been told th... [10:54:56] elukey: you 'll also note the chart bump for api-gateway (versions aside a noop). [10:55:19] I already deployed staging on Friday, but given your work on envoy I avoided production [10:56:16] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Port videoscaling to kubernetes - https://phabricator.wikimedia.org/T355292#9960409 (10Clement_Goubert) [10:56:23] 06serviceops, 10MoveComms-Support, 10MW-on-K8s, 06SRE, and 2 others: Move 100% of external traffic to Kubernetes - https://phabricator.wikimedia.org/T362323#9960410 (10Clement_Goubert) [10:56:25] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9960411 (10Clement_Goubert) [10:58:39] 06serviceops, 10MW-on-K8s, 10Release-Engineering-Team (Seen): Create mw-videoscaler helmfile deployment - https://phabricator.wikimedia.org/T321899#9960415 (10Clement_Goubert) →14Duplicate dup:03T355292 [11:00:30] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Port videoscaling to kubernetes - https://phabricator.wikimedia.org/T355292#9960417 (10Clement_Goubert) [11:09:20] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Port videoscaling to kubernetes - https://phabricator.wikimedia.org/T355292#9960447 (10Clement_Goubert) [11:09:24] 06serviceops, 10MoveComms-Support, 10MW-on-K8s, 06SRE, and 2 others: Move 100% of external traffic to Kubernetes - https://phabricator.wikimedia.org/T362323#9960448 (10Clement_Goubert) [11:16:34] 06serviceops, 10MoveComms-Support, 10MW-on-K8s, 06SRE, and 2 others: Move 100% of external traffic to Kubernetes - https://phabricator.wikimedia.org/T362323#9960456 (10Clement_Goubert) [11:21:19] 06serviceops, 10MoveComms-Support, 10MW-on-K8s, 06SRE, and 2 others: Move 100% of external traffic to Kubernetes - https://phabricator.wikimedia.org/T362323#9960472 (10Clement_Goubert) 05Open→03Resolved The work this task tracked is now completed. Remaining migrations {T352650}, {T355292}, {T355292... [11:29:07] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9960486 (10Clement_Goubert) 05Open→03In progress [11:33:06] 06serviceops, 10MW-on-K8s, 10Release-Engineering-Team (Priority Backlog 📥): Progressive rollout of MediaWiki deployment on Kubernetes - https://phabricator.wikimedia.org/T276487#9960499 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert This functionality has been added to scap. [11:33:18] 06serviceops: Support Canary releases on Kubernetes - https://phabricator.wikimedia.org/T282148#9960503 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert We can get prom metrics using the `release` label. Boldly closing. [11:33:56] 06serviceops, 06MediaWiki-Platform-Team: Benchmark baremetal vs k8s mediawiki perf (2023) - https://phabricator.wikimedia.org/T333269#9960510 (10Clement_Goubert) 05Stalled→03Invalid All traffic has been migrated to #mw-on-k8s [11:34:26] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9960507 (10Clement_Goubert) 05In progress→03Resolved All internal traffic has been migrated. [11:34:29] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Limit the concurrency of envoy in service mesh - https://phabricator.wikimedia.org/T354532#9960513 (10Clement_Goubert) [11:34:42] 06serviceops, 10MW-on-K8s: mw-on-k8s tls-proxy container CPU throttling at low average load - https://phabricator.wikimedia.org/T344814#9960517 (10Clement_Goubert) [11:34:43] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Limit the concurrency of envoy in service mesh - https://phabricator.wikimedia.org/T354532#9960516 (10Clement_Goubert) [11:36:02] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786#9960518 (10Clement_Goubert) [11:36:05] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Port videoscaling to kubernetes - https://phabricator.wikimedia.org/T355292#9960519 (10Clement_Goubert) [11:37:52] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9960524 (10Clement_Goubert) >>! In T290536#9960396, @Lucas_Werkmeister_WMDE wrote: > {T355292} should probably be a subtask of this (or maybe... [11:41:40] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9960525 (10Clement_Goubert) [11:41:52] 06serviceops, 06Content-Transform-Team, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9960531 (10hnowlan) [11:43:50] 06serviceops, 06Content-Transform-Team, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9960532 (10hnowlan) 05Open→03Stalled Are there plans on when and how we need to move... [12:00:28] 06serviceops, 10MoveComms-Support, 10MW-on-K8s, 06SRE, and 2 others: Move 100% of external traffic to Kubernetes - https://phabricator.wikimedia.org/T362323#9960571 (10Lucas_Werkmeister_WMDE) /me shakes fist at Phorge for not letting me award this task another token 🪙🪙🪙🪙🪙 [12:03:44] 06serviceops, 06Content-Transform-Team, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9960577 (10Jgiannelos) This is stalled because we are first trying to disable pregenerati... [12:37:05] 06serviceops, 07SecTeam-Processed, 07Security: Password to keystore of java certificates needs changing - https://phabricator.wikimedia.org/T361328#9960646 (10Aklapper) [12:47:12] akosiaris: o/ yep yep I did see it thanks! [12:59:30] 06serviceops, 06Content-Transform-Team, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9960706 (10Jgiannelos) FWIW I just finished running diff testing (~5000 titles) between p... [13:14:02] 06serviceops, 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service, 10wmde-wikidata-tech: Use Envoy instead of LVS to route internal federation traffic for WDQS - https://phabricator.wikimedia.org/T368972#9960770 (10Gehel) [13:18:08] 06serviceops, 10Wikifeeds, 13Patch-For-Review: Wikifeeds' tls proxy cpu usage heavily increased in April - https://phabricator.wikimedia.org/T368238#9960784 (10elukey) 05Open→03Resolved a:03elukey There was another ~150ms drop, the tls proxy is still throttled but I feel that now it is way better t... [13:18:28] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade thumbor Docker images - https://phabricator.wikimedia.org/T369144#9960789 (10elukey) 05Open→03Resolved a:03elukey All deployed! [13:44:29] 06serviceops, 06MW-Interfaces-Team, 06Traffic, 13Patch-For-Review: map the /api/ prefix to /w/rest.php - https://phabricator.wikimedia.org/T364400#9960984 (10akosiaris) >>! In T364400#9926454, @Joe wrote: >>>! In T364400#9780622, @BBlack wrote: >>>>! In T364400#9779996, @hnowlan wrote: >>> Could we impleme... [15:55:02] 06serviceops, 06SRE, 10Data Products (Data Products Sprint 16), 13Patch-For-Review, 07Service-deployment-requests: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production - https://phabricator.wikimedia.org/T361835#9961705 (10Scott_French) Thanks, @SGupta-WMF! @mforns - The v1.0.1 image is n... [18:34:51] 06serviceops, 10Wikidata, 10wmde-wikidata-tech, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 03Discovery-Search (Current work): Request permission to create 4 kafka topics in kafka-main (WDQS graph split) - https://phabricator.wikimedia.org/T367510#9962532 (10Gehel) [18:37:10] 06serviceops, 06Infrastructure-Foundations, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 13Patch-For-Review: Create a helm chart for the cloudnativepg postgresql operator - https://phabricator.wikimedia.org/T364797#9962549 (10Gehel) [21:46:01] 06serviceops, 06SRE, 10Data Products (Data Products Sprint 16), 13Patch-For-Review, 07Service-deployment-requests: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production - https://phabricator.wikimedia.org/T361835#9963189 (10mforns) @Scott_French Thank you! We would like to bring up the prod...