[08:56:56] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: shellbox-video pods being restarted prematurely - https://phabricator.wikimedia.org/T373517#10249262 (10TheDJ) Are the http requests using chunked transfer encoding. or not ? (I'm assuming its all http 1.1 and 2.0) [09:03:05] 06serviceops, 06Data-Engineering, 10Proton, 10Recommendation-API, and 2 others: WikiKube: Rename the last few "production" named helm releases to use "main" instead - https://phabricator.wikimedia.org/T377805#10249287 (10akosiaris) [09:09:06] 06serviceops, 06Data-Engineering, 10Proton, 10Recommendation-API, and 2 others: WikiKube: Rename the last few "production" named helm releases to use "main" instead - https://phabricator.wikimedia.org/T377805#10249299 (10akosiaris) p:05Triage→03Medium [10:49:23] 06serviceops, 06Infrastructure-Foundations, 06SRE, 07Datacenter-Switchover, 13Patch-For-Review: sre.discovery.datacenter should support switching the active/passive services to the other datacenter - https://phabricator.wikimedia.org/T335364#10249621 (10Clement_Goubert) 05In progress→03Resolved [11:52:12] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: shellbox-video pods being restarted prematurely - https://phabricator.wikimedia.org/T373517#10249864 (10hnowlan) >>! In T373517#10249262, @TheDJ wrote: > Are the http requests using [[ https://en.wikipedia.org/wiki/Chunked_trans... [12:08:41] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10249931 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2085.codfw.wmnet with OS bo... [12:08:49] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10249932 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2086.codfw.wmnet with OS bo... [12:09:13] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10249949 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2088.codfw.wmnet with OS bo... [12:10:12] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10249951 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2089.codfw.wmnet with OS bo... [12:45:26] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10250037 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2088.codfw.wmnet with OS bookwo... [12:50:49] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10250054 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2085.codfw.wmnet with OS bookwo... [12:53:40] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10250060 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2086.codfw.wmnet with OS bookwo... [12:55:55] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10250075 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2089.codfw.wmnet with OS bookwo... [13:22:56] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10250154 (10cmooney) >>! In T375845#10246786, @akosiaris wrote: > Good question. Let me add some data points. We currently use... [14:56:33] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Cookbook to roll-reimage k8s nodes - https://phabricator.wikimedia.org/T377857 (10kamila) 03NEW [15:26:02] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: shellbox-video pods being restarted prematurely - https://phabricator.wikimedia.org/T373517#10250917 (10hnowlan) I've mocked up a horrible Frankenstein script that mimics the TimedMediaHandler behaviour - when directly calling s... [15:59:20] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10251117 (10JMeybohm) [16:11:22] 06serviceops, 06Machine-Learning-Team, 10Data-Platform-SRE (2024.10.19 - 2024.11.08), 07Security: Migrate the ownership of DPE-Owned Docker images in production-images repo to mailing lists - https://phabricator.wikimedia.org/T373534#10251162 (10BTullis) [16:13:18] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10251171 (10cmassaro) Okay, I've managed to get your example running by replacing ` deny w/** ` with ` deny /[^d][^e][^v]/[^sn][^tu][^dl]** w, ` so leaving `/dev/null`, `/dev/stdin`, a... [16:19:51] 06serviceops, 06Data-Engineering, 10Proton, 10Recommendation-API, and 2 others: WikiKube: Rename the last few "production" named helm releases to use "main" instead - https://phabricator.wikimedia.org/T377805#10251202 (10Ottomata) Thank you! Please let us know when you plan to do eventgate-*s. IIRC, ther... [16:32:39] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: shellbox-video pods being restarted prematurely - https://phabricator.wikimedia.org/T373517#10251266 (10hnowlan) When connecting the same client to a k8s pod IP, the encoding and download of the file complete successfully, so so... [16:37:19] 06serviceops, 06Data-Persistence, 13Patch-For-Review: Sessionstore's discovery TLS cert will expire before end of May 2024 - https://phabricator.wikimedia.org/T363996#10251270 (10Eevans) `lang=sh-session eevans@deploy1003:~$ siege -f T363996-urls.txt -i -c 64 -t 2m -d 0.1 ** SIEGE 4.0.7 ** Preparing 64 concu... [16:41:11] 06serviceops, 10MW-on-K8s: Functional replacement for importImages.php on Kubernetes - https://phabricator.wikimedia.org/T377497#10251281 (10Joe) >>! In T377497#10248513, @Pppery wrote: > That documentation isn't quite accurate. The goal of server-side uploads as they are used today is to work around the fact... [16:42:50] 06serviceops, 10Prod-Kubernetes, 07Epic, 07Kubernetes: [EPIC] Docker deprecation as a container runtime enginer for kubernetes. - https://phabricator.wikimedia.org/T269684#10251277 (10JMeybohm) a:03JMeybohm [16:53:41] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876 (10JMeybohm) 03NEW [16:53:43] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877 (10JMeybohm) 03NEW [16:57:43] 06serviceops, 06Data-Persistence, 13Patch-For-Review: Sessionstore's discovery TLS cert will expire before end of May 2024 - https://phabricator.wikimedia.org/T363996#10251366 (10Eevans) `lang=sh-session eevans@deploy1003:~$ siege -f T363996-urls.txt -i -c 64 -t 2m -d 0.1 ** SIEGE 4.0.7 ** Preparing 64 concu... [17:17:02] 06serviceops, 06Machine-Learning-Team: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471#10251472 (10akosiaris) To keep the archives happy, unless I am mistaken, per {T373611} Android applications have moved from the old recommendation-api to a... [17:58:23] 06serviceops, 06Data-Persistence, 13Patch-For-Review: Sessionstore's discovery TLS cert will expire before end of May 2024 - https://phabricator.wikimedia.org/T363996#10251662 (10hnowlan) eqiad is currently using the mesh - codfw is not. We decided to leave this config in place for the evening to get certain... [18:44:11] 06serviceops, 10MW-on-K8s: Support machine-readable output for mwscript-k8s - https://phabricator.wikimedia.org/T377292#10251796 (10RLazarus) 05Open→03Resolved This is ready to use, and documented (including the JSON output format) at https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Shelling_out... [20:14:35] 06serviceops, 13Patch-For-Review: Extend x-wikimedia-debug-routing.lua to support PHP 8.1 mw-debug deployment - https://phabricator.wikimedia.org/T372605#10252063 (10Scott_French) This now works via the x-wikimedia-debug "k8s-mwdebug-next" backend (plus -codfw and -eqiad variants). To avoid any confusion about... [22:00:30] 06serviceops, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 2 others: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#10252263 (10BTullis) >>! In T352650#10058764, @xcollazo wrote: > I think there were 2 ideas: > > # Chase... [23:25:45] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10252448 (10cmassaro) Hmm, ignore the above. I realized my mistake. The set of rules I'm now running (instead of `deny /** w` is ` # Deny all file writes except for /dev/nul*, /dev/std*...