[08:56:31] 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Write a cookbook to set a k8s cluster in maintenance mode - https://phabricator.wikimedia.org/T277677 (10JMeybohm) [08:56:47] 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 3 others: Create a cookbook for depooling one or all services from one kubernetes cluster - https://phabricator.wikimedia.org/T260663 (10JMeybohm) 05Open→03Resolved Merged as `sre.k8s.pool-depool-cluster` [09:02:43] 10serviceops, 10Wikimedia-Incident: Incident: 2022-09-08 codfw appservers degradation - https://phabricator.wikimedia.org/T317340 (10Clement_Goubert) [09:02:46] 10serviceops, 10SRE-OnFire, 10Sustainability (Incident Followup): Page on etcdmirror critical status - https://phabricator.wikimedia.org/T317402 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert [11:04:41] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar), 10Performance-Team-publish: Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10Clement_Goubert) {F35519031} Last 30 days of parsoid timeouts, with migration steps added. It's still a small sample size, but... [12:28:25] _joe_: claime: hello! in half an hour I'm hosting the incident review ritual and one of the incidents we'll be talking about is 2022-09-08_codfw_appservers_degradation. if one or both of you could be there that would be great but no worries if not :) [12:59:32] cdanis: I'll be there, j.oe probably won't, he's in berlin [12:59:45] ah right berlin [12:59:56] and of course ak.osiaris will be there too :) [16:46:37] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [16:46:59] 10serviceops, 10Observability-Alerting, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Migrate kubernetes alerts away from icinga - https://phabricator.wikimedia.org/T311251 (10JMeybohm) 05Resolved→03Open Unfortunately KubernetesAPILatency fires from time to time. Ordered by cluster and timesta... [18:22:05] 10serviceops, 10API Platform, 10Growth-Structured-Tasks, 10Image-Suggestions, and 7 others: GrowthExperiments\NewcomerTasks\AddImage\ServiceImageRecommendationProvider::get Unable to decode JSON response for page {title} upstream connect error or disconnect/reset b... - https://phabricator.wikimedia.org/T313973 [18:28:26] 10serviceops, 10API Platform, 10Growth-Structured-Tasks, 10Image-Suggestions, and 7 others: GrowthExperiments\NewcomerTasks\AddImage\ServiceImageRecommendationProvider::get Unable to decode JSON response for page {title} upstream connect error or disconnect/reset b... - https://phabricator.wikimedia.org/T313973 [21:34:24] 10serviceops, 10Release-Engineering-Team (Radar): Pushes to docker-registry are too slow - https://phabricator.wikimedia.org/T306201 (10dancy) 05Open→03Resolved a:03dancy I'm closing this ticket. Recent image pushes have had satisfactory performance. [21:58:47] 10serviceops, 10MW-on-K8s, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹ī¸): Make scap deploy to kubernetes together with the legacy systems - https://phabricator.wikimedia.org/T299648 (10dancy) @joe (and others) Is there any objection to scap running `helmfile apply` for both eqiad and codf... [22:33:07] 10serviceops, 10MW-on-K8s, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹ī¸): Make scap deploy to kubernetes together with the legacy systems - https://phabricator.wikimedia.org/T299648 (10dancy) Notes on the impact on deployer experience when scap config flags `build_mw_container_image` and...