[06:43:27] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team (Seen): Improve performance of deployment to mw on k8s - https://phabricator.wikimedia.org/T323349 (10Joe) [06:43:36] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team (Seen): Improve performance of deployment to mw on k8s - https://phabricator.wikimedia.org/T323349 (10Joe) p:05Triage→03High [06:44:40] 10serviceops, 10Prod-Kubernetes: sextant: tool for helm chart management - https://phabricator.wikimedia.org/T320793 (10Joe) 05In progress→03Resolved @sgu [06:44:42] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Better management for helm charts - https://phabricator.wikimedia.org/T320782 (10Joe) [06:46:00] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Better management for helm charts - https://phabricator.wikimedia.org/T320782 (10Joe) The basic implementation is done; closing the task as I don't really see a point keeping it open for the conversion of the various charts. [06:46:08] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Better management for helm charts - https://phabricator.wikimedia.org/T320782 (10Joe) 05In progress→03Resolved [06:55:12] 10serviceops: Put mw14[57-98] in production - https://phabricator.wikimedia.org/T313327 (10Joe) p:05Medium→03High Raising priority, hoping someone will take care of this. If no one does, I'll start next week. [08:14:46] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team (Seen): Improve performance of deployment to mw on k8s - https://phabricator.wikimedia.org/T323349 (10Joe) a:03Joe [08:18:39] jayme: o/ [08:18:50] morning :) when you have a moment lemme know if https://phabricator.wikimedia.org/T322920#8403236 is ok or not [08:19:38] hi [08:43:35] Mornin' [08:59:07] elukey: yeah, that sounds pretty sane :) [09:01:37] I made a note somewhere to maybe refactor the hiera profile::kubernetes::infra_pod into image_name and image_tag and set proper defaults to the latter per k8s version [09:05:03] makes sense yes, so I'll push the new tag.. is "old" ok? [09:05:10] I don't really know the current version :D [09:05:36] that's what I assumed :D maybe something more speaking like "used_by_k8s_116" ? [09:06:09] lol that seems really verbose [09:06:38] "k8s_116" ? [09:06:53] ack [09:14:19] qq to confirm - do I also have to docker push after the tag? [09:14:26] (never done it and better safe than sorry) [09:14:41] yes. docker tag is a local thing [09:15:07] I imagined yes, so `docker push docker-registry.discovery.wmnet/pause:k8s_116` is enough? [09:15:21] yes [09:15:29] super thanks [09:16:07] yep worked :) [09:18:49] great. I've not added a phab task for the infra_pod refactor thing. Feel free to do so that helps :) [09:20:23] 10serviceops, 10Machine-Learning-Team: Fix calico, cfssl-issuer and knative-serving Helm dependencies - https://phabricator.wikimedia.org/T303279 (10JMeybohm) p:05Triage→03Low [09:22:31] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [09:22:48] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [09:23:35] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: Metrics changes with Kubernetes v1.23 - https://phabricator.wikimedia.org/T322919 (10JMeybohm) p:05Triage→03Medium [09:24:00] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Add kubernetes 1.17+ topology annotations - https://phabricator.wikimedia.org/T270191 (10JMeybohm) p:05Triage→03Medium [09:25:12] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [09:26:10] 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Write a cookbook to set a k8s cluster in maintenance mode - https://phabricator.wikimedia.org/T277677 (10JMeybohm) [09:26:19] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [10:21:22] 10serviceops, 10MW-on-K8s, 10Patch-For-Review, 10Release-Engineering-Team (Seen): Improve performance of deployment to mw on k8s - https://phabricator.wikimedia.org/T323349 (10JMeybohm) Please make sure not to pre-pull images on tainted nodes (which are masters and kask/sessionstore currently). [10:33:58] 10serviceops, 10SRE, 10Traffic, 10Patch-For-Review: _etcd-client SRV record missing for conftool cluster - https://phabricator.wikimedia.org/T320397 (10Vgutierrez) ping? [10:47:45] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Stop spamming SAL with helmfile on scap deployments - https://phabricator.wikimedia.org/T323296 (10Clement_Goubert) Merge request on `scap` to pass the `SUPPRESS_SAL` variable https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/26 [11:39:14] 10serviceops, 10Maps, 10Product Infrastructure Roadmap, 10Product-Infrastructure-Team-Backlog, and 6 others: Configure replication slots on Postgres masters - https://phabricator.wikimedia.org/T290149 (10jijiki) [11:40:15] 10serviceops, 10Maps, 10Product Infrastructure Roadmap, 10Product-Infrastructure-Team-Backlog, and 6 others: Configure replication slots on Postgres masters - https://phabricator.wikimedia.org/T290149 (10jijiki) All done! [11:41:25] 10serviceops, 10Maps, 10Product Infrastructure Roadmap, 10Product-Infrastructure-Team-Backlog, and 6 others: Configure replication slots on Postgres masters - https://phabricator.wikimedia.org/T290149 (10jijiki) 05Open→03Resolved a:03jijiki [11:52:59] 10serviceops, 10Maps, 10Product Infrastructure Roadmap, 10Product-Infrastructure-Team-Backlog, and 6 others: Configure replication slots on Postgres masters - https://phabricator.wikimedia.org/T290149 (10jijiki) [12:00:23] 10serviceops, 10WMDE-GeoInfo-FocusArea, 10Maps (Maps-data), 10WMDE-TechWish-Sprint-2022-10-26, 10WMDE-TechWish-Sprint-2022-11-09: Resync stale maps postgres replicas - https://phabricator.wikimedia.org/T321885 (10jijiki) 05Open→03Resolved a:03jijiki @hnowlan synced all replicas, I believe we can ma... [12:40:00] 10serviceops, 10Discovery-Search, 10SRE, 10serviceops-collab, and 2 others: Sunset search.wikimedia.org service - https://phabricator.wikimedia.org/T316296 (10Clement_Goubert) apple-search removed from DNS, LVS, service::catalog and conftool. Starting removal from wikikube and deployment-charts. [12:57:49] 10serviceops, 10SRE, 10Patch-For-Review: Deploy MediaWiki config change to use OpenSSL for PBKDF2 password hashing - https://phabricator.wikimedia.org/T320929 (10akosiaris) I am not sure what type of coordination is needed from #SRE either. Maybe just making sure that 1 or 2 SREs are around when the patch is... [13:45:00] 10serviceops, 10CampaignEvents, 10Wikimedia-Site-requests, 10Campaign-Registration, and 2 others: Run the timezone update script periodically in prod and in beta - https://phabricator.wikimedia.org/T320403 (10Clement_Goubert) @Daimona I can deploy it on prod whenever, but deploying it on beta will take som... [14:03:51] 10serviceops, 10WMDE-GeoInfo-FocusArea, 10Maps (Maps-data), 10Patch-For-Review, and 2 others: Look into the replica sync fails - https://phabricator.wikimedia.org/T321722 (10jijiki) [14:04:04] 10serviceops, 10CampaignEvents, 10Wikimedia-Site-requests, 10Campaign-Registration, and 2 others: Run the timezone update script periodically in prod and in beta - https://phabricator.wikimedia.org/T320403 (10Daimona) >>! In T320403#8405586, @Clement_Goubert wrote: > @Daimona I can deploy it on prod whenev... [14:04:22] 10serviceops, 10WMDE-GeoInfo-FocusArea, 10Maps (Maps-data), 10Patch-For-Review, and 2 others: Look into the replica sync fails - https://phabricator.wikimedia.org/T321722 (10jijiki) [14:58:07] 10serviceops, 10Application Security Reviews, 10secscrum, 10serviceops-collab, and 2 others: Security Readiness Reviews of Trusted GitLab Runners - https://phabricator.wikimedia.org/T304514 (10sbassett) [14:58:14] 10serviceops, 10Application Security Reviews, 10secscrum, 10serviceops-collab, and 2 others: Security Readiness Reviews of Trusted GitLab Runners - https://phabricator.wikimedia.org/T304514 (10sbassett) [15:30:47] 10serviceops, 10CampaignEvents, 10Wikimedia-Site-requests, 10Campaign-Registration, and 2 others: Run the timezone update script periodically in prod and in beta - https://phabricator.wikimedia.org/T320403 (10Clement_Goubert) >>! In T320403#8405639, @Daimona wrote: > - We're planning to deploy the extensio... [16:02:22] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Allow to address Kubernets API servers from NetworkPolicy - https://phabricator.wikimedia.org/T287491 (10JMeybohm) [16:03:36] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Allow to address Kubernets API servers from NetworkPolicy - https://phabricator.wikimedia.org/T287491 (10JMeybohm) [16:03:41] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [16:03:50] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Allow to address Kubernets API servers from NetworkPolicy - https://phabricator.wikimedia.org/T287491 (10JMeybohm) p:05Medium→03Low [16:54:43] 10serviceops, 10Discovery-Search, 10SRE, 10serviceops-collab, and 2 others: Sunset search.wikimedia.org service - https://phabricator.wikimedia.org/T316296 (10Clement_Goubert) 05In progress→03Resolved Certificates cleaned up. It's dead, Jim. [17:07:37] I'm off, have a nice week end o/