[11:02:47] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [12:48:26] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-api-int helmfile deployment - https://phabricator.wikimedia.org/T321895 (10Clement_Goubert) p:05Triage→03High [12:48:45] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-jobrunner helmfile deployment - https://phabricator.wikimedia.org/T321897 (10Clement_Goubert) p:05Triage→03High [12:48:58] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-web helmfile deployment - https://phabricator.wikimedia.org/T321900 (10Clement_Goubert) p:05Triage→03High [12:49:13] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-web helmfile deployment - https://phabricator.wikimedia.org/T321900 (10Clement_Goubert) 05Open→03In progress [12:49:25] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [12:50:13] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-api-int helmfile deployment - https://phabricator.wikimedia.org/T321895 (10Clement_Goubert) 05Open→03In progress [12:50:25] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [12:50:35] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-api-ext helmfile deployment - https://phabricator.wikimedia.org/T321896 (10Clement_Goubert) 05Open→03In progress [12:50:47] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [12:51:01] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-jobrunner helmfile deployment - https://phabricator.wikimedia.org/T321897 (10Clement_Goubert) 05Open→03In progress [12:51:13] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [13:40:40] Hello. Are there any further objections to my merging this spark and spark-operator image patch? https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/850244 Thanks. [14:46:37] I've +1ed [14:54:01] Many thanks. [15:29:05] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, and 2 others: Migrate thumbor to Kubernetes - https://phabricator.wikimedia.org/T233196 (10hnowlan) [15:56:23] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Import istio 1.1x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T322193 (10elukey) [16:01:16] <_joe_> team meeting! [16:01:26] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10ARM support: Adoption of aarch64 (aka arm64) in WMF production? (SRE Summit 2022 Session) - https://phabricator.wikimedia.org/T320811 (10jijiki) [16:47:53] claime: is there a phab task for the service removal thing or is it the gerrit reviews? [16:48:07] Yes there is as part of the mwdebug cleanup [16:48:09] Gimme a sec [16:48:18] no rush :) [16:48:21] https://phabricator.wikimedia.org/T321201#8366250 [16:48:28] 10serviceops, 10Growth-Team, 10Notifications, 10SRE, 10Wikimedia-production-error: Failed to fetch API response from {wiki}. Error code {code} - https://phabricator.wikimedia.org/T321409 (10Joe) 05Open→03Declined Hi @Sgs there is no public incident report because we usually don't do that for incident... [16:48:50] thanks [16:48:53] I've already started removing mwdebug completely following these steps btw [16:50:21] 10serviceops, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Priority Backlog 📥): WMF container registry does not accept a manifest list (aka OCI manifest index, or "fat" manifest) - https://phabricator.wikimedia.org/T322453 (10JMeybohm) I think this might require an update of our docker-registry... [16:53:15] 10serviceops, 10observability: Monitor high load on etcd/conf* hosts to prevent incidents of software requiring config reload too often - https://phabricator.wikimedia.org/T322400 (10Joe) I would suggest that the alert should be on a request per second threshold, rather than on high server load. I say this as... [16:53:16] nice! Did the "helmfile apply" actually remove the namespace or did you had to run "helmfile sync"? [16:54:12] <_joe_> effie: T322400 is actually claimed by someone not in serviceops, should we add an herald rule that moves tasks tagged serviceops but claimed by someone not in the team to -radar? [16:55:21] that sounds like a good idea actually [16:55:50] though I think it makes sense to have it in incomming [16:55:56] so it will be "seen" [16:56:02] and then move it elsewhere [17:22:47] 10serviceops, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Priority Backlog 📥): WMF container registry does not accept a manifest list (aka OCI manifest index, or "fat" manifest) - https://phabricator.wikimedia.org/T322453 (10dduvall) >>! In T322453#8375676, @JMeybohm wrote: > I think this might... [17:32:41] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Deploy new mw-debug service - https://phabricator.wikimedia.org/T321201 (10Clement_Goubert) Removing the private resources first seem to stop the correct cleanup of the puppet resources. For mwdebug, only the helmfile for staging got removed, it no-ops on codfw... [17:35:26] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Import istio 1.1x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T322193 (10elukey) ` == Step 0: scanning /srv/images/production-images/istio == Will build the following images: * docker-registry.discovery.wmnet/istio/build:1.15.... [17:35:41] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Import istio 1.1x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T322193 (10elukey) [19:45:13] 10serviceops, 10RESTBase, 10Wikipedia-iOS-App-Backlog, 10iOS-app-feature-Performance, and 2 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365 (10LGoto) [20:10:03] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹️): move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) [22:25:35] 10serviceops, 10serviceops-collab, 10Release-Engineering-Team (Radar): give releng access to logs to debug buildkit-to-wmf-registry publishing - https://phabricator.wikimedia.org/T322579 (10Dzahn)