[06:40:58] <_joe_> elukey, hnowlan, jayme, claime, btullis: I'd love to get feedback on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/837495/ and https://phabricator.wikimedia.org/T320782 by monday [07:04:09] ack yes sorry, I have been busy in these days, will try to provide feedback today/tomorrow :) [07:04:39] <_joe_> elukey: thanks [07:04:51] <_joe_> I just don't want to wait too long to merge such changes if we like them [07:05:01] <_joe_> becuase else I'll have to re-do all the work afterwards [07:05:34] <_joe_> and I plan on replacing all tls.* stuff with the new mesh.* everywhere soon, at the very least [07:05:42] <_joe_> if we go this way [07:06:26] <_joe_> then I have more changes in my mind, like creating "profile templates" that we can use when scaffolding or to say "add a memcached sidecar to chart X" [07:06:52] <_joe_> but that's for laters, once we have got used to the new stuff and to write reusable modules as much as possible [07:44:53] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Import coredns 1.8.x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T321159 (10elukey) After a chat with Janis we decided to not use the coredns Debian package anymore, since it is installed only on Docker images. The above patch... [07:45:26] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Import coredns 1.8.x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T321159 (10elukey) [07:48:25] our trainee has not shown up; something must have come up. I'll ping him on the task and we'll reschedule. in the meantime, that's it for the window for today, folks! [07:48:53] !log UTC morning backport and config training window closed [07:49:03] bah wrong channel for both [08:03:48] _joe_: ack [08:30:46] _joe_: ack, will check it out. Thanks. [09:20:30] Slightly silly question: do we have docker images with the gitlab release-cli command available in them? [which one could then use to create releases via .gitlab-ci.yml] [09:23:55] Emporer: afaik not in our registry, we used/allowed docker images from registry.gitlab.com/gitlab-org/**/* on GitLab Runners. But I guess we may want to have out own images at some point if we want to created this releases on Trusted Runners. [09:29:07] Ah, OK, so I could use the gitlab release-cli image if I wanted to try this? Thanks :) [09:58:47] Emperor: On Shared Runners yes, on Trusted Runners that gitlab.com image is not allowed. So we may want to build that and push it to our registry ourselves [10:05:14] jelto: I have a quick question that you might be able to help me answer. Is there any way that we can use docker-in-docker for GitLab-CI pipelines at the moment? [10:06:51] btullis: only if you register and manage your own runner in your project. The provided Runners (Trusted and Shared) don't have dind atm. We started using buildkit for docker builds, but nothing in the direction of dind [10:07:36] OK, thanks. For reference, we're trying to use GitLab-CI to build a conda environment and package it as a deb. https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/ci/editor?branch_name=T317210_airflow_deb_creation [10:09:37] There are two build pipelines: `publish_conda_env_debian_with_docker` which doesn't work because we can't use dind. [10:09:38] So we've had to create a second pipeline as a workaround, which copies all of the steps from the Dockerfile: `publish_conda_env_debian` [10:09:53] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Better management for helm charts - https://phabricator.wikimedia.org/T320782 (10Clement_Goubert) The overall logic makes sense to me. I have one slight issue with the versioning, possibly stemming from a misunderstanding. **The misunderstanding (update... [10:13:11] This is just a bit untidy in the .gitlab-ci.yml file, so it would be nicer if we could use docker. Would deploying our own runner be an acceptable solution, as far as you're concerned? Have any other teams done that yet? What would be our greatest challenges? [10:17:10] btullis: interesting, thanks for letting me know! I'll keep an eye on that project. I agree that a cleaner solution for debian builds would be nice. But that's still a bit in front of us. I've just done some small tests with dpkg-buildpackage. Adavanced debian builds is still a open topic. [10:17:10] Registrering your own runner is totally fine. You can obtain the registration key under Settings > CD/CD > Runners. You have to provision and manage that machine yourself and do some updates from time to time (like once a month). Beside of that it's not too complicated. [10:19:19] There are some other teams doing that too. I hope to consolidate that at some point and reduce the maintenance overhead for other teams. But for now it could be a solution :) [10:19:37] OK, that's great. Maybe I'll re-open this ticket and create a ganeti VM to act as a runner for our group then, if that's OK: https://phabricator.wikimedia.org/T295045 [10:24:10] There were some concerns of Running privileged containers in production and that's also the main blocker which stalled work on dind. I'm not sure if ganeti is an option for that. WMCS would be fine with the downside it's outside of production. But we can discuss this also in the task. [10:24:49] Ack, thanks. [10:28:47] btw there is also a wikimedia-gitlab channel ;) [10:52:17] Ah thanks again. Didn't think of that. I've added some notes to the ticket and reopened it. [12:17:44] jelto: for reference, would you prefer queries there to here? [12:40:48] I'd say most GitLab questions are fine in -gitlab channel. So RelEng also knows what's currently requested/needed. There are also more folks available with GitLab knowledge. If there is a overlap with other serviceops fields like docker images, registry, wikikube here makes more sense. [13:20:07] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Kubernetes services with externalTrafficPolicy: Local don't work - https://phabricator.wikimedia.org/T300500 (10JMeybohm) 05Open→03Resolved Now it has [15:09:34] 10serviceops, 10SRE, 10Wikibase Product Platform, 10Wikimedia-Apache-configuration: Incorrect handling of ETags taking precedence over timestamps in conditional requests - https://phabricator.wikimedia.org/T320241 (10LSobanski) [20:15:07] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹ī¸): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) [21:56:17] hello. Can https://phabricator.wikimedia.org/T320848 be added to the next serviceops meeting agenda? [22:10:52] legoktm: done! not sure if I'll be at the Monday meeting but I added it to the notes [22:14:36] Thanks :)