[00:54:40] Not sure anyone is still around at this point, but I missed this request https://phabricator.wikimedia.org/T433839, came in like 10hrs ago or something [00:55:44] Seem to be requesting some “hefty” resources too, not sure what they need that much RAM and 4 instances for. Asking [09:02:53] I forgot, morning! [09:03:03] (the heat is killing me) [11:29:26] * dcaro lunch [13:03:13] dcaro: can you tell me about adding a new image or helm chart to harbor? e.g. should I use personal credentials or are there shared admin credentials someplace? And Is there a guide someplace for this? (The docs I'm reading just say 'once a new component is added...' which is not a lot to go on.) [13:24:25] andrewbogott: hey sorry, I was in a meeting [13:24:35] np [13:24:55] andrewbogott: I can help you create an account, or you can find the credentials in the harbor config file in the VM hosting it [13:25:11] what do you want to push there? [13:25:26] (our CI has access to push stuff too) [13:27:35] for now just an updated chart for cert-manager [13:27:51] CI should be able to take care of that no? [13:27:59] wait, not sure xd [13:28:04] did we finally start mirroring charts to harbor? [13:28:09] CI only pushes things that we actually build ourselves right? [13:28:51] taavi: I don't think so, andrewbogott: yep, most of the time (things like buildpack-runner/builder images etc. are pushed manually) [13:29:09] I may misunderstand how this works entirely. Where I'm at is testing https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1355 with lima-kilo... [13:29:27] it says 'Release was not present in Helm.' and I immediately jumped to the conclustion that I need to import the new version into harbor [13:29:32] does that sound wrong? [13:29:46] andrewbogott: I think it should be pulling it from upstream [13:29:49] can you show the full error? [13:29:54] sure [13:30:26] https://www.irccloud.com/pastebin/J3UAVHzd/ [13:31:11] I suppose it's equally likely that I just have a typo in my patch [13:31:25] so, first of all, that's logs-api, not cert-manager [13:32:01] weird, it's as if cert-manager did not deploy the crds or something [13:32:16] maybe the newer version requires us changing the way we declare them [13:32:19] but "Release was not present in Helm." here means that logs-api is not already installed to the cluster (and here it means it can't show a diff to the previous version because there is no previous version) [13:32:39] "Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: "logs-api-certificate" namespace: "" from "": no matches for kind "Certificate" in version "cert-manager.io/v1"" is the actual error why that chart is failing to install [13:33:11] which comes to the same question that dcaro asked.. some lines up in the log it should show an item for installing cert-manager.. did that succeed? [13:33:36] * andrewbogott scrolls back a million lines... [13:33:58] xd, ansible/the setup can be verbose [13:34:20] https://www.irccloud.com/pastebin/GzaYwhXY/ [13:34:26] that certainly looks like something [13:35:36] hmm... that is set for helmfile no? [13:35:39] roles/k8s/tasks/toolforge-deploy_components.yaml: {% if item.wait | default(false) %}--wait --set helmDefaults.timeout=600{% endif %} [13:35:40] so lesson #1 here is: the good stuff in the lima-kilo error log is at the top not the bottom :) [13:36:22] here's a bigger snip just in case [13:36:26] https://www.irccloud.com/pastebin/rWyyQZcL/ [13:39:09] yeah, the lima-kilo setting dcaro found seems to be at fault here?? [13:39:24] I rebuilt my lima-kilo on monday and it worked :/ [13:39:39] did something change in the chart making helmDefaults no longer be a valid setting? [13:39:58] that would be my guess. [13:40:02] what versions of helm and helmfile are there installed? [13:40:27] (the vm should be up, just run `lima` or `limactl shell lima-kilo` or similar) [13:42:12] version.BuildInfo{Version:"v3.17.2", GitCommit:"cc0bbbd6d6276b83880042c1ecb34087e84d41eb", GitTreeState:"clean", GoVersion:"go1.23.7"} [13:42:26] helmfile 0.171.0 [13:42:32] same yep [13:42:40] Sounds like my first step should be to see if my setup works without my patch [13:42:52] then we'd know to blame the cert-manager chart change [13:43:11] although... I'm not sure what we do /then/ other than remove that timeout setting everywhere [13:43:31] let me try [13:43:45] also: rewinding to my original question; you'd expect us to deploy this chart from upstream in production as well? [13:44:15] I can reproduce with your patch, without it it works [13:44:28] yep, currently it's what we do [13:44:53] we started discussing if we wanted to mirror them in harbor, but did not decide anything specific afair [13:45:29] ok. That doesn't freak me out, just wondering. [13:46:09] a quick fix is removing the `wait: true` in the ansible task [13:46:21] (in the values file actually, in lima-kilo) [13:46:50] that will not pass that option, but it was there so helm would wait for everything to be up before continuing, as cert-manager is needed to be up and running for the rest to work [13:46:56] (and helm did not wait by default) [13:47:21] What if we remove --set helmDefaults.timeout=600 [13:47:35] does that mean things hang forever and we never get error logs? [13:48:16] that's what the `wait: true` is for [13:48:21] (if set, it adds the option) [13:48:38] ok [13:48:46] it means that helm might decide that cert-manager took too long to start and fail [13:49:00] oh, that setting /increases/ the timeout [13:49:16] yep [13:49:38] It's needed for systems that are slow (pulling the image is slow, etc.) [13:49:55] ok. which task am I changing? Or is there only one ansible task? [13:50:23] https://www.irccloud.com/pastebin/e8VFtkjr/ [13:50:49] `Deploy toolforge-deploy components into local k8s` this one (the paste was too much xd) [13:51:11] maybe there's a better way to pass that option [13:52:01] https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/blob/9a166829f04340393668b4792c0bfbcc5f66eb95/components/kyverno/helmfile.yaml has it in the helmfile itself [13:53:30] sorry if I'm being overly literal -- earlier you talked about marking out wait: true but that does not appear in toolforge-deploy_components.yaml [13:55:35] yes sorry, that's here [13:55:47] roles/k8s/defaults/main.yaml [13:56:08] https://www.irccloud.com/pastebin/nvPnOhL4/ [13:56:21] oh it's per-component, nice [13:56:36] want me to just add that to my patchset, or are you still thinking about alternatives? [13:57:33] oh wait, that doesn't make sense, different repos [13:57:44] I think we can add it to your patchset, in the helmfile, would be goot to add to the rest that have wait though, maybe create a task to do for all/ [13:57:44] ? [13:58:23] as in, in your patch, odd something here https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1355/diffs#6a9dbd30b4baf394e870b57559d6814d334e448a_29_29 like https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/blob/9a166829f04340393668b4792c0bfbcc5f66eb95/components/kyverno/helmfile.yaml with a timeout of 600 [13:58:33] then you can remove the `wait: true` from lima-kilo [13:59:06] (for the `cert-manager` only), but we can do the same also for the rest of components that have the `wait: true` so we can remove then that option from ansible completely [14:00:40] ok! will test first, then make the cert-manager specific change if it works... [14:00:41] thank you! [14:00:48] 👍 [14:01:00] * andrewbogott assumes it will now take 30-60 minutes to test because... lima-kilo [14:04:42] inside lima-kilo, you can `cd ~/toolforge-deploy`, `git fetch ...`, `git reset --hard FETCH_HEAD`, `toolforge_run_ansible.sh -t toolforge` [14:04:56] that runs only the toolforge part of ansible [14:05:24] the list of all the tags you can use are here [14:05:26] https://www.irccloud.com/pastebin/tNAwRd2D/ [14:06:19] oh, wait, it's actually the `k8s` tag, not `toolforge` [14:19:29] ok, here's an unrelated thing that I run into semi-constantly with gitlab: [14:19:37] https://www.irccloud.com/pastebin/7XPDi3rM/ [14:20:49] ah, wait, nevermind I see what's wrong I think [14:44:28] Does "TASK [ldap_users : Inject tool accounts into foxtrot-ldap]" take forever or does hanging there mean setup is finished? [14:47:32] that might mean that ldap is not starting up correctly inside the VM [14:47:55] ok. Trying again... [14:47:59] I have seen this work! [14:48:43] try running `toolforge_run_ansible.sh --tags ldap` instead, that will force to rebuild the ldap container [15:03:44] bah, "WARNING: top-level config key environments must be defined before releases in helmfile.yaml" [15:03:55] I think that's ok [15:04:10] (unless it breaks something xd) [15:04:53] this always produces 10,000 lines of red output and that doesn't necessarily mean anything is broken? [15:05:26] most are T429343 [15:05:27] T429343: [lima-kilo] ansible deprecation errors - https://phabricator.wikimedia.org/T429343 [15:09:30] https://www.irccloud.com/pastebin/VuhQNi8M/ [15:09:43] I assume that means the timeout I set in toolforge-deploy is not working... [15:18:38] hm, no, again I needed to look at the top [15:18:41] https://www.irccloud.com/pastebin/76eU1sZa/ [15:18:47] dcaro: anything obvious in ^ ? [15:19:47] ` * timed out waiting for the condition` [15:20:00] that's helm waiting for cert-manager to come online [15:20:06] andrewbogott: can you run `kubectl get pod -n cert-manager`? [15:20:31] ImagePullBackOff [15:20:45] So that means I'm trying to install something that doesn't exist; that should be easy to sort out [15:20:45] and in describe? [15:23:16] at least one of them says [15:23:17] Failed to pull image "docker-registry.svc.toolforge.org/cert-manager/cainjector:v1.20.3": failed to pull and unpack image "docker-registry.svc.toolforge.org/cert-manager/cainjector:v1.20.3": failed to resolve reference "docker-registry.svc.toolforge.org/cert-manager/cainjector:v1.20.3": unexpected status from HEAD request to [15:23:17] http://harbor.local/v2/cert-manager/cainjector/manifests/v1.20.3?ns=docker-registry.svc.toolforge.org: 401 Unauthorized [15:24:38] same for cert-manager/controller [15:24:48] and cert-manager/startupapicheck [15:24:51] we might have to upload them https://docker-registry.toolforge.org/#!/taglist/cert-manager/cainjector [15:25:01] the images need mirroring to the toolforge docker registry if you haven't done that already [15:25:05] I see we have no cookbook for it yet [15:25:07] https://www.irccloud.com/pastebin/xuAnvUB3/ [15:25:51] ok, so, just so I'm following... we're happy getting charts from upstream but not images, correct? [15:26:07] 'happy' is a strong word [15:26:16] tolerating :) [15:26:28] we haven't gotten around to doing something about charts yet [15:27:00] ok! I will make a cookbook [15:27:15] I would recommend copy-pasting the istio cookbook as a start [15:27:20] great [15:27:23] we started with the images before harbor, so docker-registry did not allow charts, now with harbor we can do charts too [15:27:32] +1 for copy-pasting, should be similar [15:27:45] there's also some chaos on where to put it xd [15:27:52] does harbor not support image hosting? [15:28:06] yes it does yep, now we can host everything [15:28:17] ok, so in the future it'll be just harbor [15:28:34] except then you'll have chicken-and-egg problems if/when we move harbor to run in k8s [15:28:39] maybe xd, we need to solve that yep [15:29:12] not that there's no options, just not super clear better one [15:29:23] (and no urgent need) [15:30:58] tbh I am pretty happy with the current two registries option (with maybe pre-built images moving to harbor), the 'traditional' registry is low maintenance and doesn't really need to be that highly available since infrastructure components aren't restarting very often or running cron jobs like tools, and then harbor can use that for its dependencies [15:30:58] and run in k8s and be highly available for actual tools that change and restart much more often [15:31:01] k8s is nonstop chicken+egg, that's why our magnum setup starts with puppet deploying k3s :( [15:32:50] the current setup + moving pre-built to harbor is one solution yep [15:33:20] anyway, my point being that i would not move the cert-manager images off of docker-registry at this point [15:33:30] though I'd push for either mirroring everything (then charts will need a home), or not mirroring anything (mirroring only half of the stuff gets the worst of both) [15:34:16] we can try to use a 'caching' layer sort of thing, to avoid hitting upstream for everything (and maybe speeding up some stuff + a bit more reliability) [15:38:22] New Toolforge request with reason “Interviewing as a new SRE” https://toolsadmin.wikimedia.org/tools/membership/status/2332 [15:39:29] First time seeing this. Can’t just approve, without some “verification” (else anyone can just become “new SRE”). But also no idea how to do said verification [15:39:32] Any ideas? [15:41:22] another very basic question: I imagine that the helm chart points to the default place to download an image... is that right? And if so, where is it in http://github.com/cert-manager/cert-manager/releases/download/v1.20.3/cert-manager.yaml ? [15:41:43] Raymond_Ndibe: ask them for their contact at the WMF and then confirm that it's legit with the staff contact? [15:42:28] Ok great that should work too [15:42:30] Raymond_Ndibe: see -private [15:44:21] andrewbogott: `curl -O /dev/null -L "https://github.com/cert-manager/cert-manager/releases/download/v1.20.3/cert-manager.yam [15:44:21] l" | grep "image:"` [15:44:28] bah, `curl -O /dev/null -L "https://github.com/cert-manager/cert-manager/releases/download/v1.20.3/cert-manager.yaml" | grep "image:"` [15:45:08] or look at the default values at https://artifacthub.io/packages/helm/cert-manager/cert-manager/1.20.3?modal=values to find it, although usually helm charts spread the url over a couple of different keys so that might get difficult [15:49:15] * andrewbogott wonders why searching that chart for 'image' didn't work at first, but now that taavi has suggested it it works great [15:55:17] taavi, what does it mean that the helm chart asks for image: "quay.io/jetstack/cert-manager-cainjector:v1.20.3" but my lima-kilo error was 'Failed to pull image "docker-registry.svc.toolforge.org/cert-manager/cainjector:v1.20.3"' ? I get the sub from quay to docker-registry but where is the image name change coming from? [15:57:06] the chart asks for `image: "{{ template "cert-manager.image" (tuple .Values.cainjector.image .Values.imageRegistry .Values.imageNamespace (printf ":%s" .Chart.AppVersion)) }}"` [15:57:27] the file you were looking at earlier is the chart rendered with the default values, which happens to be that quay.io/... url [15:57:54] our deployment overrides those in https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/blob/main/components/cert-manager/chart-values/cert-manager.yaml?ref_type=heads instead [15:59:56] ok, so maybe I can just... [15:59:58] pull_url=f"quay.io/jetstack/certmanager-{image}:{self.certmanager_version}", [15:59:58] push_url=f"{self.image_repo_url}/certmanager/{image}:{self.certmanager_version}", [16:00:09] oops cert-manager [16:00:46] double check whether you need a `v` before the version, but something like that [16:20:57] pylint: invalid-name / Module name "copy_certmanager_images_to_registry" doesn't conform to '[a-z0-9_][a-z0-9_-]{2,30}$' pattern ('[a-z0-9_][a-z0-9_-]{2,30}$' pattern) what am I missing? Looks like a match to me [16:21:13] Raymond_Ndibe: i can help figure out who she might be interviewing with, pinged the 2 hiring managers in SRE I know are reviewing candidates- [16:21:23] andrewbogott: i think it thinks it's too long [16:22:13] oh, is the '30' a character limit, not a number of repetitions? [16:23:00] aren't those the same thing here? [16:23:29] can that easily be suppressed? [16:23:29] oh you're right. For some reason I was thinking repetition of _ separated blocks but of course that's not what it says [16:45:45] thilp: it seems that loki still uses the metadata as labels if they already exist :/ [16:46:32] sending this: `Sending log data to loki: {'streams': [{'stream': {'source': }, 'values': [['1785861889129055000', 'silly message', {'job_name': 'myjob', 'user_name': ''}]]}]}`, that should use the metadata job_name, ends up in the labels [16:46:52] https://www.irccloud.com/pastebin/2gXaBa9W/ [16:47:08] so the migration might need something more? [16:54:19] Ohhh, I think that loki just presents them as labels, even though they are not, xd, so we might not even need to change the models to pars the new fields from the metadata [16:54:36] anyhow, /me off [16:54:39] cya tomorrow! [20:09:04] fyi, I'm rebooting ceph OSDs because it's an easy way to force them to pick up some recent config changes. As always, a few health alerts may escape the cookbook's silence attempts.