[07:38:03] hey folks, I'd need a brainbounce on something I am working on [07:38:10] <_joe_> yes [07:38:28] <_joe_> brain bounces are known to be bad for you though, you know they cause CTE [07:38:49] it is already too late for that [07:38:57] anywayy [07:39:40] the I/F team this week is trying to add support for Kubernetes to Debmonitor, and we are at the step of scanning the k8s clusters to retrive all the image:tag being used by containers [07:40:31] <_joe_> elukey: have you looked at the code created for the original attempt at doing so? [07:40:35] <_joe_> it should still work [07:40:38] in some cases, like Ceph and Knative (ML), we force image tag -> sha256 digest resolution (knative sets it by default, for ceph we explicitly configure images with the sha256 digest) [07:40:58] <_joe_> elukey: wait, what are you looking at to fetch the info? [07:41:14] <_joe_> I would go with downloading manifests of deployments [07:41:19] the kube api, image running by containers [07:41:32] <_joe_> yeah I was asking what were you querying [07:41:50] I am using python3-kubernetes, it has a specific api for that [07:42:28] not sure what it calls behind the scenes, but my understanding is that Knative sets the sha256 digest when it creates a new "revision" [07:42:40] <_joe_> something like [07:42:43] <_joe_> kubectl get deployment -o yaml | grep docker-registry [07:42:48] <_joe_> but in python [07:43:00] <_joe_> is what I'd do [07:43:11] <_joe_> this way, you see the actual images being used [07:44:07] root@deploy1003:~# kubectl get deployment arwiki-goodfaith-predictor-default-00028-deployment -o yaml -n revscoring-editquality-goodfaith | grep image image: docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services-revscoring@sha256:786894c325a4da4b8cb19db1de1c03f51ee28d8839edd9a635c7d1ec6e1123a2 [07:44:31] this is the issue, I can see --^ as well from my call [07:44:41] <_joe_> sigh [07:44:41] but going from sha256 -> tag is not super easy [07:44:45] <_joe_> yes [07:44:52] so I wanted to check why we use it [07:45:04] <_joe_> we use what, tags or sha256? [07:45:12] the latter [07:45:17] <_joe_> tbh we have always said we'd use tags [07:45:27] <_joe_> so no idea, I'd ask the folks in ML [07:45:34] for ceph it is set in deployment-charts, so maybe it is just something to fix in there [07:45:39] for knative, there is https://knative.dev/docs/serving/tag-resolution/ [07:45:49] <_joe_> yeah for ceph I'd just do that [07:46:08] there is a link explaining why it is set by default.. and IIUC they did it to prevent the side effects of mutable tags in the registries [07:46:23] something that we don't really use, so it should be safe to disable [07:46:40] (there is a presentation called "Why we resolve tags in Knative") [07:47:05] I recall that I've read it at the time (2/3y ago) and thought about changing it, but never really did it [07:47:36] <_joe_> I agree [07:47:56] <_joe_> it's a good idea to use sha256 if people mutate tags behind the obvious (latest) [07:48:25] exactly same thought, but we never did it and we'll never do it (at least afaik) [07:49:28] also didn't answer the "have you looked at the code created for the original attempt at doing so? [07:50:21] I am planning to check image catalog, if this is the question, but the main idea at the moment (for the hackathon) is just to extend docker-report or have a quick script that polls the kube apis and sends what it finds (images broken down by namespaces) to debmonitor [07:50:28] as we discussed in Atlanta [07:50:52] if in the future image catalog will be completed etc.., it will start updating debmonitor as well [07:51:09] this is the idea, so we don't block on image-catalog for the moment [07:54:37] <_joe_> yeah [07:54:50] <_joe_> yeah I think image catalog alreayd has that code, I am saying [07:54:51] <_joe_> I was saying that image-catalog should already have the code to do what you want [07:55:08] <_joe_> sigh, nodejs sometimes disappearing my messages [07:55:30] oook yes yes I'll look for it, thanks :) [07:55:39] also it has already all the kubeconfigs that I'd need [09:35:05] to keep archives happy - created the chain https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1163711 [09:40:08] ultimately I'd say that using/storing the sha256 is the better approach tbh - given a tag can definitely change and the hash can't [09:41:22] would it be an option to have docker-report store the digest in debmonitor (together with the tag) and resolve tag to digest when you ingest the cluster state? [09:41:53] but we don't use digest except for some corner cases, and we trust the rest :) [09:42:07] either we use tag@sha256 everywhere or not in my opinion [09:42:14] a mixture makes only tooling complicated [09:42:38] and we don't have rules to allow a mutable tag, at least not that I am aware of [09:43:14] so to mutate a tag one has to deliberately do it for some reason [09:53:43] yeah, that's correct. Just saying that it might be easier to work with the digest right away as it's the actual unique identifier of the image running. If we want to be able to reliably track whats running, even if someone deliberately changes a tag [09:53:57] Sorry to be late for the brainbounce. [09:55:57] The use of sha256 was something that b.rouberol introduced fairly recently. It was, as you say, in order to guard against the possibility of attack by someone pushing a different image with the same tag, leading to malicious code running. [09:56:42] If I understand correctly, this was an attack vector that he saw used in the wild at $former_employer so was keen to introduce the mitigation technique here. [09:57:00] He's off today, by the way. [09:57:58] okok, thanks for the context, it makes more sense.. Even if I'd be curious to understand his experience at $former_employer, if the docker registry was self managed or not [09:58:22] Here is the original ticket: https://phabricator.wikimedia.org/T373000 [09:58:31] in our case, if an attacker gains access to the registry and they are able to modify tags, we are in bad shape [10:00:56] True, but belt and braces and all that. They could get access to a protected branch in gitlab, but if we're deploying images based on the sha256, they wouldn't be able to force an image to be deployed to k8s. [10:02:13] true true, but even if they modify the tag it doesn't mean that a new deployment is triggered straight away [10:02:29] I am not saying it is not important, trying to reason out loud [10:04:03] the image tag would be cached on the k8s nodes, only new ones (or workers without the image in the cache) would be forced to pull the new image [10:04:04] With airflow on k8s we have a lot of dynamic deployments, so it's not just helmfile creating deployments, these days. Also there is mw-cron. [10:04:33] yeah but let's say I flip a tag on the registry, pointing to a malicious image [10:04:49] (that I was able to push to the registry, so we are talking about a mass compromise) [10:04:53] I am also just thinking out loud and not trying to be combative. [10:04:58] yes yes :) [10:05:58] the image would be pulled only if not present on a k8s node, so even if we have all sorts of deployments, we wouldn't see the malicious one getting up anywhere [10:06:08] unless we are doing a big reimage etc.. of course [10:07:00] They might only need one. [10:07:44] we also have dragonfly, so hitting the registry is less frequent for some images [10:08:02] We don't have dragonfly on dse-k8s. [10:08:07] anyway, yes there is a remote chance that it may happen, still a bit hesitant to be worried about it [10:08:10] true right [10:08:26] ok so I'll try to think about a workaround [10:09:15] the main problem from the debmonitor pov is that it will see a report for images like docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services-revscoring@sha256:786894c325a4da4b8cb19db1de1c03f51ee28d8839edd9a635c7d1ec6e1123a2 [10:09:23] (ML has the same issue, different configs) [10:09:31] Much appreciated. I'm sure that Balthazar will be happy to talk about his previous experience of the attack, when he is back. [10:09:57] and following up on a sha256 digest is not very actionable [10:10:07] the reverse mapping is difficult too [10:10:23] but we may have to live with it [10:12:20] From our point of view, with Airflow in particular in the mix, there is a lot of dynamism. Users can supply image names and tags as part of the airflow-dags repo, as part of their job spec. They can only pull from our registry, obviously, but it's more diverse than just having everything specified in deployment-charts, now. [10:15:04] a solution could be that debmonitor shows the sha256 instead of the tag, alongside with the image name [10:15:22] that is not as readable as a tag, but it may be ok [10:15:53] the solution that jayme pointed out may also be good, we store the sha256 too in debmonitor [10:16:17] I'll talk with the other folks :) [10:18:04] Thanks, yeah I like the option described by jayme as well, at first glance. You won't be offended if I put a -1 on this patch will you, elukey? https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1163713 - Just while it is discussed more. [10:19:02] definitely, not that I'd merge it without consensus :D [10:22:56] ideally if we were able to set the tags as immutable in our registry we'd be done [10:23:02] but I doubt docker distribution allows it [10:23:58] Cool, thanks. I would say it's even more important for the ceph images than anything else, because these nodeplugin and related pods run as root. [10:24:54] the solution about using docker-report to send the sha256 may not work in all use cases, since it may happen that some images running on k8s are not in debmonitor yet [10:25:12] in that case, if those have the @sha256, we wouldn't get its tag [10:25:24] anyway, it is more complicated now, we'll see [10:27:48] Yep, sorry about that. [13:27:28] jayme or anyone else, is there a ticket I can follow to see about the wikikube K8s update? I'm asking as we had a couple of our apps running in wikikube codfw fail on Monday around 1100UTC and I want to see if the timeline matches up. More details in Slack: https://wikimedia.slack.com/archives/C055QGPTC69/p1750772117571739 [13:29:25] https://phabricator.wikimedia.org/T397148 inflatador [13:30:47] inflatador: depool happened at 1030UTC [13:31:10] it's not in the task we forgot to add a log linking it [13:31:25] I was out Monday so it's entirely possible I missed stuff too ;) [13:31:39] wipe started at 1123UTC [13:32:20] so yeah, timeline wise it seems to fit [13:32:28] we have to wipe and redeploy the cluster for the upgrade [13:32:38] yeah, that matches when we had the issue https://grafana.wikimedia.org/goto/eO2r5EPHR?orgId=1 [13:33:16] (which explains why you saw the application being completely redeployed) [13:34:25] ACK. Not a huge deal but we'll need to be better prepared for the EQIAD updates [13:34:43] if you find out we need to do something extra for the eqiad update, please add it to the parent task [13:34:57] it'll happen after the next dc switchover so you've got some time [13:36:33] If there an email list or something I need to subscribe to LMK, I don't think anyone on Search Platform was aware of this upgrade. Although again it's possibly we just missed it ;) [13:38:46] I don't think we particularly announced it except from the task and adding it to the deployment calendar [13:39:06] I'll make a note to make a wider announcement for the next time [13:41:47] Cool, thanks! I'll add the instructions to the parent task as you requested [13:41:57] Thanks :D [13:47:41] my last state of knowledge was that since the flink-operator updates no manual intervention is required (apart from depooling wdqs) - did that change again [13:52:33] jayme sorry, that's on us if we told ya that ;( . It needs a little bit of human help. I'm working on cleaning up the docs and I'll link 'em in that parent ticket [13:53:39] hmm...might be I misunderstood or misremembered things. Sorry for not communicating the update properly [13:54:23] NP at all, this was a good opportunity for me to realize that me and David are probably the only people who remember doing this from when we first deployed Flink on k8s ;) [15:46:35] for what is worth and regarding the tag vs sha256 etc, "Rosetta stone" is my general approach to things being identified by > 1 identifiers. Put them all in the tool and have the tool provide the proper lookups to users to help them out. The deployment pipeline anyway tags images with up to 3 different "mutable" tags. The sha1 of the last git [15:46:35] commit, the git tag of that commit and the 202X-MM-DD-production-blah string [15:46:59] so there are way more than the sha256 vs 1 single per image identifiers already [15:47:34] In order news, I am deploying https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1161535 to the ml-serve, dse-k8s, aux-k8s [15:48:02] wikikube and staging cluster done already and are ok for quite some time, this is expected to be a noop, but if it backfires, it's on me, let me know ASAP