[07:48:28] Who should I ask about getting software installed on the cumin nodes? It'd be useful to have s3cmd available there, but I figure I should Ask First :) [07:49:19] to be used with swift? [07:50:03] well, also the new apus ceph cluster; more for testing &c. than "production use" [07:50:54] I like it has very little deps :D [07:51:06] no objections from me, but you should ask I/F :D [07:51:34] make CR, post link to their IRC channel? [10:37:14] head's up, Netbox upgrade at 12 UTC, be prepared to no do any Netbox related changes for 1 to 2 hours. [12:03:41] XioNoX: just as an fyi, I was trying to run homer to add the 2 new mw hosts to BGP and I met lvs2011 and lvs2012 being added as neighbors to cr*-codfw [12:03:53] topranks: ^ [12:03:56] I am gonna back out since you got the netbox upgrade right now (nothing urgent on my side) [12:04:02] akosiaris: cool, thx [12:04:43] alright, doing a dump of the Netbox DB, please refrain from any NB change from now on [12:27:30] akosiaris: hmm... I was afk let me have a quick look [12:30:06] seems I messed up when doing some changes yesterday - need to work out exactly how but let me patch it up [12:33:31] XioNoX: doh I made some changes - don't worry I will double check them later and make sure [12:38:29] TL;DR - something went wrong with the puppetdb import script I ran after the lvs maintenance last night - and it changed the untagged vlan on the switch ports in Netbox [15:14:50] how did the netbox migration go btw? [15:15:27] akosiaris: cookbooks are being tested/fixed now, I think [15:15:40] some ongoing work in #-sre-foundations [15:16:09] exactly yes, last ones standing [15:16:36] ah, thanks [15:29:07] we still seem to have quite a few buster hosts [15:33:16] ~90 [15:36:50] and a lot of pods running buster too :) [15:38:21] Netbox has (finally :)) been upgraded ! You can resume using it (directly and through cookbooks). There might still be bugs introduced by this upgrade, so please be careful and report to I/F any issues you might notice! Thanks [15:42:21] gg! [15:52:23] XioNoX: the DNS repo on netbox1003 is behind the one on netbox1002 [15:52:34] by 6 days apparenty [15:55:48] volans: good point, let me run the dns cookbook, it should update it? [15:55:57] no [15:56:00] don't [15:56:11] there is no way to ensure you're not modifying the DNS that way [15:56:18] and you'll loose histopry [15:56:27] first sync from netbox1002 (why was it not synced already???) [15:56:37] then once you have it at the latest run before the migration [15:56:43] you can run the cookbook (was it not tested?) [15:56:45] and ensure it's a noop [15:56:56] and doesn't remove/add any record compared to old netbox [15:57:10] XioNoX: ^^^ [15:57:24] volans: which commands should I run ? https://wikitech.wikimedia.org/wiki/DNS/Netbox [15:58:08] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/cookbooks/sre/dns/netbox.py#114 [15:58:24] volans: lots of moving parts and not all documented in the same area :( [15:59:12] volans: that needs to be run from which hosts? which one is passive now? :) [15:59:40] that needs to run on the new primary fetching from the old primary [15:59:52] (or any of the old host that has the latest SHA) [16:03:32] volans: cool, done [16:04:08] volans: cookbook was run in dry mode, and was a noop [16:04:25] so probably why it didn't get caught [16:05:31] XioNoX: what do you mean? [16:06:29] test runs of the dns cookbook were a noop, so probably why we didn't think of that extra sync step [16:06:46] a test run 5 minutes ago had hundreds of diffs [16:07:05] weird [16:07:19] all good now [19:12:38] If I wanted to see where a docker image hosted at docker-registry.wikimedia.org came from (like its source repo), what's the easiest way to do that? [19:26:11] https://gerrit.wikimedia.org/r/admin/repos/operations/docker-images/production-images,general ? [19:26:27] inflatador: ^ [19:27:59] brett that doesn't seem to cover all of the images...like I picked https://docker-registry.wikimedia.org/wikimedia/operations-software-thumbor-plugins/tags/ at random and had to do some digging before I found its repo [19:28:12] I don't think anything that uses Gitlab is hosted there either [19:28:55] Well that's all I know! Sorry I can't help any more 🙃 [19:29:37] brett NP, I'm dreaming of better image metadata ;) Will get a phab task started [19:37:57] inflatador: dduvall recently built some support for SBOM stuff into blubber. That might be helpful to your cause. [19:42:09] * inflatador starts reading thru https://github.com/moby/buildkit/blob/master/docs/attestations/sbom.md [19:47:50] inflatador: provenance metadata is now supported as well [19:48:03] bd808 thanks for the advice. I started T371549 and I'll probably start playing around with this on some of the images I control [19:48:03] T371549: Publish more metadata tags with docker registry images - https://phabricator.wikimedia.org/T371549 [19:48:05] Judging by your comment that might be more useful here [19:48:56] dduvall is https://github.com/in-toto/attestation the best place to learn more about the provenance standards? I'm new to all this [19:50:17] At lunch but I will link you asap [19:51:56] But yeah, in-toto attestations are used for both sbom and provenance [19:53:42] no rush...my main goal is discoverability. It looks like there are some built-in gitlab vars we could use for the image repos on gitlab https://docs.gitlab.com/ee/ci/variables/predefined_variables.html [20:07:41] inflatador: https://docs.docker.com/build/attestations/slsa-provenance/#provenance-attestation-example shows an example provenance attestation. you can see vcs info (source repo, commit) under the metadata section. There is also information about base images used in both the build processes and the final image [20:09:07] The attestations are stored alongside the image manifest as part of the same manifest index in the repo, so given an image ref, the attestations can be discovered by simply querying the ref and seeing whether it’s a manifest index and contains entries for the attestations [20:10:48] It’s a nice model I think. The metadata is there in the registry if you need it, but it doesn’t bloat the actual image that gets pulled by typical runtime operations [20:11:36] Which is more important when it comes to sboms since they’re quite large [20:12:48] See https://github.com/moby/buildkit/blob/master/docs/attestations/attestation-storage.md [20:14:23] still wrapping my head around this, but it sounds like you could find this attestation by doing a `docker image inspect`? Or do you need `docker buildx imagetools` to find it? [20:15:47] The latter is easier but you can use `docker manifest inspect` to see whether the ref points to a manifest index or not [20:16:10] Honestly I found using curl to be easier due to some weird edge cases [20:17:02] It's good to hear that there is a schema for this at the very leasty [20:19:13] and that blubber has the ability to do this automatically <3 [20:19:47] from there it shouldn't be all that difficult to expose the metadata on the docker-registry web UI [20:53:31] inflatador: that would be awesome! [20:57:30] dduvall yeah. I'm a little worried it's overkill for what I'm looking for, but on the other hand I'd rather use something that's a standard/understood outside of WMF. Do any of your images have the attestations? Just wondering what it "looks like" when blubber applies them [20:58:22] good question. it's so new and i can't recall if i'd actually published an image with attestations yet [20:58:39] kokkuri supports enabling provenance as a CI variable [20:59:06] https://gitlab.wikimedia.org/repos/releng/kokkuri/-/blob/main/includes/images.yaml?ref_type=heads#L141 [21:50:05] lurker +1 for adding: where did this image come from to the registry web page via image maniphests/metadata. That is the question I field most often about particular images and there have been a non-zero number of times where I didn't know the answer :) [22:26:21] inflatador: i went ahead and did a minor version release of blubber and enabled provenance generation. you can see the general discovery process and raw attestation data here https://phabricator.wikimedia.org/P67172 [22:27:48] you can also use `docker buildx imagetools inspect docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.0.1 --format '{{ json .Provenance }}'` but note that imagetools groups the provenance results by arch since blubber is published for both arm64 and amd64