[09:01:08] <_joe_> hi everyone, I posted what should be a semi-final version of what the new scaffolding system for charts would look like at https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/885281; a description of the system can be found here: https://gitlab.wikimedia.org/repos/sre/sextant/-/blob/scaffold/README.md#create-a-new-chart-from-scaffolding-models [09:01:28] <_joe_> I would welcome comments/suggestions for improvement [09:02:21] <_joe_> the idea is basically that we can have multiple scaffolding models, that will allow us to make creating a chart from scratch simpler and easier to adapt to the needs of various kubernetes clusters [09:15:32] I'll try it with Ilias for the ores-migration endpoint that we are building, and will provide feedback [09:19:14] <_joe_> elukey: my patch still lacks one detail - changing rake to use sextant to do scaffolding [10:57:28] I'd need some help with https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/900240 [10:57:50] I'd basically need to remove a ns deployment on ml-serve-{eqiad,codfw} [10:58:04] do I need to clean up helm releases before proceeding? [11:05:55] <_joe_> you're asking if it's possible to remove a namespace without emptying it first? [11:07:12] I am asking what's the best procedure in this case, never done it [11:07:50] <_joe_> I'm not 100% sure if with helm 3 just deleting everything would be an issue [11:09:19] <_joe_> with helm 2, deleting the NS would delete tiller, so any info about the release would be removed [11:09:47] <_joe_> uhm it's k8s secrets [11:10:29] <_joe_> in the same namespace [11:10:36] <_joe_> so yeah it should wipe everything out [11:19:08] ok perfect, I'll try to see what CI complains about and then proceed [11:23:33] it seems breaking when I remove the release config from helmfile.yaml [13:24:25] I'm not following too much docker upstream, but got this from a friend, thought to paste it here too: [13:24:28] https://blog.alexellis.io/docker-is-deleting-open-source-images/ [13:32:45] <_joe_> volans: stop vindicating our choice of only allowing locally-built images to be used, thanks [13:33:11] :D [14:28:20] _joe_: Thanks for the work on the new scaffolding system. Will check it out. Where would you like any comments? In GitLab or in this channel, or is there a ticket? [14:31:47] Unrelated; I have a question about locally built images, which I thought I would ask here... [14:32:49] Essentially, would it be permissible to extract binary build artifacts from a third-party container and copy them to one of our own base images for use? [14:33:05] Like this: https://gerrit.wikimedia.org/r/c/analytics/datahub/+/900310/2/.pipeline/frontend/blubber.yaml [14:35:46] To-date we have been building them all from source, but it is unwieldy to maintain our own fork of datahub, just so that we can build containers that run on Debian instead of Alpine. [14:36:21] <_joe_> alpine uses MUSL, so I doubt a binary from alpine would work in general on debian [14:36:50] also one thing that I don't get is where acryldata/datahub-frontend-react lives [14:36:54] is it on dockerhub? [14:36:58] <_joe_> I guess so [14:37:20] <_joe_> btullis: we should really avoid relying on dockerhub for any automated procedure at this point I think. [14:37:21] _joe_: But it's a java war file. [14:37:59] elukey: Yes, it's hosted on Dockerhub. https://hub.docker.com/r/acryldata/datahub-frontend-react/tags?page=1&name=v0.10 [14:38:19] btullis: okok, did you see what volans posted above? :D [14:39:31] another question is - what is the current pain point with the war file? Does it need to be created by our build image etc..? (trying to understand, not familiar with datahub) [14:40:56] Yes, I did see the message above. I don't think that datahub is likely to be affected by the changes to the Docker policy on deleting open source applications. [14:41:29] sure sure, but the less we rely on dockerhub the better, also for security reasons [14:41:56] I think that this is the main point of building our own images (and I can related that it is painful at times) [14:43:54] btullis: what is painful about building the war? Very long, breaks often, etc..? [14:44:01] The main pain point at the moment is that currently we maintain a fork of https://github.com/datahub-project/datahub in gerrit. [14:44:36] We keep the pipelinelib files in the same repo, but in a wmf branch. [14:44:53] https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/datahub/+/refs/heads/wmf [14:45:54] When we want to upgrade datahub we have to pull the main branch, rebase our changes against it, then try to work out what changes they have made to their build system and adapt our build system to those changes. [14:46:23] It also takes a long time to build. [14:47:05] The thing is that they don't make any other binary releases apart from the Alpine based containers. No tarballs other than source. [14:48:41] The consequences are 1) that we struggle to be able to implement new features 2) that we have to spend large amounts of time troubleshooting errors in the build process. [14:49:28] Copying the jars and supporting plaintext files out of the alpine containers into Debian would ease a lot of this frustration, I think. [14:52:28] sure I am not trying to go against simplifications, I am just trying to tell you my understanding of why we don't allow Dockerhub :) [14:53:15] we could try to follow up with upstream to ask if they have plans to release jars in their releases [14:53:44] <_joe_> I would maintain that having a script to download such jars to our artifact repository, then download them from there in the image build process would look like a more solid and equally simple system [14:54:13] <_joe_> but yes, if those are only distributed inside a container, *sigh* [14:58:49] <_joe_> to my point above, having those jars in our artifact repo would also help devs when doing their modifications [15:04:09] We're not doing any modifications on it. We want to run it unmodified, but on approved containers. The only modification we've made so far is the logo, which is trivial. If we wish to modify it we will send patches upstream. [15:04:29] Sorry, I realise that this is a corner case and a bit contentious. [15:05:05] we would certainly revert the logo change for an easier build process. [15:16:31] btullis: have you looked at the build artifacts, do the rely on musl? [15:16:57] *they [15:18:07] If you have an example container, I'm happy to investigate [15:29:30] jhathaway: thanks there are two main ones. https://hub.docker.com/r/acryldata/datahub-frontend-react/ and https://hub.docker.com/r/acryldata/datahub-gms - others are ancillary. There are quite a few supporting files in addition to the jars, particularly in the front end. [15:30:05] Currently looking at v0.10.0.6 [15:30:34] thanks [16:02:46] btullis: on initial glance I would assume extracting artifacts from those containers would work fine, since the musl dependency is restricted to the java binary, which you would be replacing with our base java container [16:03:15] unless those jars also use JNI to connect certain C libraries [16:16:54] jhathaway: Thanks. That's my assumption too. I don't think that JNI is in use, but I'll keep looking into it. [16:18:38] btullis: one alternative could also be to have a script, part of your release process, that pulls the binaries from the image and stores them into our archiva repo [16:19:38] still not ideal but it could be a compromise [16:20:07] (so any rebuild would fetch jars from our archiva instances, not from dockerhub, etc..) [16:20:56] we'd need to accept the security risk that if the dockerhub image is compromised we'll add the jars directly into archiva without vetting, but it is acceptable [16:21:03] (especially given how maven works..) [16:22:03] https://www.irccloud.com/pastebin/nvUOCx6y/ [16:22:53] ? [16:23:38] There are 320 files [16:23:43] https://www.irccloud.com/pastebin/CJr6KYwA/ [16:24:15] 313 of them are jars. That's quite a lot of extra stuff in archiva. [16:25:10] Apologies for being terse, I'm doing two things at once. :-) [16:25:37] ah okok it was difficult to understand :) [16:26:24] sure but IIUC their size is not a lot, right? [16:26:33] I mean 300 jars for a release is not that much for archiva [16:26:43] and it would really take nothing to upload them [16:27:20] we could keep archiva clean and delete old releases, it is is some hundreds of MBs we should be fine [16:27:51] and probably shorter than rebuilding etc.., so a win/win [16:27:55] why not store the docker image in our archives, and then extract from the image? [16:28:45] separate jars make probably more sense since they can be retrieved with more generic tools, like say mvn etc.. [16:29:01] ah [16:29:49] if we store the archive it may be difficult to get stuff from it etc.., seems easier to follow standard paths in my opinion [16:47:02] btullis: strangely the C shared libraries in the gms war seem to built against glibc: [16:47:05] $ ldd ./META-INF/native/libnetty_transport_native_epoll_x86_64.so [16:47:08] linux-vdso.so.1 (0x00007ffdf43ed000) [16:47:10] libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb66d5ba000) [16:47:12] libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb66d01f000) [16:47:14] /lib64/ld-linux-x86-64.so.2 (0x00007fb66d5d4000) [16:47:47] so perhaps java is using some fallback code when running on musl? [17:26:58] Sorry to be a pain. I just don't see what benefit this method of using archiva as an intermediary buys us. Anyway, we can talk about it another day.