[07:43:03] btullis: np, it is just a brainbounce, nobody is forcing you to adopt new things that you are not convinced of :) [07:44:34] re: benefit - The way I see it is that you'll end up with a simple script that 1) pulls the docker image locally 2) pushes jars to archiva, bypassing the build workflow entirely. The other benefit is that any other rebuild of the datahub docker images will not need anything from Dockerhub, but at most fetching jars/wars from archiva. [07:45:49] in the solution that you proposed, dockerhub would be used every time you rebuild, so you'll have a different tie to it than with Archiva [07:46:36] and if in the future, for some reason, you'll need to explicitly build jars/wars to apply patches etc.. you'll have a simple way to hook in your build workflow without issues [09:39:05] <_joe_> ok, pondering this the next day, I feel like you gave us conflicting information btullis [09:39:17] <_joe_> so either we rebuild the software from unmodified sources [09:39:39] <_joe_> in that case, we should just be able to make it a CI process [09:40:06] <_joe_> and I frankly would be against extracting binaries from dockerhub images in that case [09:40:28] <_joe_> or, we do modify the upstream stuff and we want to be able to rebuild the image fast [09:40:40] <_joe_> so having these jars ready is useful for development [09:40:58] <_joe_> I'd like to see a proper problem statement in a task. [09:41:36] <_joe_> frankly, and don't take it personally, it's a general tendency I'm seeing across technology, I don't think "it's faster" is a good reason to compromise policies. [09:41:59] <_joe_> "The policy gains us nothing in terms of security" is a good reason OTOH. [09:42:36] <_joe_> when a policy is there for security reasons [09:43:49] <_joe_> now in your case I'm not sure either is necessarily true. I *think* we might find there's a better way to do what we're trying to do with the current gerrit -> deployment pipeline workflow that was set up [09:47:42] Hi, we are currently designing a solution to be able to have spark jobs running on K8S DSE cluster to access HDFS and Hive (kerberized secured service). [09:47:42] For the retrieval of the credentials to be used by the spark jobs we have some good solutions like relying on Hadoop Delegation Token or kerberos ticket cache. [09:47:42] But our main challenge is on how to share this credential within spark jobs running on K8S as it will be used by multiple users (human and service account). [09:47:42] We have a proposal but it is based on vault (https://www.vaultproject.io/) which is not an available tool at wikimedia. [09:47:42] Some other approaches have been tested/designed. One of them that could work rely on the only mechanism I know to ensure mutli tenancy on K8S which is based in multiple namespace (mean one namespace per user running jobs on the cluster). Not a big fan of that solution due to the number of namespace we could have at the end. [09:47:43] Could you please take a look/review our proposal in that doc https://docs.google.com/document/d/1Aub7lUr1nPGN3MXz8FI7CCCZ5a5Y1BRpY3poVmui6AM/edit# and let us know if you have some other ideas or strong preferences for some alternative solutions? [09:48:11] * volans|off has no access to the doc... [09:53:09] elukey: _joe_: jhathaway: Thanks for your kind input. I'm sorry if I came across as argumentative yesterday and if the information I provided was inconsistent. [09:54:54] I will summarise the problem statement in a ticket as suggested. That's certainly a better place for long-form and clear discussion. [10:00:15] <_joe_> nfraison_: so as I was saying in the other channel, we should probably treat the problem a bit more holistically. We have gone other ways than vault for e.g. internal pki generation [10:00:33] <_joe_> I'll ask jbond to take a look at the doc too [10:00:34] For clarity, 1) the build pipeline is the only part of the upstream source code that we modify, replacing Alpine with Debian and 2) it's not really about being faster, it's about how difficult it is to maintain that parallel build process. [12:37:00] https://www.docker.com/blog/we-apologize-we-did-a-terrible-job-announcing-the-end-of-docker-free-teams/ [12:37:20] Look what some directed anger does [13:45:21] btullis: not at all, happy to have the discussion! [14:47:45] nfraison_: I would love to look at the doc, can you open it up to the whole org? or public? [14:48:57] jhathaway: should be fine now (open to the org) [14:49:07] <3 [15:11:18] <_joe_> nfraison_: yeah in general I advise to make docs permissions "wiki-like" [15:11:24] <_joe_> so 'anyone in the org can edit' [15:11:45] <_joe_> it's not like we don't have revision history anyways :) [15:51:54] _joe_: updated with edit rights [15:52:15] <_joe_> <3 [15:53:52] <_joe_> nfraison_: also let me say - I feel the pain of having HDFS, kubernetes and kerberos to make cooperate :) [15:54:10] :)