[03:04:32] I'm a little behind on things, is it possible to have CI on projects under toolforge-repos? [12:24:28] 10GitLab (Project Migration), 10Quarry, 10Patch-For-Review: Move quarry to gitlab or github - https://phabricator.wikimedia.org/T308978 (10rook) a:03rook [12:27:11] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto) Backups on production `gitlab1004` fail with ` Errno::EACCES: Permission denied @ dir_s_mkdir - /srv/gitlab-backup/db ` since Sep... [12:30:47] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jbond) @jelto i think this is fall out from https://gerrit.wikimedia.org/r/c/operations/puppet/+/809095 which changed the permission of t... [12:38:36] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto) @jbond thanks for the context! I also found the following line in the puppet log: ` Sep 7 11:42:31 gitlab1004 puppet-agent[93484... [12:39:57] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jbond) > I guess this permission was automatically setup when installing/bootstrapping GitLab apt package for the first time. yes i imagi... [12:49:10] 10GitLab (Project Migration), 10Quarry, 10Patch-For-Review: Move Quarry from Gerrit to GitHub - https://phabricator.wikimedia.org/T308978 (10Aklapper) [14:23:22] 10GitLab (Project Migration), 10Quarry: Move Quarry from Gerrit to GitHub - https://phabricator.wikimedia.org/T308978 (10rook) [15:29:08] legoktm: there are no runners allocated there as far as I know. Access to "very untrusted" runners was part of the discussion that brennen and I had way back in February. I do not know that anyone has quite figured out what "very untrusted" means and where they will live yet. [15:29:48] I was hoping to start some conversations about this "soon" [15:30:52] happy to see someone else pick it up as a project though to work out how we can safely provide CI resources there [15:37:42] 10GitLab (Project Migration), 10Quarry: Move Quarry from Gerrit to GitHub - https://phabricator.wikimedia.org/T308978 (10rook) 05Open→03Resolved [16:12:12] bd808: the runners are in a VPS right? Everyone with access to toolforge-repos has Toolforge shell access and can execute code there anyways, so is there a possible escalation? [16:14:57] I think the concern is more how to shut off access if found to be misbehaving. For Toolforge the kill switch is blocking the Developer account on wikitech. For gitlab I don't know what the equivalent is. It's a tiny percentage, but we will have people try to use CI to run crypto miners when they can control the CI config and the code under test. [16:15:31] these are moles we already have to whack elsewhere in our ecosystem [16:16:27] a merge request is also not quite the same as shell access [16:17:30] by which I mean an MR can come from anyone who has created a Developer account, but shell on Toolforge (at least currently) has a bit more access control [16:18:12] but I do think that we should be able to allow at worst something like the current "trusted contributors" setup in jenkins [16:18:24] I just don't know how to do it :) [16:19:18] I think the theory I have heard in the past for the "very untrusted" runners is that they would be in a 3rd party cloud, but I might be wrong about that. [16:21:26] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto) Backups on production `gitlab1004` are fixed again with https://gerrit.wikimedia.org/r/831083. The puppet run reported ` Notice:... [16:23:48] mutante: circling back to yesterday's conversation re: buildkit dns woes, were we always restricting network egress so much from trusted runners or is this new? [16:24:35] i'm wondering why this problem is only surfacing now. i thought we'd done a full round of testing image builds on buildkitd after deployment [16:27:56] or jelto ^ [18:09:12] dduvall: I am not aware of new restrictions at least [19:13:06] 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall) Yesterday I circled back to this task to verify that building a basic image via buildkitd and... [21:47:18] bd808: how is it better to pay e.g. amazon for the cryptomining we are unable to avoid (plus all legitimate usage) than having that on one of our servers? [21:47:37] Platonides: not my decision [21:47:38] sure, that means that if a job compromises the machine, it compromises *Amazon*'s machine [21:48:13] I understand one would want to have the very untrusted runner as a separate physical machine [21:48:59] but it should be relatively simpel to setup [21:49:19] *and* it doesn't require expertise on that third party cloud to correctly build that [21:50:39] there is more to most things than we think of on first encounter. one thing to realize is that we can buy managed kubernetes clusters from vendors and that may be easier to do than to find hardware & rack space & network & SREs to build one on-prem. [21:54:28] * Platonides was thinking in leaving for the untrusted attackers just an "old and filthy" server [21:54:54] (or not so old, its power bill shouldn't be too inefficient) [21:55:27] it seems strange to go to a third party at this point [21:59:52] the times where we had "misc pools" with "old servers" to use are over as well, actually [22:00:03] nowadays we have to order from the menu that dcops offers [22:00:22] that is not a comment on any of the other parts of this [22:35:51] 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall) I ran an even more minimal test to see if I could at least build a scratch image containing a... [22:42:04] 10GitLab (Project Migration), 10Quarry: Move Quarry from Gerrit to GitHub - https://phabricator.wikimedia.org/T308978 (10rook) [22:43:24] 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall) Also cc'ing task {T317341} in case this is related to any changes made on account of security...