[09:29:28] GitLab needs a short maintenance break in one hour [10:07:35] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Filburt - https://phabricator.wikimedia.org/T361191 (10Filburt) 03NEW [10:33:28] GitLab upgrade done [13:35:32] I hid the "assigned to" field from https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/117/ given it's been causing some confusion recently [13:38:26] thanks [16:06:54] taavi: good call [16:33:17] 10GitLab (Infrastructure), 06collaboration-services, 10Release-Engineering-Team (Radar): Add GitLab upgrades and maintenance to deployment calendar - https://phabricator.wikimedia.org/T336470#9670168 (10CodeReviewBot) thcipriani updated https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/69 a... [16:33:18] 10GitLab (Infrastructure), 06collaboration-services, 10Release-Engineering-Team (Radar): Add GitLab upgrades and maintenance to deployment calendar - https://phabricator.wikimedia.org/T336470#9670169 (10CodeReviewBot) thcipriani merged https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/69 ad... [16:38:18] 10GitLab (Account Approval), 06Release-Engineering-Team: 14Requesting GitLab account activation for Filburt - 14https://phabricator.wikimedia.org/T361191#9670191 (10brennen) 05Open→03Resolved [18:32:10] 10GitLab, 06collaboration-services: gitlab: enforce 2fa for admins - https://phabricator.wikimedia.org/T361277 (10Dzahn) 03NEW [18:32:51] 10GitLab, 06collaboration-services: gitlab: enforce 2fa for admins - https://phabricator.wikimedia.org/T361277#9670756 (10Dzahn) [20:24:17] hey folks! I'm rebuilding the gitlab-runner puppetserver and for unclear reasons 'profile::gitlab::runner::token' was set with the old puppetmaster but is unset with the new one... some kind of subtle hiera order thing I'm guessing. [20:24:27] So my question is... can I just set it to 'false' throughout the project? [20:24:56] That might be a jelto question [20:26:02] brennen, any idea? (pinging you because there's a higher chance of you being awake) [20:26:13] hmm [20:27:50] The runner need a private Profile::GitLab::Runner::token to register to the GitLab test instance. This should not be public, as it would be annoying to remove additional runners [20:28:02] That project is generally weird since it has a bunch of hiera things checked into operations/puppet rather than in horizon so I'm on my back foot [20:28:33] ok, so that would be checked into the labs/private repo on your puppetmaster right? [20:28:35] So... [20:28:37] * andrewbogott looks [20:28:55] it isn't [20:28:59] However the token is used only for registration of the runner, so it wont break stuff immediately [20:29:08] so jelto where would you expect that token to come from? [20:30:25] The tokens are generated on the gitlab server by an administrator and then distributed to the individual runners (using hiera) [20:30:33] old puppet server is gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud, new one is gitlab-runners-puppetserver-01.gitlab-runners.eqiad1.wikimedia.cloud; the new one has clones of both puppet git repos from the old master [20:30:42] jelto: and then removed from hiera afterwards? [20:31:05] no they stay in hiera in case the runner has to be re-registered (for example when doing a reimage) [20:31:22] at least that is what we do in production and we copied that workflow to wmcs as well [20:31:24] ok, can you show me where they are then? [20:33:06] That project has its own puppetmaster which is usually done to maintain private secrets. In this case, though, there aren't any secrets or any other local changes. [20:33:14] So I guess I'm not sure why it has a puppetmaster at all [20:33:40] I think they are in /etc/puppet/secret [20:34:24] oh wow I've never heard of that being used before [20:34:38] why not... just make a patch in labs/private? [20:35:35] If you don't object, I'll move those secrets into a proper patch on the new server. [20:36:08] I think in the other wmcs projects we also use a different location. But we did not touch it again after that one worked in gitlab-runners. We could do that yes. But do you mean the public labs private? [20:37:24] The way every other project does this (as far as I know) is with a local patch in the checkout of the puppet repo. Then, there are elaborate automatic scripts that periodically rebase with the upstream changes, preserving local patches [20:37:29] and making noise if the rebase fail [20:37:31] *fails [20:37:53] ah yes that sounds good and similar what we do in other projects afaik [20:38:39] ok [20:38:46] I'll ping again if it goes poorly :) [20:40:32] it _should_ not have a immediate effect and the runners on the test host should keep working. It would break if someone triggers a reimage for example [20:41:34] but thanks for moving the puppetserver, let me know if you have any more questions (I might answer next week though) [20:42:41] one typo: the runners are registered to the prod host not the test host, but anyway they should keep working and we also have others as well ;) [20:49:25] jelto: I think things are set now. FYI, the git repos are now in /srv/git (which is a cinder volume so can be transferred to new puppetservers at upgrade time) [20:49:55] thats nice, thank you! [20:51:36] I'm going to turn off the old puppetmaster but I'll leave it for you to delete once you're convinced that all is well. [20:56:19] sounds good, I'll do that next week [21:10:11] 10GitLab (Infrastructure), 06collaboration-services, 10Release-Engineering-Team (Radar): Add GitLab upgrades and maintenance to deployment calendar - https://phabricator.wikimedia.org/T336470#9671533 (10Dzahn) The new deployment calender will be generated tomorrow. Then we should see our new GitLab window. [21:10:30] 10GitLab (Infrastructure), 06collaboration-services, 10Release-Engineering-Team (Radar): Add GitLab upgrades and maintenance to deployment calendar - https://phabricator.wikimedia.org/T336470#9671534 (10Dzahn) 05Open→03In progress [22:24:13] 10GitLab-Application-Security-Pipeline, 13Patch-For-Review, 07Security: Design AppSec Pipeline metrics approach - https://phabricator.wikimedia.org/T342467#9671648 (10sbassett) Should be ready for review now: https://gitlab.wikimedia.org/repos/security/gitlab-ci-security-templates-stats/-/merge_requests/1#no... [22:29:55] 10GitLab-Application-Security-Pipeline, 06Security-Team, 07SecTeam-Processed, 07Security: Update the image used by the python-bandit appsec include - https://phabricator.wikimedia.org/T360721#9671655 (10sbassett) Via the new appsec pipeline metrics cli (T342467), I was able to find that only a couple of re... [22:37:12] 10GitLab (Integrations), 06collaboration-services, 10Phabricator, 10Release-Engineering-Team (Now this 🫠): Get GitLab to render `T{\d}+` in MR overviews, comments, etc. as links to Phabricator - https://phabricator.wikimedia.org/T337570#9671679 (10brennen) a:05dduvall→03brennen > This seems to have tra... [22:37:14] 10GitLab (Integrations), 06collaboration-services, 10Phabricator, 10Release-Engineering-Team (Now this 🫠): Get GitLab to render `T{\d}+` in MR overviews, comments, etc. as links to Phabricator - https://phabricator.wikimedia.org/T337570#9671683 (10brennen) 05Open→03In progress