[09:54:19] Hi! Is there any image in docker-registry.wikimedia.org that I can use on CI and has the git binary? the default buster one does not have it it seems [10:05:11] this seems small enough and has git: docker-registry.wikimedia.org/releng/curl would it be ok to use that? [11:53:45] 10GitLab (CI & Job Runners), 10Data-Engineering, 10Performance Issue: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10hashar) I am adding back #data-engineering since the slowness comes from how the test environments have been defined. See my analysis from last year above: T311111#8... [12:04:31] Hi! I'm getting errors when trying to start a job on gitlab using the cloud runners: ERROR: Job failed (system failure): prepare environment: setting up credentials: Post "https://10.245.0.1:443/api/v1/namespaces/gitlab-runner/secrets": EOF. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information [12:04:50] retried a few times: https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/jobs/114008 [12:05:32] has anything happened? [12:28:27] it might be just one specific runner, rerunning ended up eventually passing [12:37:11] this runner in specific: gitlab-runner-cbdb8c847-42dxt [12:37:15] I'll open a task [12:40:43] 10GitLab: [gitlab runner] Jobs failing on runner gitlab-runner-cbdb8c847-42dxt with Post "https://10.245.0.1:443/api/v1/namespaces/gitlab-runner/secrets": EOF - https://phabricator.wikimedia.org/T340841 (10dcaro) [12:55:21] 10GitLab (CI & Job Runners), 10Data-Engineering, 10Performance Issue: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10xcollazo) I think @Antoine_Quhen took care of this via custom docker images. Right Antoine? [17:44:13] 10GitLab (CI & Job Runners), 10Data Engineering and Event Platform Team: Allow a shared, protected runner for the data-engineering group in GitLab - https://phabricator.wikimedia.org/T295045 (10JArguello-WMF) [17:44:48] 10GitLab (Project Migration), 10API Platform, 10Anti-Harassment, 10Cloud-Services, and 18 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10JArguello-WMF) [19:54:10] 10GitLab, 10Release-Engineering-Team (Priority Backlog 📥): WMCS GitLab runners running frequently running out of disk space - https://phabricator.wikimedia.org/T340887 (10dduvall) [20:07:53] 10GitLab (CI & Job Runners), 10commit-message-validator, 10Patch-For-Review, 10User-bd808: Update commit-message-validator to work nicely with GitLab repos - https://phabricator.wikimedia.org/T339307 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/repos/ci-tools/commit-message-validator/-/merge... [20:08:09] 10GitLab (CI & Job Runners), 10commit-message-validator, 10Patch-For-Review, 10User-bd808: Update commit-message-validator to work nicely with GitLab repos - https://phabricator.wikimedia.org/T339307 (10CodeReviewBot) [20:25:59] 10GitLab, 10Release-Engineering-Team (Priority Backlog 📥): WMCS GitLab runners running frequently running out of disk space - https://phabricator.wikimedia.org/T340887 (10dduvall) 05Open→03In progress p:05Triage→03Medium a:03dduvall After poking around a bit on `runner-1029.gitlab-runners.eqiad1.wiki... [20:27:56] 10GitLab, 10Release-Engineering-Team (Priority Backlog 📥): WMCS GitLab runners running frequently running out of disk space - https://phabricator.wikimedia.org/T340887 (10dduvall) Just adding that `buildctl prune` does not remove the cache entries either, which confirms there are indeed not functionally reclai... [20:53:52] 10GitLab (CI & Job Runners), 10commit-message-validator, 10Patch-For-Review, 10User-bd808: Update commit-message-validator to work nicely with GitLab repos - https://phabricator.wikimedia.org/T339307 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/repos/ci-tools/commit-message-validator/-/merge... [21:08:15] our prod gitlab servers have just been upgraded to 15.11.10 [21:08:46] reason https://about.gitlab.com/releases/2023/06/29/security-release-gitlab-16-1-1-released/ [21:14:29] thanks for handling that on a friday mutante [21:15:37] :) very long weekend. it kind of seemed not ideal either way.. but better now :) [21:28:11] shared and trusted runners done [22:06:44] 10GitLab, 10Release-Engineering-Team (Priority Backlog 📥): WMCS GitLab runners running frequently running out of disk space - https://phabricator.wikimedia.org/T340887 (10dancy) I'm very curious to learn what you discover.