[07:44:26] 10GitLab (Infrastructure), 10SRE, 10ops-codfw, 10serviceops-collab: Install additional SSDs on gitlab2003.wikimedia.org (B5) - https://phabricator.wikimedia.org/T333304 (10Jelto) p:05Triage→03Medium Thanks @Papaul for the quick installation! I can confirm new disks are available on the host: ` gitlab... [08:15:01] 10GitLab (Infrastructure), 10SRE, 10ops-codfw, 10serviceops-collab: Install additional SSDs on gitlab2003.wikimedia.org (B5) - https://phabricator.wikimedia.org/T333304 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye [08:48:03] 10GitLab (Infrastructure), 10SRE, 10ops-codfw, 10serviceops-collab: Install additional SSDs on gitlab2003.wikimedia.org (B5) - https://phabricator.wikimedia.org/T333304 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye... [09:31:28] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10hashar) >>! In T333586#8743249, @xcollazo wrote: >>>! In T333586#8743236, @hashar wrote: >> (I am pretty sure airflows... [09:48:49] 10GitLab (CI & Job Runners), 10mwbot-rs: mwbot-rs large cache usage on Gitlab CI - https://phabricator.wikimedia.org/T333663 (10hashar) [10:14:24] 10GitLab (CI & Job Runners), 10mwbot-rs: mwbot-rs large cache usage on Gitlab CI - https://phabricator.wikimedia.org/T333663 (10hashar) I have inspected a cache for `countcount/mwbot` which has 1734MBytes: A friendly view: | .cargo/registry | 427 MB |--|-- | \- cache | 22 MB | \- index | 225 MB | \- src |... [10:34:52] 10GitLab (CI & Job Runners), 10mwbot-rs: mwbot-rs large cache usage on Gitlab CI - https://phabricator.wikimedia.org/T333663 (10Lucas_Werkmeister_WMDE) >>! In T333663#8744903, @hashar wrote: > ./.cargo/registry/index/github.com-1ecc6299db9ec823/.git is apparently a full clone of a repository with a 214 M packf... [10:35:46] 10GitLab (CI & Job Runners), 10mwbot-rs: mwbot-rs large cache usage on Gitlab CI - https://phabricator.wikimedia.org/T333663 (10hashar) [10:39:33] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10hashar) I have filed {T333663} for `mwbot-rs/mwbot` and `countcount/mwbot`. For data engineering I think {T311111} is... [10:40:14] 10GitLab (CI & Job Runners), 10mwbot-rs: mwbot-rs large cache usage on Gitlab CI - https://phabricator.wikimedia.org/T333663 (10hashar) [11:11:43] 10GitLab (Infrastructure), 10SRE, 10ops-codfw, 10serviceops-collab: Install additional SSDs on gitlab2003.wikimedia.org (B5) - https://phabricator.wikimedia.org/T333304 (10Jelto) 05Open→03Resolved The reimage happend on `gitlab2003` but it seems the partman config is not producing the expected result.... [11:23:17] 10GitLab (Infrastructure), 10serviceops-collab: Troubleshoot partman config for two additional disks on GitLab hosts - https://phabricator.wikimedia.org/T333674 (10Jelto) [12:03:46] *Brief Gitlab Interruption*: We'll be updating the version of Gitlab in production in the next while, so please expect some brief interruptions in the next 45-60 minutes [15:19:19] 10GitLab (CI & Job Runners), 10mwbot-rs: mwbot-rs large cache usage on Gitlab CI - https://phabricator.wikimedia.org/T333663 (10Legoktm) I actually don't know why files are getting cached in the first place?? I removed (or tried to) all caching back in November: https://gitlab.wikimedia.org/repos/mwbot-rs/rust... [15:52:40] 10GitLab (Project Migration), 10Patch-For-Review, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Rename mainline branch from "master" to "main" in GitLab:repos/releng/release - https://phabricator.wikimedia.org/T329770 (10dancy) The `master` branch still exists at the moment which... [16:02:14] 10GitLab (CI & Job Runners), 10mwbot-rs: mwbot-rs large cache usage on Gitlab CI - https://phabricator.wikimedia.org/T333663 (10dancy) >>! In T333663#8745979, @Legoktm wrote: > I actually don't know why files are getting cached in the first place?? I removed (or tried to) all caching back in November: https://... [16:04:23] Oooh [16:57:01] 10GitLab (CI & Job Runners), 10Data-Engineering-Planning, 10Release-Engineering-Team, 10Performance Issue: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10xcollazo) @lbowmaker for your consideration. We could win significant developer productivity by tackling this one. [17:10:52] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10xcollazo) >>! In T333586#8743160, @Dzahn wrote: >>>! In T333586#8743072, @xcollazo wrote: >> Does it make sense to consider running the cro... [17:34:56] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) >>! In T333586#8746429, @xcollazo wrote: > Now that I think about it more, perhaps running `clear-docker-cache.timer` hourly is too a... [17:57:32] 10GitLab (CI & Job Runners), 10mwbot-rs: mwbot-rs large cache usage on Gitlab CI - https://phabricator.wikimedia.org/T333663 (10Legoktm) a:03Legoktm Ohhhh got it, thanks. So then I can add `rm -rf target && rm -rf .cargo` to the end of the job as cleanup. Will get to that in the next few days. [18:17:05] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) >>! In T333586#8746491, @Dzahn wrote: >>> I wouldn't mind merging a patch or uploading one that goes back to "every 8 hours". But t... [19:05:10] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10xcollazo) >>! In T333586#8746569, @Dzahn wrote: >>>! In T333586#8746491, @Dzahn wrote: >>>> I wouldn't mind merging a patch or uploading one... [20:31:33] 10GitLab (Project Migration), 10Patch-For-Review, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Rename mainline branch from "master" to "main" in GitLab:repos/releng/release - https://phabricator.wikimedia.org/T329770 (10jeena) >>! In T329770#8746098, @dancy wrote: > The `master`...