[15:43:31] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team, 10User-brennen: GitLab MR widget sometimes errors out on a missing index - https://phabricator.wikimedia.org/T332607 (10brennen) 05Open→03In progress p:05Triage→03Medium I //think// this should be fixed after this morning's Phabric... [16:54:06] 10GitLab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10xcollazo) [16:54:32] Follow up on my question yesterday about running tox against multiple python versions: https://gitlab.wikimedia.org/toolforge-repos/python-toolforge/-/merge_requests/1/diffs?commit_id=830436e006e2127284c4dc59bfe0726f0ae9372d seems to work fine with 1 container per python version and tox setup with `skip_missing_interpreters = True`. [17:00:09] 10GitLab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) usage right now in / is 40% and in /var/lib/docker it's 85% After running an `apt-get clean` usage in / is down to 27%. seems like something already cleaned up meanwhile. still of co... [17:02:00] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) [17:04:06] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) I wonder if this happens to be the runner from: ` < dancy> We enabled another instance wide Gitlab Runner that accepts untagged job... [17:36:28] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10dancy) >>! In T333586#8742789, @Dzahn wrote: > I wonder if this happens to be the runner from: > > > ` > < dancy> We enabled another insta... [17:43:16] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) Alright, thanks dancy! So, the remaining space in /var/lib/docker is right now about 5.9GB. The `clear-docker-cache.timer` ran 42 m... [18:27:47] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10xcollazo) Seems like I just happened to run my pipeline on an unlucky window? Does it make sense to consider running the cron jobs more fre... [18:58:44] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10hashar) You can get a view of the partitions free space at https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-b... [19:08:11] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) >>! In T333586#8743072, @xcollazo wrote: > Does it make sense to consider running the cron jobs more frequently... [19:35:39] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10hashar) For the immediate action one can delete the volumes on the runner and that solves this task. Among the larges... [19:47:20] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10xcollazo) >>! In T333586#8743236, @hashar wrote: > (I am pretty sure airflows-dag uses conda which duplicate a large a... [21:24:09] 10GitLab (Project Migration), 10Patch-For-Review, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Rename mainline branch from "master" to "main" in GitLab:repos/releng/release - https://phabricator.wikimedia.org/T329770 (10thcipriani) > (Internal Release-Engineering-Team) update tr... [21:26:41] 10GitLab (Project Migration), 10Patch-For-Review, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Rename mainline branch from "master" to "main" in GitLab:repos/releng/release - https://phabricator.wikimedia.org/T329770 (10jeena) >>! In T329770#8743492, @thcipriani wrote: >> (Inter... [21:52:15] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) ` root@runner-1030:/var/lib/docker/volumes# for volume in $(du -hs * | grep G | cut -d "G" -f2 | xargs); do ls... [22:02:14] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) >>! In T333586#8743236, @hashar wrote: > For the immediate action one can delete the volumes on the runner and... [22:04:15] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) 05Open→03Resolved a:03Dzahn available disk space in /var/lib/docker is back to: used: 21G available: 17... [22:05:39] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) well there is still https://gerrit.wikimedia.org/r/c/operations/puppet/+/904616 as a follow-up action.. so mayb... [22:52:16] 10GitLab (Infrastructure), 10SRE, 10ops-codfw, 10serviceops-collab: Install additional SSDs on gitlab2003.wikimedia.org (B5) - https://phabricator.wikimedia.org/T333304 (10Papaul) @Jelto the 2 disks are in place in gitlab2003 [22:52:54] 10GitLab (Infrastructure), 10SRE, 10ops-codfw, 10serviceops-collab: Install additional SSDs on gitlab2003.wikimedia.org (B5) - https://phabricator.wikimedia.org/T333304 (10Papaul) a:03Jelto