[08:13:48] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10858282 (10MatthewVernon) >>! In T378922#10848007, @Jelto wrote: >>>! In T378922#10847339, @MatthewVerno... [08:17:59] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10858295 (10Jelto) Thanks a lot @MatthewVernon, I can confirm the buckets are gone. I'll re-create the bu... [08:18:36] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10858296 (10Jelto) [08:22:29] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10858303 (10MatthewVernon) Yeah, the apus cluster isn't ideal for buckets with a very large number of obj... [08:34:16] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10858327 (10Jelto) [10:58:56] 10GitLab (Infrastructure), 06collaboration-services: Check GitLab artifact retention time - https://phabricator.wikimedia.org/T395014#10858878 (10Jelto) p:05Triage→03High a:03Jelto [12:20:07] 10GitLab (Infrastructure), 06collaboration-services: Check GitLab artifact retention time - https://phabricator.wikimedia.org/T395014#10859138 (10Jelto) I've done a bit more research using [https://docs.gitlab.com/user/storage\_management\_automation/#manage-cicd-pipeline-storage](https://docs.gitlab.com/user/... [14:44:38] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Recent incidents of buildkitd's storage volume filling up - https://phabricator.wikimedia.org/T395097#10859698 (10dancy) I think this is related to the `publish-build` job of https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker/-/blob/main/.gitl... [18:05:16] 10GitLab (CI & Job Runners), 06cloud-services-team, 10Release-Engineering-Team (Priority Backlog 📥): Recent incidents of buildkitd's storage volume filling up - https://phabricator.wikimedia.org/T395097#10860917 (10thcipriani) >>! In T395097#10859698, @dancy wrote: > I think this is related to the `publish-b... [18:10:32] 10GitLab (CI & Job Runners), 06cloud-services-team, 10Release-Engineering-Team (Priority Backlog 📥): Recent incidents of buildkitd's storage volume filling up - https://phabricator.wikimedia.org/T395097#10860942 (10dancy) >>! In T395097#10860917, @thcipriani wrote: >>>! In T395097#10859698, @dancy wrote: >>... [18:13:33] 10GitLab (CI & Job Runners), 06cloud-services-team, 10Release-Engineering-Team (Priority Backlog 📥): Recent incidents of buildkitd's storage volume filling up - https://phabricator.wikimedia.org/T395097#10860953 (10dancy) 05Open→03In progress p:05Triage→03Low a:03dancy