[14:35:15] <wikibugs>	 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto) #### New partman config  The new partman config on `gitlab2003` increased the size of the backup volume:  ` vg-root...
[14:40:31] <jelto>	 brennen: are there any thoughts or policies about maximum repo size? I noticed there are some projects using multiple GB of space in https://gitlab.wikimedia.org/admin/projects?sort=storage_size_desc.
[14:40:32] <jelto>	 Sadly limiting this in config is a premium feature: https://docs.gitlab.com/ee/user/admin_area/settings/account_and_limit_settings.html#repository-size-limit
[14:54:01] <brennen>	 jelto: i think a lot of the size on projects here is artifacts, and we _can_ tune max_artifacts_size afaik
[14:55:01] <brennen>	 ...or maybe it's packages rather than artifacts, considering i see some packages ~750 MiB
[15:07:45] <jelto>	 brennen: ah yes correct there are dedicated settings for package/artifact file sizes and artifact sizes and they are in the region of 300-500MiB. However some project (mostly reseach) have multiple 400mb files in their packages totaling in multiple GiBs.
[15:07:47] <jelto>	 Are you ok if I reach out to them and ask if that is intended and needed?
[15:09:35] <brennen>	 sure - we don't have a policy as such, yet, but no harm in asking what their needs are here.
[15:10:54] <brennen>	 some previous discussion on artifact sizes here: https://phabricator.wikimedia.org/T292372
[16:03:44] <thcipriani>	 speaking of size use, I was looking at: https://gitlab.wikimedia.org/admin/usage_trends today -- looks like growth of...everything...is pretty linear on gitlab :|
[16:39:35] <bd808>	 thcipriani: we can make things look exponential for a bit when we move the toolforge tool repos. ;)
[16:40:25] <thcipriani>	 oh. good. :)
[16:40:58] <bd808>	 I fear that storage is going to end up being a thing somebody has to care about. We have to poke people about once a quarter to cleanup piles of garbage files in the Toolforge NFS shares. Hoarding will continue until space is exhausted.
[16:44:52] <thcipriani>	 I was looking at that chart because storage is a thing we're starting to care about (following along with https://phabricator.wikimedia.org/T274463 ). There are some knobs for us to turn here (limiting artifact size, for example), but, yeah, figuring out how/when to poke is probably also going to have to be a thing
[16:45:23] <thcipriani>	 do you have automated poking? Or is it manual investigation on toolforge bd808 ?
[16:47:22] <bd808>	 thcipriani: we have a usage alert that goes off when we hit some % of capacity. Then someone usually runs a `nice -9 find ...` sort of command to find the top N $HOMEs by size.
[16:48:59] <thcipriani>	 *nod* makes sense
[16:49:16] <bd808>	 We used to have a grafana dashboard, but I think it died as data collectors changed over time
[16:51:39] <bd808>	 T301508 is an example task
[16:51:40] <stashbot>	 T301508: 2022-02-10: tools nfs share cleanup - https://phabricator.wikimedia.org/T301508
[22:57:17] <wikibugs>	 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10thcipriani) @LSobanski and I talked about predicting storage space of GitLab as we grow. We've learned that storage growth is much differe...