[10:29:46] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10serviceops-radar, and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10TheresNoTime) Was pointed to this over in {T308929}, do y'all have a test instance of GitLab running? I started to hack arou... [16:01:20] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10serviceops-radar, and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10brennen) > Was pointed to this over in T308929: Error: json: cannot unmarshal array on running mw toolhub tools list, do y'a... [16:45:20] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aokoth@cumin1001 for host gitlab2002.wikimedia.org with OS bullseye [17:17:56] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aokoth@cumin1001 for host gitlab2002.wikimedia.org with OS bullseye completed: - gi... [17:22:01] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aokoth@cumin1001 for host gitlab2003.wikimedia.org with OS bullseye [17:29:36] hey, here's a thought re: disk space on gitlab1001 - thcipriani and i did a bit of digging just now and it seems like the majority of user data that's taking up space is /var/opt/gitlab/gitlab-rails/shared/{artifacts,packages} rather than git repo data (~18 gigs vs. 7 for repos). [17:30:05] would it be reasonable to exclude those artifacts from backups for the time being? [17:31:00] we were looking at temporary limits on repo size but i don't think that'd net us much. most git repos just aren't that big. [17:31:54] for example we have 7GB of git data and 5GB of "example-job-project-0.16.0.dev0.conda.tgz" files [17:32:23] Is there an outstanding issue related to backups? [17:32:34] or just anticipating future problems? [17:33:30] lack of disk space is the core issue, we're upgrading hardware, but in the interim we're attempting to curtail disk use...somehow :) [17:34:27] the thing we just learned is we have a problem we've never had in gerrit: artifacts. [17:37:33] currently we have maximum artifacts size set to 350MB and expiration set to 2 weeks [17:38:28] we also have it set to keep the latest artifact forever. Which means we could probably expire artifacts faster. [17:39:21] that sounds pretty reasonable to me. [17:39:29] nod [17:41:20] the package registry has some pretty gnarly numbers for file sizes. They're all in bytes and none are readily recognizable as a power of 2 offhand [17:42:41] very bizarre. [17:44:12] https://docs.gitlab.com/ee/user/packages/package_registry/reduce_package_registry_storage.html [17:53:10] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aokoth@cumin1001 for host gitlab2003.wikimedia.org with OS bullseye completed: - gi... [17:53:53] > The Usage Quotas page (Settings > Usage Quotas > Storage) displays storage usage for Packages. [17:53:59] somehow I can't seem to find that page :\ [17:54:59] oh...nevermind...per repo [18:05:06] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Arnoldokoth) Reimage of gitlab2002 and gitlab2003 is complete. [18:42:19] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Update Blubber documentation, codesearch, and other references for new GitLab location - https://phabricator.wikimedia.org/T307535 (10jeena) https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/3 [19:18:56] OK: I set artifact expiration for 1 day. And I reset package size limits to ~500MB—bigger than any package we currently have. [19:19:03] but less than the previous limit [19:19:17] I also poked some folks to ask if we can delete old versions of packages they have laying around. [19:20:24] [20:27:46] thcipriani: ack - i'll check if that should live in https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/blob/main/settings.yaml [21:17:32] ack, sorry if I broke stuff there :\ [21:18:17] hrm... "Something went wrong while deleting the package." [21:18:20] "fun" [21:22:50] the "something" was "you're not signed in, dummy" [21:26:13] haha [21:27:42] alright, I deleted a bunch of packages from the web-ui, should free up some space, but not reflected on disk yet, guessing that happens as a queued job ¯\_(ツ)_/¯ [21:35:55] GC left as an exercise for the user. [21:49:06] * thcipriani worries about this [21:51:40] https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/commit/578e435c3deb8e030a386ddb00ae1d860d8cdf7f [21:52:53] one does occasionally get a "storage is infinite, why worry" vibe from this project. [21:54:22] What, me worry? [21:55:09] there are some cleanup rake tasks, I note. There are sidekiq jobs, but it's not really clear what they do... [21:55:45] like "delete_stored_files" is a queue in sidekiq, it's an empty queue, but what's it for ¯\_(ツ)_/¯ [22:03:06] * thcipriani considers tinkering with https://docs.gitlab.com/ee/raketasks/cleanup.html [22:53:43] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Release-Engineering-Team (Radar), and 4 others: Evaluate and confirm potential licensing issues for gitlab appsec pipeline tools - https://phabricator.wikimedia.org/T304737 (10sbassett) Thanks for the info on the recent case law, @bd80...