[08:58:09] 10GitLab, 06collaboration-services: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066#10021889 (10Jelto) I deleted a lot of very old GitLab logs (mostly migration and upgrade logs from 2023), freeing up some disk space. Now the disk usage fluctuates between 60% and 80%. I hav... [09:05:25] 10GitLab, 06collaboration-services: Backups are failing on the GitLab test instance - https://phabricator.wikimedia.org/T371222 (10Jelto) 03NEW [09:06:12] 10GitLab, 06collaboration-services: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066#10021914 (10Jelto) [09:06:15] 10GitLab (Infrastructure), 06collaboration-services: Increase disk size for GitLab test instance - https://phabricator.wikimedia.org/T369837#10021917 (10Jelto) →14Duplicate dup:03T371066 [09:47:58] 10GitLab, 06collaboration-services: Backups are failing on the GitLab test instance - https://phabricator.wikimedia.org/T371222#10022121 (10Jelto) p:05Triage→03Medium a:03Jelto Manually triggering a backup with ` sudo /usr/bin/gitlab-backup create CRON=1 STRATEGY=copy GZIP_RSYNCABLE=yes SKIP=builds,art... [12:36:44] 10GitLab, 06collaboration-services: Backups are failing on the GitLab test instance - https://phabricator.wikimedia.org/T371222#10022726 (10Jelto) The backup works again with reduced `max_storage_concurrency`, but it still generated a lot of load, making the host almost unusable and triggering probe down alert... [13:04:50] 10GitLab, 06collaboration-services: Backups are failing on the GitLab test instance - https://phabricator.wikimedia.org/T371222#10022827 (10Jelto) a:05Jelto→03eoghan After scaling the gitlab test instance to `g4.cores4.ram8.disk20` backups are working fine again and don't slow down the system significantly... [15:26:14] 10GitLab (Integrations), 06collaboration-services, 06Infrastructure-Foundations, 06serviceops, and 2 others: Container image reports in debmonitor are broken - https://phabricator.wikimedia.org/T348876#10023573 (10elukey) To keep archives happy - the code is fully deployed on build2001, and I have raised t... [15:28:36] 10GitLab, 06collaboration-services: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066#10023584 (10LSobanski) a:03Dzahn [15:28:38] 10GitLab, 06collaboration-services: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066#10023586 (10LSobanski) p:05Triage→03Medium [22:09:09] 10GitLab, 06collaboration-services: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066#10025636 (10Dzahn)