[00:38:53] 10GitLab (Upstream pit of despair 🕳️), 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: GitLab stops posting to gitlab-phabricator system hook after a failed request to the hook - https://phabricator.wikimedia.org/T329793 (10brennen) [00:53:43] 10GitLab (Upstream pit of despair 🕳️), 10Phabricator, 10Release-Engineering-Team (Radar), 10User-brennen: GitLab stops posting to gitlab-phabricator system hook after a failed request to the hook - https://phabricator.wikimedia.org/T329793 (10brennen) 05Open→03Stalled p:05Medium→03Low It looks like... [07:15:00] 10GitLab (Integrations), 10Phabricator, 10phabricator maintenance bot, 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: GitLab comments should come from a CodeReviewBot instead of gerritbot - https://phabricator.wikimedia.org/T327424 (10hashar) Thank you @brennen and @Ladsgroup , that will sli... [07:30:17] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟): Create new GitLab project group: Technical documentation - https://phabricator.wikimedia.org/T336058 (10KBach) No worries, thank you! [08:35:17] GitLab needs a short maintenance break in one hour [08:35:50] (5 minutes) [08:47:01] 10GitLab (Infrastructure), 10serviceops-collab: Cookbook sre.gitlab.upgrade fails when unpausing runners - https://phabricator.wikimedia.org/T335855 (10Jelto) 05In progress→03Resolved This is solved, cookbook uses correct number of retries now. Thanks for the fix @Volans and @eoghan ! [08:55:32] gitlab upgrade will happen later today due do ongoing deployment of mediawiki and hard dependeny, see T336162 [08:55:33] T336162: Gitlab downtime blocking scap backport - https://phabricator.wikimedia.org/T336162 [09:06:51] 10GitLab (Infrastructure), 10serviceops-collab: Add GitLab upgrades and maintenance to deployment calendar - https://phabricator.wikimedia.org/T336470 (10Jelto) [09:54:17] 10GitLab (Infrastructure), 10serviceops-collab: GitLab test instances fails to reconfigure/restart due to letsencrypt issues - https://phabricator.wikimedia.org/T336476 (10Jelto) [09:54:36] 10GitLab (Infrastructure), 10serviceops-collab: GitLab test instances fails to reconfigure/restart due to letsencrypt issues - https://phabricator.wikimedia.org/T336476 (10Jelto) [11:06:49] GitLab needs a short maintenance break in one hour (for around 5 minutes) [11:15:10] 10GitLab (Auth & Access), 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) a:03Jelto Refactoring of omniauth providers looks good on all instances. Changes as expected. Thanks again @jbond for pre... [11:16:38] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: GitLab test instances fails to reconfigure/restart due to letsencrypt issues - https://phabricator.wikimedia.org/T336476 (10Jelto) [12:34:38] 10GitLab (Infrastructure), 10serviceops-collab: Cookbook sre.gitlab.upgrade fails when another backup is in progress - https://phabricator.wikimedia.org/T336490 (10Jelto) [12:40:21] 10GitLab (Infrastructure), 10serviceops-collab: Cookbook sre.gitlab.upgrade fails when another backup is in progress - https://phabricator.wikimedia.org/T336490 (10Jelto) [12:48:57] 10GitLab (Infrastructure), 10serviceops-collab: Investigate incremental backups for GitLab - https://phabricator.wikimedia.org/T324506 (10Jelto) 05Open→03Resolved Then let's close the task and focus on T316935. [12:49:01] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops-collab, 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto) [15:33:12] 10GitLab (Infrastructure), 10Release-Engineering-Team, 10serviceops-collab: Add GitLab upgrades and maintenance to deployment calendar - https://phabricator.wikimedia.org/T336470 (10brennen) Regular windows can be added to [[https://gitlab.wikimedia.org/repos/releng/release/-/blob/main/make-deployment-calend... [18:31:58] does anyone have any ideas why https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/jobs/98453 is not seeing the $TWINE_* variables set in https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/settings/ci_cd correctly? [19:31:11] taavi: lines 226227 seem strange [19:31:41] *226-227