[01:02:19] 10Phabricator, 10Content-Transform-Team: Remove Herald rule tagging #Product-Infrastructure-Team-Backlog-Deprecated (H228) - https://phabricator.wikimedia.org/T336151 (10Aklapper) @Msantos: Hi, were you aware of T328586? [01:44:12] 10Release-Engineering-Team (Seen), 10Keyholder, 10SRE: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10Dzahn) The repo on Phabricator has been closed meanwhile. I wonder if there are things left to do here now in 2023. [04:25:48] what's the correct way to fix a disk full error on integration-agent-docker-1026? just wait for a cron job to fix it? [05:24:04] based on https://phabricator.wikimedia.org/T215278 it seems possible it would be something like .. sudo docker image prune ... [07:14:59] 10Gerrit, 10Release-Engineering-Team: Upgrade Gerrit from 3.5.5 to 3.5.6 - https://phabricator.wikimedia.org/T336339 (10hashar) [07:15:35] 10Gerrit, 10Release-Engineering-Team (Seen): Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" SSH_MSG_CHANNEL_WINDOW_ADJUST - https://phabricator.wikimedia.org/T263293 (10hashar) [07:15:37] 10Gerrit, 10Release-Engineering-Team: Upgrade Gerrit from 3.5.5 to 3.5.6 - https://phabricator.wikimedia.org/T336339 (10hashar) [07:15:39] 10Gerrit, 10Upstream: OpenSSH 9 scp does not work with Gerrit due to lack of sftp subsystem - https://phabricator.wikimedia.org/T330740 (10hashar) [07:15:41] 10Gerrit, 10Release-Engineering-Team: Upgrade Gerrit from 3.5.5 to 3.5.6 - https://phabricator.wikimedia.org/T336339 (10hashar) [07:23:31] 10Release-Engineering-Team (Seen), 10Keyholder, 10SRE: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10hashar) One can compare the content of https://phabricator.wikimedia.org/source/keyholder.git with https://gerrit.wikimedia.org/r/operations/software/keyholder Also per Fa... [07:24:14] (03PS1) 10Hashar: Merge tag 'v3.5.6' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918352 [07:26:20] (03PS2) 10Hashar: Merge tag 'v3.5.6' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918352 (https://phabricator.wikimedia.org/T336339) [07:29:04] (03PS3) 10Hashar: Merge tag 'v3.5.6' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918352 (https://phabricator.wikimedia.org/T336339) [08:07:47] (03PS1) 10Hashar: Update Gerrit to v3.5.6 [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918361 (https://phabricator.wikimedia.org/T336339) [08:15:02] (03PS2) 10Hashar: Update Gerrit to v3.5.6 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918361 (https://phabricator.wikimedia.org/T336339) [08:15:46] (03CR) 10Hashar: [C: 03+2] Merge tag 'v3.5.6' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918352 (https://phabricator.wikimedia.org/T336339) (owner: 10Hashar) [08:16:01] (03CR) 10Hashar: [C: 03+2] Update Gerrit to v3.5.6 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918361 (https://phabricator.wikimedia.org/T336339) (owner: 10Hashar) [08:20:41] 10Phabricator (Upstream), 10Upstream: Custom Policy window overflows when task name is too long - https://phabricator.wikimedia.org/T316308 (10Peachey88) Edit Task -> Visible To: [`S1`] [Public (No Login)] Drop down on the Public and select "Custom Policy" [08:21:13] (03Merged) 10jenkins-bot: Merge tag 'v3.5.6' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918352 (https://phabricator.wikimedia.org/T336339) (owner: 10Hashar) [08:21:21] (03Merged) 10jenkins-bot: Update Gerrit to v3.5.6 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918361 (https://phabricator.wikimedia.org/T336339) (owner: 10Hashar) [08:49:15] FYI I'm getting no space left on device on PCC [08:49:17] https://www.irccloud.com/pastebin/WW5tayWZ/ [08:54:35] I am stopping and upgrading Gerrit right now [08:55:20] arturo: please file it as a bug against #puppet-compiler and #continuous-integration-infrastructure and subscribe jbond :) [08:56:29] ok thanks hashar [08:57:38] 10Continuous-Integration-Infrastructure, 10Infrastructure-Foundations, 10puppet-compiler: PCC: worker out of disk space - https://phabricator.wikimedia.org/T336350 (10aborrero) [08:59:41] hashar: we saw this once while you where on holiday. from a curious look it didn;t seem to be an issue on the pcc workes i wondered if it was something on one of the CI ochastrators [09:00:18] arturo: can you add the link to the failing jenkins job [09:00:31] not sure what `FATAL: Unable to produce a script file` is about :( [09:01:01] sure [09:01:39] jbond: done [09:01:40] 10Continuous-Integration-Infrastructure, 10Infrastructure-Foundations, 10puppet-compiler: PCC: worker out of disk space - https://phabricator.wikimedia.org/T336350 (10aborrero) [09:01:48] https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/41103/console [09:02:17] 10Gerrit, 10Upstream: OpenSSH 9 scp does not work with Gerrit due to lack of sftp subsystem - https://phabricator.wikimedia.org/T330740 (10hashar) [09:02:20] 10Gerrit, 10Release-Engineering-Team (Seen): Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" SSH_MSG_CHANNEL_WINDOW_ADJUST - https://phabricator.wikimedia.org/T263293 (10hashar) [09:02:22] 10Gerrit, 10Release-Engineering-Team: Upgrade Gerrit from 3.5.5 to 3.5.6 - https://phabricator.wikimedia.org/T336339 (10hashar) 05Open→03Resolved Should be all good now. I have also resumed monitoring in Icinga. [09:02:34] cheers [09:08:59] 10Gerrit, 10Upstream: OpenSSH 9 scp does not work with Gerrit due to lack of sftp subsystem - https://phabricator.wikimedia.org/T330740 (10hashar) 05Open→03Resolved a:03hashar I have deployed Gerrit 3.5.6 which comes with [[ https://gerrit-review.googlesource.com/c/plugins/download-commands/+/359823 | do... [09:09:29] 10Gerrit, 10Release-Engineering-Team (Seen): Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" SSH_MSG_CHANNEL_WINDOW_ADJUST - https://phabricator.wikimedia.org/T263293 (10hashar) >>! In T263293#8708846, @hashar wrote: > Upstream has another potentially related fix will... [09:11:19] arturo: thanks for the task :) [09:11:26] np [09:11:37] 10Phabricator (Upstream), 10Upstream: Custom Policy window overflows when task name is too long - https://phabricator.wikimedia.org/T316308 (10Aklapper) 05Stalled→03Open p:05Triage→03Low Uh! Thank you! Indeed, confirming now [09:26:07] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Infrastructure-Foundations, 10puppet-compiler: PCC: worker out of disk space - https://phabricator.wikimedia.org/T336350 (10hashar) Looking on the Jenkins controller logs at https://integration.wikimedia.org/ci/log/WARNING/ **pcc-worker1003.puppet-diff... [09:26:23] did some investigation [09:26:29] I lack ssh access to those hosts right now [09:26:56] jbond: there is 4G free, the java stacktrace fails when creating a directory so maybe the partition is out of inodes [09:27:17] or the directory /srv/jenkins/remoting/jarCache has too many entries (if that is such a thing) [09:28:20] or it is something lower level in the underlying file system (ceph? the compute node out of disk or whatever) [09:33:00] hashar: ack thanks ill take a look and fix your access [09:39:02] jbond: is it possible to grant me shell access to the pcc host too? [09:45:30] jnuche: yes sure thing [09:45:56] jbond: thanks a lot [09:49:00] 10Phabricator, 10Release-Engineering-Team (Blocking 🧱), 10User-AKlapper, 10user-sbassett: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10Aklapper) p:05Triage→03Medium a:05Bmueller→03Aklapper Removing #Security-Team per T306708#8661705;... [10:01:08] hashar: jnuche: you shold both have access now,m also hashar you where toight look slike an inode issue [10:01:11] /dev/sda1 1310720 1310719 1 100% / [10:02:22] AH GREAT [10:02:48] also https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board only shows relative disk space availability [10:02:53] which lacks the bytes free [10:03:05] and well ... inodes but there I could not find a metric for them :/ [10:03:45] $ ssh pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud [10:03:45] Connection closed by UNKNOWN port 65535 [10:03:46] :D [10:04:07] anyway, please paste your finding on T336350! [10:04:07] T336350: PCC: worker out of disk space - https://phabricator.wikimedia.org/T336350 [10:04:35] same for me, no access :( [10:07:38] which hoss did you try? [10:07:53] if not could yuo try pcc-worker1001 [10:09:17] pcc-worker1001 works, but not pcc-worker1003 [10:09:34] ack ythat makes senses 1003 is the one with issues so puppet has nt ran there [10:10:36] gotcha, should eventually fix itself then, thx again [10:15:03] 10Phabricator, 10Release-Engineering-Team (Blocking 🧱), 10User-AKlapper, 10user-sbassett: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10Aklapper) [10:16:46] 10Phabricator, 10Release-Engineering-Team (Blocking 🧱), 10User-AKlapper, 10user-sbassett: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10Aklapper) [10:17:52] jnuche: hashar: ok should be sorted now and you should have access. ftr the issue was in /srv/jenkins/puppet-compiler some of the old check outs had not been cleaned up. ill keep an eye on things going forward possibly a bug in pcc. if we continue to have issues ill create a new /srv/jenkins/puppet-compiler partitions with a high number of inodes [10:33:20] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Infrastructure-Foundations, 10puppet-compiler: PCC: worker out of disk space - https://phabricator.wikimedia.org/T336350 (10jbond) 05Open→03Resolved a:03jbond This was caused by an exhaustion of inodes in `/srv/jenkins/puppet-compiler`. ultimatel... [10:34:12] ack, have access now [10:44:38] yeah works for me too! thank you jbond! [10:44:41] lunch time & [10:45:13] great [11:04:19] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review, 10User-brennen: Replace existing aphlict1001 with puppet-managed bullseye host - https://phabricator.wikimedia.org/T333452 (10eoghan) 05Open→03Resolved aphlict1001 was decommissioned and references in dns and wikitech... [11:04:23] 10Phabricator, 10serviceops-collab: create aphlict2001 (Phabricator realtime notifications codfw) - https://phabricator.wikimedia.org/T322369 (10eoghan) [11:25:18] 10Phabricator: Remove "puppet" tag from herald rule to assign tasks to I/F - https://phabricator.wikimedia.org/T336150 (10Aklapper) 05Stalled→03Resolved a:03Aklapper Removed #Puppet from Herald rule H389 [11:25:50] 10Project-Admins: Add cumin tag - https://phabricator.wikimedia.org/T336154 (10Aklapper) 05Open→03Resolved a:03Aklapper * Removed "cumin" from #SRE-tools alias * Created #cumin at https://phabricator.wikimedia.org/project/view/6541/ * Replaced #sre-tools with #cumin tag on 78 closed tasks with "cumin" in t... [11:46:08] 10Project-Admins: Add bitu tag - https://phabricator.wikimedia.org/T336155 (10Aklapper) 05Open→03Resolved a:03Aklapper * Created #Bitu project tag: https://phabricator.wikimedia.org/project/view/6542/ ** linked project tag description to https://wikitech.wikimedia.org/wiki/Wikimedia_IDM ** linked repo to h... [11:52:38] 10Project-Admins: Add puppet-core, puppet-infra tags - https://phabricator.wikimedia.org/T336153 (10Aklapper) * Created two project tags: https://phabricator.wikimedia.org/project/view/6543/ , https://phabricator.wikimedia.org/project/view/6544/ * Edited Herald rule H389 to once add #infrastructure-foundations w... [13:04:16] 10Project-Admins: Create project tag for MediaWiki-Metal - https://phabricator.wikimedia.org/T336170 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project has been created: https://phabricator.wikimedia.org/project/view/6545/ (In case you need to edit the project or project workboard itself... [13:24:29] 10Project-Admins: Create project tag for MediaWiki-Metal - https://phabricator.wikimedia.org/T336170 (10kostajh) Thanks! [13:37:08] (03PS1) 10AikoChou: inference-services: add RevertRisk Wikidata pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/918474 (https://phabricator.wikimedia.org/T333125) [13:48:58] (03PS1) 10Hashar: scap: use gerrit dsh group from deployment server [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918482 [13:49:02] 10Phabricator, 10Release-Engineering-Team (Blocking 🧱), 10serviceops-collab, 10User-AKlapper, 10user-sbassett: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10Dzahn) [14:02:04] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Infrastructure-Foundations, 10puppet-compiler: PCC: worker out of disk space - https://phabricator.wikimedia.org/T336350 (10hashar) 05Resolved→03Open Thank you so much @jbond ! As a side track on T336356 I have tried to add inodes to the Grafana da... [14:05:52] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Infrastructure-Foundations, 10puppet-compiler: PCC: worker out of disk space - https://phabricator.wikimedia.org/T336350 (10hashar) 05Open→03Resolved My bad, it has 87% FREE inodes or 13% usage: ` pcc-worker1003:~$ df -hi / Filesystem Inodes IUs... [14:10:51] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Seen), 10Zuul: Display Zuul status of jobs for a change on Gerrit UI - https://phabricator.wikimedia.org/T214068 (10hashar) I gave it a try again this week-end and could not find a way for a plugin to trigger an alert with a cal... [14:14:49] 10GitLab (Infrastructure), 10serviceops-collab: Investigate incremental backups for GitLab - https://phabricator.wikimedia.org/T324506 (10Arnoldokoth) Yeah, I've also poked around and I'm not really seeing the value of incremental backups at the moment... It only works with repositories as you said,,, I tested... [14:20:09] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review: Add job definitions to CasC configuration of releases Jenkins - https://phabricator.wikimedia.org/T334669 (10hashar) https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requ... [14:43:02] Project beta-code-update-eqiad build #443137: 04FAILURE in 0.69 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/443137/ [14:44:01] PROBLEM - Check systemd state on contint2001 is CRITICAL: CRITICAL - degraded: The following units failed: git_pull_charts.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [14:45:27] ACKNOWLEDGEMENT - Check systemd state on contint2001 is CRITICAL: CRITICAL - degraded: The following units failed: git_pull_charts.service daniel_zahn migration https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [14:53:17] RECOVERY - Check systemd state on contint2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [14:55:06] Yippee, build fixed! [14:55:07] Project beta-code-update-eqiad build #443138: 09FIXED in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/443138/ [16:00:27] (03CR) 10Hashar: [C: 03+2] "Dependent Puppet patch got deployed:" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918482 (owner: 10Hashar) [16:01:06] (03Merged) 10jenkins-bot: scap: use gerrit dsh group from deployment server [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/918482 (owner: 10Hashar) [16:20:16] 10Gerrit, 10Release-Engineering-Team, 10SRE, 10serviceops-collab: gerrit1003 service implementation task - https://phabricator.wikimedia.org/T326368 (10hashar) Upstream released a security update of Gerrit yesterday (3.5.6) I thus upgraded gerrit1001 and gerrit2002 to the new version this morning shortly a... [16:29:15] 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab: Isolate buildkitd build container namespaces on trusted runners - https://phabricator.wikimedia.org/T332804 (10thcipriani) 05Open→03Resolved a:03dduvall Worked first try! [16:29:52] 10Release-Engineering-Team (They Live 🕶️🧟): Test out alternative k8s platforms for gitlab-cloud-runner - https://phabricator.wikimedia.org/T330520 (10thcipriani) a:03thcipriani Will talk with other managers re:what options there are here. [16:31:03] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10phabricator maintenance bot, 10User-brennen: GitLab comments should come from a CodeReviewBot instead of gerritbot - https://phabricator.wikimedia.org/T327424 (10brennen) [16:33:11] 10GitLab (Auth & Access), 10Release-Engineering-Team (Priority Backlog 📥), 10CAS-SSO, 10Infrastructure-Foundations, 10User-brennen: GitLab sessions expire frequently - https://phabricator.wikimedia.org/T330359 (10thcipriani) Keeping an eye on this one. hypothesis: 2fa seems to be expiring on the gitlab... [16:35:19] 10GitLab (Upstream pit of despair 🕳️), 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: GitLab stops posting to gitlab-phabricator system hook after some amount of time - https://phabricator.wikimedia.org/T329793 (10thcipriani) [16:35:54] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: Migrate mediawiki/ namespace from Gerrit to GitLab - https://phabricator.wikimedia.org/T335921 (10brennen) [16:36:04] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟): Create new GitLab project group: Technical documentation - https://phabricator.wikimedia.org/T336058 (10brennen) [16:36:59] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: Migrate mediawiki/ namespace from Gerrit to GitLab - https://phabricator.wikimedia.org/T335921 (10thcipriani) a:03brennen [16:37:32] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟): Create new GitLab project group: Technical documentation - https://phabricator.wikimedia.org/T336058 (10thcipriani) a:03brennen [16:37:44] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen, 10User-dduvall: Write a GitLab "Migrating a Project" runbook / manual - https://phabricator.wikimedia.org/T307538 (10brennen) [16:38:41] 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10brennen) [16:38:47] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10User-brennen: scap deploy-promote uses latest train blocker, which is wrong if a train has been blocked over the weekend - https://phabricator.wikimedia.org/T336179 (10thcipriani) [16:39:20] 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: Reports: "Older" bar link in Histogram has wrong task query URL - https://phabricator.wikimedia.org/T336175 (10thcipriani) [16:39:53] your "Sprint name" rules. They Live was a cool movie [16:40:48] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Disallow adding Project Tags in the Subscribers field - https://phabricator.wikimedia.org/T303829 (10brennen) [16:41:57] 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops, 10serviceops-collab: Gitlab downtime blocking scap backport - https://phabricator.wikimedia.org/T336162 (10thcipriani) [16:42:12] John Carpenter FTW [16:42:23] 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: Reports: Age histogram shows nonsensical additional bucket after "Older" for projects created in last days - https://phabricator.wikimedia.org/T336152 (10thcipriani) [16:42:40] 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: InvalidArgumentException trying to render workboards when "Other Assignee" undefined - https://phabricator.wikimedia.org/T336135 (10thcipriani) [16:42:53] jnuche: did you know this one too? https://en.wikipedia.org/wiki/The_Tripods [16:43:32] 10Gerrit, 10GitLab (Misc), 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Wikimedia-GitHub, 10User-brennen: A tool for displaying the canonical location of all Wikimedia repositories and any known mirrors - https://phabricator.wikimedia.org/T330347 (10brennen) [16:43:47] did not, but will check it out now :) [16:46:18] 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: Reports: RuntimeException due to wrong assumption that boards have more columns than default Backlog - https://phabricator.wikimedia.org/T336105 (10thcipriani) [16:47:03] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Platform Engineering, 10Patch-For-Review: Migrate mediawiki/services/kask to GitLab - https://phabricator.wikimedia.org/T335691 (10thcipriani) [16:50:37] 10GitLab (Integrations), 10Release-Engineering-Team, 10User-brennen: CodeReviewBot / gitlab-phabricator should parse a bug link in Gitlab Markdown in merge request - https://phabricator.wikimedia.org/T330792 (10brennen) 05Open→03Declined Re-reading discussion above, I'm declining this on the theory that:... [16:50:42] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10brennen) [16:51:28] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MediaWiki-General, 10User-brennen: Use semantic(-ish) versioning for MediaWiki by dropping the "1." part of the version number - https://phabricator.wikimedia.org/T336128 (10brennen) [16:53:08] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Infrastructure-Foundations, 10puppet-compiler: PCC runs failing with complaints about disk space - https://phabricator.wikimedia.org/T335111 (10hashar) [16:53:10] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Infrastructure-Foundations, 10puppet-compiler: PCC: worker out of disk space - https://phabricator.wikimedia.org/T336350 (10hashar) [16:53:38] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Infrastructure-Foundations, 10puppet-compiler: PCC runs failing with complaints about disk space - https://phabricator.wikimedia.org/T335111 (10hashar) That one got solved by @jbond today via T336350. The host was out of in... [16:54:59] 10Release-Engineering-Team, 10Scap, 10Performance-Team (Radar): Scap sync stops when searching in GNU screen, starts running hours later when resuming screen - https://phabricator.wikimedia.org/T334420 (10brennen) This does indeed feel like the ghost of {T223287}. [16:56:28] 10Release-Engineering-Team (They Live 🕶️🧟), 10Scap: `scap backport` dependency check for cherry-picked mediawiki commits - https://phabricator.wikimedia.org/T334984 (10thcipriani) [16:57:06] 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521 (10thcipriani) [17:36:10] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟): Create new GitLab project group: Technical documentation - https://phabricator.wikimedia.org/T336058 (10brennen) 05Open→03Resolved Sorry for the delay on this one, done! [17:49:35] 10Gerrit, 10Release-Engineering-Team, 10SRE, 10serviceops-collab: gerrit1003 service implementation task - https://phabricator.wikimedia.org/T326368 (10Dzahn) [17:51:51] 10Gerrit, 10Release-Engineering-Team, 10SRE, 10serviceops-collab: gerrit1003 service implementation task - https://phabricator.wikimedia.org/T326368 (10Dzahn) I made several edits to the migration plan doc that is transcluded here from https://phabricator.wikimedia.org/P47782. See my comments there for de... [17:54:34] 10Gerrit, 10Release-Engineering-Team, 10SRE, 10serviceops-collab: gerrit1003 service implementation task - https://phabricator.wikimedia.org/T326368 (10Dzahn) >>! In T326368#8842150, @hashar wrote: > ..but can't ssh into it. Turns out it does not have any of the git repositories under /srv/gerrit/git and... [18:13:57] 10Phabricator, 10Trust-and-Safety, 10cloud-services-team, 10wikitech.wikimedia.org: Account recovery help needed for Developer account [Oleksandr Tsyba (WMDE), oleksandr_tsyba_WMDE] - https://phabricator.wikimedia.org/T336429 (10Aklapper) [18:17:39] 10Gerrit, 10Release-Engineering-Team, 10SRE, 10serviceops-collab: gerrit1003 service implementation task - https://phabricator.wikimedia.org/T326368 (10Dzahn) @hashar @thcipriani status update: I have freshly: - rsynced /srv/gerrit - rsynced /var/lib/gerrit2 - for the lfs path change: copied /srv/gerrit/... [18:18:54] hashar: ^ update on ticket, cheers [18:19:03] you can try deploying [18:42:04] 10Gerrit, 10Release-Engineering-Team, 10SRE, 10serviceops-collab: gerrit1003 service implementation task - https://phabricator.wikimedia.org/T326368 (10Dzahn) Just realized one more thing we have to remember / add to the plan. After we switch we must active replication from gerrit1003 and deactivate it fr... [18:53:54] 10Gerrit, 10Release-Engineering-Team, 10SRE, 10serviceops-collab, 10Patch-For-Review: gerrit1003 service implementation task - https://phabricator.wikimedia.org/T326368 (10Dzahn) steps we have to add: - move/copy lfs data which is in new location on new host (for a good reason) (https://gerrit.wikimedia... [20:03:04] 10Release-Engineering-Team (Seen), 10Scap, 10SRE, 10serviceops: Enable scap to roll back broken changes to MediaWiki - https://phabricator.wikimedia.org/T225207 (10Dzahn) [20:27:40] 10Release-Engineering-Team, 10SRE, 10serviceops-collab: URL shortener subdomains for useful Wikimedia infrastructure - https://phabricator.wikimedia.org/T223319 (10Dzahn) [20:33:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops-collab: Upgrade ci ssh key to ecdsa - https://phabricator.wikimedia.org/T177826 (10Dzahn) [20:38:41] 10Phabricator, 10Trust-and-Safety, 10cloud-services-team, 10wikitech.wikimedia.org, 10User-bd808: Account recovery help needed for Developer account [Oleksandr Tsyba (WMDE), oleksandr_tsyba_WMDE] - https://phabricator.wikimedia.org/T336429 (10bd808) 05Open→03Resolved a:03bd808 ` $ ssh root@bastion-... [20:40:13] 10Phabricator, 10Trust-and-Safety, 10cloud-services-team, 10wikitech.wikimedia.org, 10User-bd808: Account recovery help needed for Developer account [Oleksandr Tsyba (WMDE), oleksandr_tsyba_WMDE] - https://phabricator.wikimedia.org/T336429 (10bd808) @oleksandr_tsyba_WMDE Please make a separate task for t... [23:17:45] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10CodeReviewBot) brennen closed https://gitlab.wikimedia.org/brennen/test/-/merge_requests/13... [23:19:39] !log reconfigured and restarted gitlab-webhooks to use CodeReviewBot for messages instead of gerritbot (T327424) [23:19:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:19:42] T327424: GitLab comments should come from a CodeReviewBot instead of gerritbot - https://phabricator.wikimedia.org/T327424 [23:20:21] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Comment on Phabricator tasks for new, merged, and abandoned changes on GitLab - https://phabricator.wikimedia.org/T324150 (10brennen) [23:20:41] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Comment on Phabricator tasks for new, merged, and abandoned changes on GitLab - https://phabricator.wikimedia.org/T324150 (10brennen) 05Open→03Resolved [23:20:49] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10brennen) [23:20:58] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10phabricator maintenance bot, 10User-brennen: GitLab comments should come from a CodeReviewBot instead of gerritbot - https://phabricator.wikimedia.org/T327424 (10brennen) 05In progress→03Resolved