[00:21:34] (03CR) 10Jeena Huneidi: [C: 03+2] "Tested sync-file and scap backport, all looks good in train-dev." [tools/scap] - 10https://gerrit.wikimedia.org/r/805763 (https://phabricator.wikimedia.org/T308476) (owner: 10Ahmon Dancy) [00:26:02] (03Merged) 10jenkins-bot: scap backport: deploy to mwdebug (revised) [tools/scap] - 10https://gerrit.wikimedia.org/r/805763 (https://phabricator.wikimedia.org/T308476) (owner: 10Ahmon Dancy) [01:07:09] brennen: I encounter the same issue, both in a task creation, a comment, and a phame post. [01:07:34] normally, if you upload a file this way, it starts private but becomes public once you save/submit the form [01:07:38] regression in today's upgrade I guess? [01:08:15] oh and each permission grant counts as an edit that notifies IRC.. [01:45:03] Krinkle: Not a regression, that change is intended behavior. [01:46:23] If you drag a file to the edit window, it will attach the file with the edit. You otherwise need to attach it explicitly in other cases, [01:48:06] (The previous behavior of automatic attachment causes security issues) [03:05:59] it's a UX regression for sure [03:06:44] > If you drag a file to the edit window, it will attach the file with the edit [03:06:45] Not for me, needed to manually attach a file that I just uploaded by dragging into the edit window [03:10:40] https://secure.phabricator.com/T13683 is the upstream btw [03:12:31] for those who can see, https://phabricator.wikimedia.org/T310763#8007965 was me adding a comment that uploaded a file, and then I still had to manually attach [05:36:52] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @MarcoAurelio Hi! I've added you to #acl_Project-Admins - thanks! [07:48:00] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TheresNoTime - https://phabricator.wikimedia.org/T305191 (10TheresNoTime) Attended today's UTC early deployment training, trained by @ArielGlenn — I'll be attending future training sessions to get more experience with a wi... [08:06:06] 10Release-Engineering-Team (Radar), 10Scap, 10Patch-For-Review, 10User-jijiki: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10JMeybohm) Hey @dancy, do you think we're good to proceed with disabling revalidation for all of api,app,parsoid ? [09:01:43] 10GitLab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10Antoine_Quhen) 05Resolved→03Open Hi, it's still happening now: * https://gitlab.wikimedia.org/repos/data-engineering/conda-base-env/-/jobs/21111 `E: You don't have enough free space in /var/cach... [09:13:42] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10MarcoAurelio) >>! In T706#8008066, @Aklapper wrote: > @MarcoAurelio Hi! I've added you to #acl_Project-Admins - thanks! Thank you :-) [09:59:43] hi folks [09:59:46] I'd need to run scap install-world --batch [10:00:02] on deploy1002 as it was done yesterday to allow scap to operate on ml-cache100[2,3] [10:02:24] !log ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3] [10:02:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:58:57] (03CR) 10Awight: scap backport: deploy to mwdebug (revised) (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/805763 (https://phabricator.wikimedia.org/T308476) (owner: 10Ahmon Dancy) [12:11:29] wrt to file attachments, it probably make the most sense to just have edit forms automatically attach any files referenced which mostly restores the old UX but still fixes the second security issue mentioned in the upstream task. [12:16:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS, 10observability, 10cloud-services-team (Kanban): Replacement needed for obsolete Diamond/Graphite monitoring of integration instances - https://phabricator.wikimedia.org/T307655 (10hashar) That is affecting any Debian Bulls... [12:21:21] 10GitLab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10hashar) > ⚠ rerunning the job is not working currently. The issue is partly related to the jobs. If their builds take a lot of GigaBytes, specially in the cache they have a high chance of filing the r... [12:22:33] 10GitLab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10hashar) I have bring the topic to the Wednesday Gitlab sync meeting but this week it got used to upgrade Phabricator and we haven't had a chance to talk about it. The workaround is to manually prune t... [12:24:07] !log gitlab: runner-1026: `docker volume prune -f` [12:24:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:24:12] !log gitlab: runner-1030: `docker volume prune -f` [12:24:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:51:37] 10Beta-Cluster-Infrastructure, 10MediaWiki-Search, 10PageImages, 10Readers-Web-Backlog, and 3 others: PageImages ignores MediaWiki:Bad image list, (uses MediaWiki:Pageimages-denylist instead) displaying search results that are inappropriate for some readers - https://phabricator.wikimedia.org/T306246 (10ova... [12:56:23] 10GitLab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10hashar) Note to self, one can see the volumes size with: `docker system df -v`. On runner-1024 there are a few few 1 GB, 1.2G and 1.5G volumes piled up: ` runner-4kunvzhc-project-359-concurrent-0-cach... [13:00:00] 10Beta-Cluster-Infrastructure, 10MediaWiki-Search, 10PageImages, 10Readers-Web-Backlog, and 3 others: PageImages ignores MediaWiki:Bad image list, (uses MediaWiki:Pageimages-denylist instead) displaying search results that are inappropriate for some readers - https://phabricator.wikimedia.org/T306246 (10Ahe... [13:07:04] 10Release-Engineering-Team, 10Platform Engineering, 10Similar Editors, 10Anti-Harassment (AHaT Sprint 10: The Mokorotlo): Configure SimilarEditors in production with Similarusers credentials - https://phabricator.wikimedia.org/T308670 (10Tchanders) Now that PrivateSettings.php is updated, I believe all we... [13:41:09] 10Beta-Cluster-Infrastructure, 10MediaWiki-Search, 10PageImages, 10Readers-Web-Backlog, and 3 others: PageImages ignores MediaWiki:Bad image list, (uses MediaWiki:Pageimages-denylist instead) displaying search results that are inappropriate for some readers - https://phabricator.wikimedia.org/T306246 (10Ale... [14:35:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS, 10observability, 10cloud-services-team (Kanban): Replacement needed for obsolete Diamond/Graphite monitoring of integration instances - https://phabricator.wikimedia.org/T307655 (10hashar) 05Open→03Stalled @Majavah indi... [14:44:45] 10Release-Engineering-Team, 10Scap: scap-o-scap: Bootstrapping a new host fails - https://phabricator.wikimedia.org/T310740 (10elukey) To keep archives happy - even after running puppet on deploy1002 (to update the dsh target lists) I wasn't able to run puppet on the ml-cache nodes. We had to run `scap install... [14:59:29] (03CR) 10Ahmon Dancy: scap backport: deploy to mwdebug (revised) (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/805763 (https://phabricator.wikimedia.org/T308476) (owner: 10Ahmon Dancy) [15:00:48] (03PS1) 10Ahmon Dancy: add a missing word to a comment [tools/scap] - 10https://gerrit.wikimedia.org/r/806231 [15:01:07] (03CR) 10Ahmon Dancy: [C: 03+2] add a missing word to a comment [tools/scap] - 10https://gerrit.wikimedia.org/r/806231 (owner: 10Ahmon Dancy) [15:04:15] (03Merged) 10jenkins-bot: add a missing word to a comment [tools/scap] - 10https://gerrit.wikimedia.org/r/806231 (owner: 10Ahmon Dancy) [15:08:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS, 10observability, 10cloud-services-team (Kanban): Replacement needed for obsolete Diamond/Graphite monitoring of integration instances - https://phabricator.wikimedia.org/T307655 (10hashar) 05Stalled→03Resolved a:03hash... [15:21:37] (03PS1) 10Hashar: integration: link to Prometheus dashboard for agents [integration/docroot] - 10https://gerrit.wikimedia.org/r/806233 (https://phabricator.wikimedia.org/T307655) [15:26:43] (03CR) 10Ahmon Dancy: [C: 03+2] "Tested in train-dev." [tools/scap] - 10https://gerrit.wikimedia.org/r/805860 (owner: 10Hashar) [15:28:40] thanks DannyS712 ! [15:28:42] errr [15:28:44] thanks dancy [15:28:53] hehe [15:29:00] Thanks for the mods! [15:29:37] and I forgot to ask SRE about the ssh key pair generation in order for scap to push commits to gerrit [15:29:41] will poke them [15:29:42] https://phabricator.wikimedia.org/T310620 [15:29:47] (03Merged) 10jenkins-bot: scap prep: make Gerrit source URL configurable [tools/scap] - 10https://gerrit.wikimedia.org/r/805860 (owner: 10Hashar) [15:33:54] (03CR) 10Ahmon Dancy: [C: 04-1] Support a different user for Gerrit ssh interactions (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [15:38:46] (03PS8) 10Hashar: Support a different user for Gerrit ssh interactions [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) [15:38:50] (03CR) 10Hashar: Support a different user for Gerrit ssh interactions (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [15:39:01] I am clearly NOT paying attention [15:42:54] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10brennen) Blocking on T310747. [15:49:08] 10GitLab, 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team: Gitlab pipeline not working with "docker-registry.wikimedia.org/releng/" images - https://phabricator.wikimedia.org/T310718 (10sbassett) Hey #release-engineering-team - guessing the `releng` path might need to be explicitly allow-l... [15:51:03] Krinkle: thanks for the log triage runbook :] https://wikitech.wikimedia.org/wiki/Performance/Runbook/Monitor_production_errors [15:57:55] 10GitLab, 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team: Gitlab pipeline not working with "docker-registry.wikimedia.org/releng/" images - https://phabricator.wikimedia.org/T310718 (10brennen) Bit slammed at the moment but this probably dupes {T310535}. [16:00:20] 10GitLab, 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team: Gitlab pipeline not working with "docker-registry.wikimedia.org/releng/" images - https://phabricator.wikimedia.org/T310718 (10sbassett) [16:00:55] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Patch-For-Review, 10User-brennen: GitLab runners: allowed_images patterns need to be loosened to include subdirectories - https://phabricator.wikimedia.org/T310535 (10sbassett) [16:01:01] (03CR) 10Krinkle: [C: 03+2] integration: link to Prometheus dashboard for agents [integration/docroot] - 10https://gerrit.wikimedia.org/r/806233 (https://phabricator.wikimedia.org/T307655) (owner: 10Hashar) [16:01:15] 10GitLab, 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team: Gitlab pipeline not working with "docker-registry.wikimedia.org/releng/" images - https://phabricator.wikimedia.org/T310718 (10sbassett) >>! In T310718#8009379, @brennen wrote: > Bit slammed at the moment but this probably dupes {T... [16:02:18] (03Merged) 10jenkins-bot: integration: link to Prometheus dashboard for agents [integration/docroot] - 10https://gerrit.wikimedia.org/r/806233 (https://phabricator.wikimedia.org/T307655) (owner: 10Hashar) [16:22:51] 10GitLab, 10Data-Engineering: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10JAllemandou) [16:44:04] 10Gerrit, 10Data-Engineering: Remove unused Gerrit repository mediawiki/services/aqs/deploy - https://phabricator.wikimedia.org/T309731 (10Milimetric) 05Open→03Resolved a:03Milimetric Sorry, Andre, I didn't even know there was a Gerrit tag. I'm marking this as resolved for now. If we ever come up with... [16:54:45] 10Release-Engineering-Team, 10Platform Engineering, 10Similar Editors, 10Anti-Harassment (AHaT Sprint 10: The Mokorotlo): Configure SimilarEditors in production with Similarusers credentials - https://phabricator.wikimedia.org/T308670 (10Niharika) @Tchanders is there a list of users who will have access to... [17:04:30] (03Abandoned) 10C. Scott Ananian: Add 'parsoid' dependency to 'ImageMap', remove 'ImageMap' dep from Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/745930 (https://phabricator.wikimedia.org/T94793) (owner: 10C. Scott Ananian) [17:56:44] 10Release-Engineering-Team, 10Scap: scap-o-scap: Bootstrapping a new host fails - https://phabricator.wikimedia.org/T310740 (10Volans) @dancy was asking on IRC so I had a quick look. From what I can tell: * `ml-cache1002` was added to `/etc/dsh/group/scap_targets` by a puppet run at 09:45:11 UTC * this trigge... [17:58:41] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10brennen) [18:14:40] 10Phabricator, 10Release-Engineering-Team, 10Regression: Editing tasks results in "You cannot add more than 0 objects to the relationship" error - https://phabricator.wikimedia.org/T310742 (10brennen) 05Open→03Resolved a:03brennen [18:18:38] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Patch-For-Review, 10User-brennen: GitLab runners: allowed_images patterns need to be loosened to include subdirectories - https://phabricator.wikimedia.org/T310535 (10brennen) @sbassett: With that patch merged, this should take effect once runners... [18:19:20] 10GitLab, 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team: Gitlab pipeline not working with "docker-registry.wikimedia.org/releng/" images - https://phabricator.wikimedia.org/T310718 (10brennen) From other task: > @sbassett: With that patch merged, this should take effect once runners are... [18:19:44] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Patch-For-Review, 10User-brennen: GitLab runners: allowed_images patterns need to be loosened to include subdirectories - https://phabricator.wikimedia.org/T310535 (10sbassett) >>! In T310535#8009874, @brennen wrote: > @sbassett: With that patch me... [18:36:04] 10Release-Engineering-Team, 10Scap: scap-o-scap: Bootstrapping a new host fails - https://phabricator.wikimedia.org/T310740 (10jbond) @Volans pretty sure T303559 is the relevant task and [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/771441 | change ]] (disclaimer the CR is from me but don't take that... [18:44:00] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10brennen) [18:51:30] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) [18:53:26] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: `gitlab-runner1001.eqiad.wmnet` - gitlab-runner1001.eqiad.w... [19:01:02] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: `gitlab-runner1001.eqiad.wmnet` - gitlab-runner1001.eqiad.w... [19:39:45] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by aokoth@cumin1001 for hosts: `gitlab-runner2001.codfw.wmnet` - gitlab-runner2001.codfw.... [19:46:01] 10Phabricator: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10Quiddity) [19:59:38] 10Phabricator, 10Release-Engineering-Team: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10thcipriani) Thanks for the report! [20:01:58] 10Phabricator, 10Release-Engineering-Team: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10Quiddity) It looks like there's a 3-step workflow for making attached images public (if we used drag-and-drop to upload). I will test it on these 3 fi... [20:02:05] 10Phabricator, 10Release-Engineering-Team: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10Quiddity) [20:02:14] 10Phabricator, 10Release-Engineering-Team: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10Quiddity) [20:02:21] 10Phabricator, 10Release-Engineering-Team: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10Quiddity) [20:03:13] 10Phabricator, 10Release-Engineering-Team: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10Quiddity) [20:05:03] 10Phabricator, 10Release-Engineering-Team: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10TheresNoTime) I'll put some money on [[ https://github.com/wikimedia/phabricator/commit/7693a711a725ea1c28743b240468c411506acc6f#diff-767d7ed103d79f543... [20:09:00] 10Phabricator, 10Release-Engineering-Team: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10alexhollender_WMF) >However if we use the "upload file" form, then things work as expected. for me this issue is also present when I use the "upload f... [20:26:32] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10brennen) a:03brennen See: - Upstream notes: https://secure.phabricator.com/T13683 - Upstream [[https:... [20:33:17] 10Release-Engineering-Team, 10Scap: Scap pool counter error while backporting - https://phabricator.wikimedia.org/T310835 (10cjming) [20:35:58] 10Release-Engineering-Team, 10Scap: Scap pool counter error while backporting - https://phabricator.wikimedia.org/T310835 (10thcipriani) [20:36:06] 10Release-Engineering-Team, 10Scap: scap-o-scap: Bootstrapping a new host fails - https://phabricator.wikimedia.org/T310740 (10dancy) [20:36:10] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10dancy) [20:38:21] 10Deployments, 10Release-Engineering-Team (Deployment Autopilot 🛩️): Scap sync-file fails with error about pause_after_testserver_sync - https://phabricator.wikimedia.org/T310699 (10dancy) 05Open→03Resolved a:03dancy [20:39:18] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10brennen) 05Open→03Resolved Stable at all wikis for a couple of hours. Optimistically resolving. [20:53:54] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10Dzahn) https://gerrit.wikimedia.org/r/c/operations/puppet/+/806250 [20:54:02] Is there a reason we are using "merge if necessary" and not "rebase if necessary" in mediawiki repos? The jenkins merge commits make the commit log very cloudy and imo harder to follow. [20:59:40] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Seen), 10Scap, 10SRE, 10serviceops: Scap can't clear opcache on mw servers in Beta Cluster - https://phabricator.wikimedia.org/T237033 (10Krinkle) [21:42:46] hey guys. look at this: [21:42:53] sudo cumin 'gitlab-runner*' "systemctl status buildkitd | grep Active | cut -d: -f1" [21:42:59] (6) gitlab-runner[2002-2004].codfw.wmnet,gitlab-runner[1002-1004].eqiad.wmnet [21:43:04] Active [21:43:16] buildkitd running on all gitlab-runners justnow [21:44:15] It needed 2 manual steps. follow-up patches for puppet will be desired. But just done for now. [21:50:14] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Patch-For-Review, 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10Dzahn) buildkitd is now running on all (6) gitlab-runners. It's 6 because the VMs 1001 and 2001 have been... [21:55:46] \o/ [22:00:02] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Patch-For-Review, 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dancy) Thank you very much @Dzahn ! [22:01:15] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.39.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T308073 (10thcipriani) 05Open→03Declined Declining for WMF Inspiration week. [22:08:48] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Patch-For-Review, 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10Dzahn) p:05High→03Medium It's deployed but we have some follow-ups. I guess lowering the prio a bit is... [22:18:45] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TheresNoTime - https://phabricator.wikimedia.org/T305191 (10TheresNoTime) @thcipriani I mentioned this on IRC, but just logging here too 🙂 @ArielGlenn suggested that in the next training session it would be good for me to... [22:21:03] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TheresNoTime - https://phabricator.wikimedia.org/T305191 (10thcipriani) >>! In T305191#8010455, @TheresNoTime wrote: > @thcipriani I mentioned this on IRC, but just logging here too 🙂 @ArielGlenn suggested that in the next... [23:35:04] (03PS1) 10Jeena Huneidi: scap backport: add revert option [tools/scap] - 10https://gerrit.wikimedia.org/r/806352 (https://phabricator.wikimedia.org/T287046)