[06:56:22] 10Continuous-Integration-Infrastructure, 10EntitySchema, 10Wikidata, 10wmde-wikidata-tech, 10Wikidata Dev Team (Wikidata.org Slice): EntitySchema CI broken on master with OutOfMemory error in browser test - https://phabricator.wikimedia.org/T354179#9686937 (10ArthurTaylor) a:05ArthurTaylorβ†’03None [08:25:40] 10Release-Engineering-Team (Onboarding πŸš€), 10Scap, 07git-lfs: 14scap git LFS only works for submodules - 14https://phabricator.wikimedia.org/T324664#9687220 (10hashar) [08:25:40] 10Gerrit, 06Release-Engineering-Team, 07git-lfs: 14Migrate Gerrit deployment from git-fat to git-lfs - 14https://phabricator.wikimedia.org/T333465#9687221 (10hashar) [08:25:42] 10Gerrit, 10Release-Engineering-Team (Radar), 06Machine-Learning-Team, 07git-lfs: ML-Team will soon stop using LFS on Gerrit (for ORES deployment) - https://phabricator.wikimedia.org/T342765#9687219 (10hashar) [08:25:44] 10Gerrit, 07git-lfs: 14Make sure we backup Gerrit's LFS data - 14https://phabricator.wikimedia.org/T254155#9687223 (10hashar) [08:25:45] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap, 06Discovery-Search, 07git-lfs: 14wikimedia/discovery/analytics: replace git-fat with git-lfs - 14https://phabricator.wikimedia.org/T316877#9687222 (10hashar) [08:25:52] 10Gerrit, 06Release-Engineering-Team, 10Wikimedia-GitHub, 07git-lfs: 14Enable git lfs fetch for gerrit mirrors - 14https://phabricator.wikimedia.org/T222955#9687225 (10hashar) [08:26:10] 10Gerrit, 10ORES, 07git-lfs: 14Write a cookbook for the workaround for getting LFS to gerrit - 14https://phabricator.wikimedia.org/T226055#9687231 (10hashar) [08:26:14] 14Release-Engineering-Team (Kanban), 10Scap, 07git-lfs, 13Patch-For-Review: 14Support git-lfs in scap - 14https://phabricator.wikimedia.org/T180627#9687232 (10hashar) [08:26:24] 10Gerrit, 06collaboration-services, 07git-lfs: 14Gerrit LFS objects lack an automatic sync to gerrit replicas - 14https://phabricator.wikimedia.org/T257741#9687235 (10hashar) [08:26:32] 10Release-Engineering-Team (Now this 🫠), 10Scap, 06Data-Engineering, 06Data-Platform-SRE, 07git-lfs: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#9687234 (10hashar) [08:26:38] 10Gerrit, 06SRE, 07git-lfs: 14Initial backup run for Gerrit LFS data - 14https://phabricator.wikimedia.org/T254162#9687236 (10hashar) [08:26:49] 10Diffusion, 10Release-Engineering-Team (Seen), 06Machine-Learning-Team, 10Wikimedia-GitHub, 07git-lfs: 14LFS objects are not mirroring from Github through Phab to Gerrit consistently - 14https://phabricator.wikimedia.org/T212818#9687241 (10hashar) [08:26:55] 10Gerrit, 06Release-Engineering-Team, 06SRE, 07git-lfs: 14Automatic pickup of Gerrit clone master doesn't happen due to missing git-lfs – new deployment env - 14https://phabricator.wikimedia.org/T235677#9687239 (10hashar) [08:27:03] 10Gerrit, 14Release-Engineering-Team (Development services), 14Release-Engineering-Team-TODO (201908), 10Wikidata, and 2 others: 14Make "analytics/wmde/toolkit-analyzer-build" use git lfs - 14https://phabricator.wikimedia.org/T230015#9687238 (10hashar) [08:27:25] 10Phabricator, 10Release-Engineering-Team (Seen), 10SRE-swift-storage, 07git-lfs: 14Connect Phabricator to swift for storage of git-lfs and file uploads. - 14https://phabricator.wikimedia.org/T182085#9687244 (10hashar) [08:27:33] 14Release-Engineering-Team (Next), 10Scap, 06Machine-Learning-Team, 10ORES, and 2 others: 14scap support for git-lfs - 14https://phabricator.wikimedia.org/T181855#9687245 (10hashar) [08:27:41] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap, 06Data-Platform-SRE, 10Wikidata, and 2 others: wdqs: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316876#9687250 (10hashar) [08:27:49] 10Gerrit, 10ORES, 06SRE, 07git-lfs, 13Patch-For-Review: 14Plan migration of ORES repos to git-lfs - 14https://phabricator.wikimedia.org/T181678#9687247 (10hashar) [08:28:21] 10Release-Engineering-Team (Onboarding πŸš€), 10Scap, 07git-lfs: 14scap git LFS only works for submodules - 14https://phabricator.wikimedia.org/T324664#9687258 (10hashar) 14See also https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/226 //deploy: Fix git-lfs support (submodules)// [08:28:25] 10Release-Engineering-Team (Seen), 10Wikimedia Design Style Guide, 07git-lfs: 14Use `git lfs` for large binary files of Design Style Guide - 14https://phabricator.wikimedia.org/T235013#9687260 (10hashar) [08:28:31] well I guess that was a little bit spammy :/ [08:28:51] 10Scap, 06Infrastructure-Foundations, 10Packaging, 06SRE, and 2 others: 14Install git-lfs client (at least on scap targets & masters) - 14https://phabricator.wikimedia.org/T180628#9687246 (10hashar) [08:54:09] 06Project-Admins, 10MediaWiki-libs-UpdateHistory: 14Create project tag for wikimedia/update-history library - 14https://phabricator.wikimedia.org/T361723#9687363 (10Peachey88) 05Openβ†’03Resolved a:03Peachey88 14Requested project has been created :) [09:03:27] 10Phabricator, 10Release-Engineering-Team (Seen), 10SRE-swift-storage, 07git-lfs: 14Connect Phabricator to swift for storage of git-lfs and file uploads. - 14https://phabricator.wikimedia.org/T182085#9687425 (10MatthewVernon) 14there was maybe a suggestion of using it for files uploaded to phab? [09:25:52] (03PS1) 10Hashar: dockerfiles: jar-updater remove git fat and update OS [integration/config] - 10https://gerrit.wikimedia.org/r/1017027 (https://phabricator.wikimedia.org/T328472) [09:28:25] (03CR) 10Hashar: [C:03+2] dockerfiles: jar-updater remove git fat and update OS [integration/config] - 10https://gerrit.wikimedia.org/r/1017027 (https://phabricator.wikimedia.org/T328472) (owner: 10Hashar) [09:29:50] (03Merged) 10jenkins-bot: dockerfiles: jar-updater remove git fat and update OS [integration/config] - 10https://gerrit.wikimedia.org/r/1017027 (https://phabricator.wikimedia.org/T328472) (owner: 10Hashar) [10:06:43] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 06serviceops-radar: Helm deployment of MediaWiki now takes 6 minutes - https://phabricator.wikimedia.org/T360403#9687736 (10akosiaris) Moving it to our radar too as we intend to revisit various parts of all of this (e.g. how we do MultiVersion once we are no l... [12:46:27] Project beta-scap-sync-world build #149109: 04FAILURE in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/149109/ [12:56:24] Yippee, build fixed! [12:56:24] Project beta-scap-sync-world build #149110: 09FIXED in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/149110/ [13:05:42] hashar: Meeting? Or skip? [13:08:16] (03PS1) 10Hashar: dockerfiles: set git remote in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1017051 (https://phabricator.wikimedia.org/T328472) [13:08:47] (03PS1) 10Hashar: jjb: update-jars no more need to set the git remote [integration/config] - 10https://gerrit.wikimedia.org/r/1017053 (https://phabricator.wikimedia.org/T328472) [13:10:21] (03PS2) 10Hashar: jjb: update-jars no more need to set the git remote [integration/config] - 10https://gerrit.wikimedia.org/r/1017053 (https://phabricator.wikimedia.org/T328472) [13:10:43] 10Release-Engineering-Team (Now this 🫠), 10Scap, 06Data-Engineering, 06Data-Platform-SRE, and 2 others: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#9688404 (10hashar) @Sfaci mentioned the [[ https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docke... [13:48:29] hashar: Are you not going to review and deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1016827 ? [13:48:44] nop [13:48:49] :-( [13:48:58] I can do it over the weekend when it's quiet. [13:49:15] I don't know what the hell it is going to break :D [13:49:23] Yeah. [13:49:50] that is why I am currently reviewing it [13:50:46] I'd also like to get rid of Docker [13:50:48] and docker-pkg [13:50:53] and pretend that stack never existed [13:50:54] :D [13:51:35] there is some oddity as well which is that the central cache will end up being useless [13:51:40] since it is namespaced by the job name [13:51:55] so yeah that is not as easy as just renaming the jobs [13:52:04] or I'd done it :] [13:52:38] It's easy if we also delete all the cache entries over a weekend. [13:52:47] Or they'll expire themselves anyway. [14:23:02] Hi! There is this ticket related to scap and RESTBase: https://phabricator.wikimedia.org/T361608 [14:24:02] Can somebody help us to unblock deployments? [14:24:44] jnuche or I yeah though I am gone in some minutes for an appointment :/ [14:27:00] James_F: +1 ed [14:27:26] but the cache invalidation will definitely cause all jobs to be slower than expected / timeout / lead to some issues here and there [14:27:30] :) [14:27:46] nemo-yiannis: yes that one is easy [14:27:49] the tag got renamed [14:27:53] that is forbidden :) [14:28:30] or to be more precise, recentish git versions refuse to update a tag unless forced [14:28:59] I think this happened after running scap with limits to some of the hosts [14:29:30] the next deployment to all the targets failed [14:31:22] hmm [14:31:32] dancy supposedly fixed it in scap :/ [14:33:15] ! [rejected] scap/sync/2024-02-19/0011 -> scap/sync/2024-02-19/0011 (would clobber existing tag) [14:33:26] yeah so that tag is different between the deployment server and the targets [14:34:06] and my guess is the easiest is to delete the tag from all the targets [14:34:49] or at least on the affected one [14:35:01] nemo-yiannis: probably it won't work, but have you tried passing `-f` to `scap deploy`? [14:35:12] probably hashar is right though and the tags need to be fixed manually [14:35:23] yeah definitely [14:36:16] on restbase2030.codfw.wmnet in `/srv/deployment/restbase/deploy-cache/cache` you would need to `git fetch --tags -f` , using `sudo` as what ever user owns the directory (restbase-deploy?) [14:36:24] based on https://phabricator.wikimedia.org/T311336#8038925 [14:36:37] basically you need to force fetch the tags [14:37:00] (or alternatively, maybe the faulty tag can be deleted from the deployment server ) [14:37:11] it is probably not a big deal if we loose that ref, but that is not ideal) [14:38:49] Good morning folks! [14:38:57] Deleting the offending tag from the deploy server sounds like a good choice. [14:39:37] nemo-yiannis: ^^ [14:40:45] ok i can try that [14:40:46] I have summarized on the task [14:40:56] and a huge thanks to Ahmon for fixing that bug :] [14:41:10] Not in time, apparently! [14:41:18] thanks folks [14:41:26] I am off for the appointment! [14:41:32] See ya! [14:43:03] ciao πŸ‘‹ [14:44:42] (03CR) 10Hashar: [C:03+1] "I have looked at the diff using git to benefit from its renaming detection algorithm. Something such as:" [integration/config] - 10https://gerrit.wikimedia.org/r/1016827 (https://phabricator.wikimedia.org/T360327) (owner: 10Jforrester) [14:46:51] 06Release-Engineering-Team, 10Scap, 10RESTBase: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T361608#9688801 (10hashar) Recentish git versions refuse to update a tag unless forced. That is supposedly fixed by e98df2095d16c1cba159f425f77c263b243f1a7e / T311336 :( [14:47:27] 06Release-Engineering-Team, 10Scap, 10RESTBase: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T361608#9688835 (10hashar) [14:49:07] 06Release-Engineering-Team, 10Scap, 10RESTBase: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T361608#9688902 (10hashar) The faulty tag has the date `2024-02-19` and the fix in scap was written in Feb 23 and I guess deployed not so long after but definitely AFTER the faulty tag. Once t... [14:55:43] (03CR) 10Ahmon Dancy: [C:03+2] dockerfiles: set git remote in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1017051 (https://phabricator.wikimedia.org/T328472) (owner: 10Hashar) [14:57:02] (03Merged) 10jenkins-bot: dockerfiles: set git remote in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1017051 (https://phabricator.wikimedia.org/T328472) (owner: 10Hashar) [14:59:47] !log Updating docker-pkg files on contint primary for T328472 [14:59:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:59:50] T328472: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472 [15:02:28] (03CR) 10Ahmon Dancy: [C:03+2] jjb: update-jars no more need to set the git remote [integration/config] - 10https://gerrit.wikimedia.org/r/1017053 (https://phabricator.wikimedia.org/T328472) (owner: 10Hashar) [15:03:45] (03Merged) 10jenkins-bot: jjb: update-jars no more need to set the git remote [integration/config] - 10https://gerrit.wikimedia.org/r/1017053 (https://phabricator.wikimedia.org/T328472) (owner: 10Hashar) [15:13:22] 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 06MW-Interfaces-Team, 07Code-Health: 14Enable API integration tests in CI for MediaWiki core - 14https://phabricator.wikimedia.org/T236680#9689046 (10daniel) 05Openβ†’03Resolved a:03daniel 14Nobody has been asking for more docume... [15:31:38] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: 14Add GitLab upgrades and maintenance to deployment calendar - 14https://phabricator.wikimedia.org/T336470#9689121 (10CodeReviewBot) 14thcipriani merged https://gitlab.wikimedia.org/repos/relen... [15:31:54] 06Release-Engineering-Team, 10Scap, 06serviceops, 13Patch-For-Review, 10Sustainability (Incident Followup): scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#9689119 (10CodeReviewBot) jiji opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requ... [15:36:37] (03PS1) 10Ahmon Dancy: dockerfiles/ci-src-setup-simple: Update to 0.6.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1017098 (https://phabricator.wikimedia.org/T328472) [15:38:07] (03CR) 10CI reject: [V:04-1] dockerfiles/ci-src-setup-simple: Update to 0.6.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1017098 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [15:52:23] (03CR) 10Ahmon Dancy: [C:03+2] dockerfiles/ci-src-setup-simple: Update to 0.6.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1017098 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [15:53:17] (03Merged) 10jenkins-bot: dockerfiles/ci-src-setup-simple: Update to 0.6.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1017098 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [15:55:43] (03PS1) 10Ahmon Dancy: jjb: Bump ci-src-setup-simple from 0.6.0 to 0.6.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1017100 (https://phabricator.wikimedia.org/T328472) [15:57:31] (03CR) 10Ahmon Dancy: [C:03+2] jjb: Bump ci-src-setup-simple from 0.6.0 to 0.6.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1017100 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [15:58:51] (03Merged) 10jenkins-bot: jjb: Bump ci-src-setup-simple from 0.6.0 to 0.6.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1017100 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [16:00:47] !log Updating docker-pkg files on contint primary for T328472 [16:00:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:00:58] T328472: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472 [16:46:20] 10Diffusion, 10Toolforge, 10wikitech.wikimedia.org, 07Documentation: Document diffusion->github mirroring to https://github.com/toolforge/ on wikitech - https://phabricator.wikimedia.org/T361859 (10bd808) 03NEW [16:55:59] 10Beta-Cluster-Infrastructure, 06MediaWiki-Platform-Team, 10MW-1.42-notes (1.42.0-wmf.25; 2024-04-02), 13Patch-For-Review: Cannot create a new wiki on beta cluster - https://phabricator.wikimedia.org/T358236#9689604 (10pmiazga) The `addWiki.php` script execution was finally successful. [17:00:00] (03PS1) 10Ahmon Dancy: dockerfiles: Bump ci-buster and ci-src-setup-simple [integration/config] - 10https://gerrit.wikimedia.org/r/1017103 (https://phabricator.wikimedia.org/T328472) [17:01:55] (03CR) 10Ahmon Dancy: [C:03+2] dockerfiles: Bump ci-buster and ci-src-setup-simple [integration/config] - 10https://gerrit.wikimedia.org/r/1017103 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [17:02:45] !log deployment-prep T355281 executed β€œmwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki --skipclusters=main,echo,growth,mediamoderation,extstore en wikipedia test2wiki test2.wikipedia.beta.wmcloud.org” on deployment-deploy03. [17:02:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:02:48] T355281: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281 [17:03:50] (03Merged) 10jenkins-bot: dockerfiles: Bump ci-buster and ci-src-setup-simple [integration/config] - 10https://gerrit.wikimedia.org/r/1017103 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [17:06:09] 10Beta-Cluster-Infrastructure, 06MediaWiki-Platform-Team, 10MW-1.42-notes (1.42.0-wmf.26; 2024-04-09), 13Patch-For-Review: Cannot create a new wiki on beta cluster - https://phabricator.wikimedia.org/T358236#9689646 (10pmiazga) >>! In T358236#9674176, @Ladsgroup wrote: > Let me know once you can create a n... [17:07:31] (03PS1) 10Ahmon Dancy: jjb: Bump ci-src-setup-simple from 0.6.1 to 0.6.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1017104 (https://phabricator.wikimedia.org/T328472) [17:09:10] (03CR) 10Ahmon Dancy: [C:03+2] jjb: Bump ci-src-setup-simple from 0.6.1 to 0.6.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1017104 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [17:11:02] (03Merged) 10jenkins-bot: jjb: Bump ci-src-setup-simple from 0.6.1 to 0.6.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1017104 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [17:19:52] 10Diffusion, 10Toolforge, 10wikitech.wikimedia.org, 07Documentation: Document diffusion->github mirroring to https://github.com/toolforge/ on wikitech - https://phabricator.wikimedia.org/T361859#9689712 (10bd808) The mystery of who poked the account and triggered the first email has been solved. Nothing ne... [17:25:31] (03PS1) 10Ahmon Dancy: dockerfiles: jar-updater 0.1.2: Include curl package [integration/config] - 10https://gerrit.wikimedia.org/r/1017108 (https://phabricator.wikimedia.org/T328472) [17:26:33] (03PS2) 10Ahmon Dancy: dockerfiles: jar-updater 0.1.2: Include curl package [integration/config] - 10https://gerrit.wikimedia.org/r/1017108 (https://phabricator.wikimedia.org/T328472) [17:28:47] (03CR) 10Ahmon Dancy: [C:03+2] dockerfiles: jar-updater 0.1.2: Include curl package [integration/config] - 10https://gerrit.wikimedia.org/r/1017108 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [17:30:42] 10Diffusion, 10Phabricator, 14Release-Engineering-Team (Social Piranhas 🐟): 14Understand which repositories we mirror, observe, host in Diffusion (and fix some findings) - 14https://phabricator.wikimedia.org/T347577#9689792 (10Aklapper) 14FYI: A list of active repos mirrored from Diffusion into GitHub.... [17:30:44] (03Merged) 10jenkins-bot: dockerfiles: jar-updater 0.1.2: Include curl package [integration/config] - 10https://gerrit.wikimedia.org/r/1017108 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [17:31:36] 10Diffusion, 10Toolforge, 10wikitech.wikimedia.org, 07Documentation: Document diffusion->github mirroring to https://github.com/toolforge/ on wikitech - https://phabricator.wikimedia.org/T361859#9689794 (10Aklapper) I might misunderstand the task scope but see also T347577#9689792 [17:35:37] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Now this 🫠), 06collaboration-services: Get GitLab to render `T{\d}+` in MR overviews, comments, etc. as links to Phabricator - https://phabricator.wikimedia.org/T337570#9689816 (10brennen) 05In progressβ†’03Stalled The upstream integrati... [17:44:13] 10GitLab-Application-Security-Pipeline, 13Patch-For-Review, 07Security: Design AppSec Pipeline metrics approach - https://phabricator.wikimedia.org/T342467#9689854 (10CodeReviewBot) mmartorana merged https://gitlab.wikimedia.org/repos/security/gitlab-ci-security-templates-stats/-/merge_requests/1 Initial cr... [17:50:28] (03PS1) 10Ahmon Dancy: jjb: analytics.yaml: Bump to docker-registry.wikimedia.org/releng/jar-updater:0.1.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1017112 (https://phabricator.wikimedia.org/T328472) [17:52:05] (03CR) 10Ahmon Dancy: [C:03+2] jjb: analytics.yaml: Bump to docker-registry.wikimedia.org/releng/jar-updater:0.1.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1017112 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [17:53:49] (03Merged) 10jenkins-bot: jjb: analytics.yaml: Bump to docker-registry.wikimedia.org/releng/jar-updater:0.1.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1017112 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [17:53:49] 10GitLab-Application-Security-Pipeline, 07SecTeam-Processed, 07Security: 14Design AppSec Pipeline metrics approach - 14https://phabricator.wikimedia.org/T342467#9689888 (10sbassett) 05In progressβ†’03Resolved 14I'd like to add more tests, but the basic cli is done now. [17:53:57] 10GitLab-Application-Security-Pipeline, 07SecTeam-Processed, 07Security: 14Design AppSec Pipeline metrics approach - 14https://phabricator.wikimedia.org/T342467#9689893 (10sbassett) [17:57:25] dancy: :) [17:57:38] that lfs config bug is a nice one [17:57:46] I am happy to see that leads to some cleanup [18:00:20] (03CR) 10Hashar: "My bad sorry :( I went too fast on that one." [integration/config] - 10https://gerrit.wikimedia.org/r/1017108 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [18:01:53] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Now this 🫠), 06collaboration-services: Get GitLab to render `T{\d}+` in MR overviews, comments, etc. as links to Phabricator - https://phabricator.wikimedia.org/T337570#9689923 (10brennen) Also noting after some slack discussion with @bd80... [18:08:44] 10Release-Engineering-Team (Now this 🫠), 10Scap, 06Data-Engineering, 06Data-Platform-SRE, and 2 others: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#9689959 (10dancy) @Sfaci The issue with the `analytics-refinery-update-jars-docker` job should be resolved now. https:/... [18:10:11] hashar: It took quite a while to take care of. I've been working on it all morning. [18:15:38] dancy: I hear you [18:15:46] I am not sure why you rebuild the buster image though [18:15:57] I have rebuild the jar updater image on top of bookworm [18:16:12] anyway it is not important [18:16:45] it looks like you got a successful build! [18:18:53] hashar: I don't think I really needed to change ci-buster. I was fumbling around because https://gerrit.wikimedia.org/r/c/integration/config/+/1017051 didn't update the ci-src-setup-simple image. [18:19:16] OHof course [18:19:19] Hand-optimization strikes again [18:19:29] yeah sorry I did bunch of things in parallel this afternoon [18:19:48] including digressing about replacing docker/docker-pkg with something that is more convenient to merge layers [18:19:49] :D [18:20:11] looks like you got it fixed and all passing \o/ [18:21:20] dancy: https://phabricator.wikimedia.org/T328472#9689959 :D [18:21:26] 10Release-Engineering-Team (Now this 🫠), 10Scap, 06Data-Engineering, 06Data-Platform-SRE, and 2 others: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#9690038 (10hashar) antoine-approve [18:22:06] hopefully we will get rid of that job entirely one day [18:22:52] * hashar dinnes [18:23:09] or whatever is the proper term for saying I am having the evening food thing [18:29:15] 06Release-Engineering-Team, 10Scap, 06serviceops, 07Wikimedia-Incident: Helm was left in limbo due to interrupted deployment/rollback - https://phabricator.wikimedia.org/T361720#9690071 (10dancy) [18:29:40] 06Release-Engineering-Team, 10Scap, 06serviceops, 07Wikimedia-Incident: Helm was left in limbo due to interrupted deployment/rollback - https://phabricator.wikimedia.org/T361720#9690063 (10dancy) 05Resolvedβ†’03Open Scap already has a check for helm releases in `pending-upgrade` state but it looks like i... [18:34:20] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07Performance Issue: Submitting actions taking ~10 seconds to load after making changes - https://phabricator.wikimedia.org/T360484#9690115 (10Aklapper) > Getting the payload/response for that request in a [private paste](https://phabricator.wik... [18:58:14] 06Release-Engineering-Team, 10Scap, 06serviceops, 13Patch-For-Review, 10Sustainability (Incident Followup): scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#9690204 (10dancy) There is a tmux/screen check for `scap stage-train`, but nothing else. This c... [19:43:18] dancy: do you know how scap is deployed on the beta cluster? My guess is one has to manually deploy it, but maybe there is a timer doing it constantly for us :) [19:45:18] It's manual. Run release-scripts/update-scap-in-beta [19:45:20] looks like that is manual, so I am not going to do it myself or I am sure something will explode [19:45:24] ah [19:45:57] that is a convenient script [19:47:10] I might try that tomorrow morning [20:04:54] 06Release-Engineering-Team, 10Scap, 06serviceops, 13Patch-For-Review, 10Sustainability (Incident Followup): scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#9690603 (10hashar) I might have used `screen` back in the old days (like in 2005 or so) and might... [20:19:22] 06Release-Engineering-Team, 10Scap, 06serviceops, 07Wikimedia-Incident: Helm was left in limbo due to interrupted deployment/rollback - https://phabricator.wikimedia.org/T361720#9690655 (10dancy) p:05Unbreak!β†’03Medium [20:23:32] 10Release-Engineering-Team (Now this 🫠), 10Scap, 06Data-Engineering, 06Data-Platform-SRE, and 2 others: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#9690668 (10Sfaci) The job finished great! I can keep working on the deployment that was blocked for this. Thank you very... [20:41:48] 06Release-Engineering-Team, 10Scap, 06serviceops, 13Patch-For-Review, 07Wikimedia-Incident: Helm was left in limbo due to interrupted deployment/rollback - https://phabricator.wikimedia.org/T361720#9690727 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merg... [20:50:26] (03CR) 10C. Scott Ananian: [C:03+1] Remove Parsoid dependency on Poem [integration/config] - 10https://gerrit.wikimedia.org/r/1006605 (https://phabricator.wikimedia.org/T358054) (owner: 10Arlolra) [20:59:58] 06Release-Engineering-Team, 10Scap, 06serviceops, 13Patch-For-Review, 07Wikimedia-Incident: Helm was left in limbo due to interrupted deployment/rollback - https://phabricator.wikimedia.org/T361720#9690805 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/267... [21:01:35] 06Release-Engineering-Team, 10Scap, 06serviceops, 13Patch-For-Review, 07Wikimedia-Incident: Helm was left in limbo due to interrupted deployment/rollback - https://phabricator.wikimedia.org/T361720#9690811 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merg...