[00:27:55] 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Where is my change running tool - https://phabricator.wikimedia.org/T434726#12210276 (10bd808) [00:50:41] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎), 10Cloud-VPS (Debian Bullseye Deprecation): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12210303 (10bd808) Update: Southparkfan won't be available for a bit, but he and I plan to sync up in t... [01:19:12] (03Abandoned) 10Pwangai: castor: Log load failures and rsync stats [integration/config] - 10https://gerrit.wikimedia.org/r/1321038 (https://phabricator.wikimedia.org/T433915) (owner: 10Pwangai) [02:09:01] (03PS3) 10Pwangai: castor: measure cache-load rsync [integration/config] - 10https://gerrit.wikimedia.org/r/1313076 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog) [02:32:05] (03PS2) 10Pwangai: castor: Add rsync stats and timing to the cache save [integration/config] - 10https://gerrit.wikimedia.org/r/1321049 (https://phabricator.wikimedia.org/T432685) [02:38:14] (03PS3) 10Pwangai: castor: Add rsync stats and timing to the cache save [integration/config] - 10https://gerrit.wikimedia.org/r/1321049 (https://phabricator.wikimedia.org/T432685) [02:51:56] (03open) 10tstarling: make-release: Add Produnto to production extensions [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/258 (https://phabricator.wikimedia.org/T421436) [03:05:10] !log created Produnto tables on beta T421436 [03:05:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:05:13] T421436: Deploy Produnto extension to production - https://phabricator.wikimedia.org/T421436 [05:44:42] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12210581 (10ABran-WMF) [05:45:20] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12210582 (10ABran-WMF) >>! In T425441#12206548, @ABran-WMF wrote: > gitlab-replica-b is behind CDN and behaving as intended. same for replica-a [08:22:26] hashar: if you happen to have time today to take a look at https://gerrit.wikimedia.org/r/c/integration/config/+/1324699, i'd appreciate it - i'm trying to update the envoy-future docker package. happy to provide any other context if needed :) [08:22:57] bjensen: sure thing, I'll rebuild the image immediately [08:23:06] thanks! [08:23:14] and there is already the patch for switching to the Jenkins jobs, that is great [08:23:21] (03CR) 10Hashar: [C:03+2] helm-linter: Update envoy-future to 1.39.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1324699 (https://phabricator.wikimedia.org/T421418) (owner: 10Blake) [08:25:06] (03Merged) 10jenkins-bot: helm-linter: Update envoy-future to 1.39.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1324699 (https://phabricator.wikimedia.org/T421418) (owner: 10Blake) [08:25:53] (03PS2) 10Blake: jjb: update job to releng/helm-linter:0.8.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1324730 (https://phabricator.wikimedia.org/T421418) [08:29:40] (03CR) 10Hashar: [C:03+2] jjb: update job to releng/helm-linter:0.8.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1324730 (https://phabricator.wikimedia.org/T421418) (owner: 10Blake) [08:30:11] bjensen: note, Gerrit automatically rebase the change when it submits it in the repo :) [08:30:25] ahhhh, okay, nice :D [08:30:31] at least on the integration/* repos [08:30:45] it looked like the shellcheck test was failing pre-rebase, is that expected? [08:30:49] well I think that is nowadays the case for all repositories with some exceptions [08:30:54] ah [08:31:21] 10Phabricator, 06Security-Team, 07SecTeam-Processed, 07Security: Create a security issue reporting template - https://phabricator.wikimedia.org/T406313#12210960 (10Aklapper) 05Stalledβ†’03Declined No updates for half a year; declining as part of cleaning up the Phab backlog. Feel free to reopen once... [08:31:24] (03Merged) 10jenkins-bot: jjb: update job to releng/helm-linter:0.8.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1324730 (https://phabricator.wikimedia.org/T421418) (owner: 10Blake) [08:31:25] sorry I missed that! [08:31:43] no worries! it looks like it passed on the rebase, anyway [08:32:07] 00:01:33.310 Waiting for the completion of castor-save-workspace-cache [08:32:07] 00:02:11.333 Build step 'Execute scripts' changed build result to FAILURE [08:32:12] that is unrelated to shellcheck itself [08:32:38] it is some CI infra issue, that `castor-save-workspace-cache` is a sub job which collects/save the package managers caches [08:32:45] (03CR) 10Hashar: [C:03+2] "I have updated the job!" [integration/config] - 10https://gerrit.wikimedia.org/r/1324730 (https://phabricator.wikimedia.org/T421418) (owner: 10Blake) [08:32:52] I'll look at it [08:32:59] bjensen: the helm-lint job now uses the new image! [08:33:14] hmmm, where was that output? i was looking in the jenkins console output, and don't see those lines [08:33:23] (also, thanks very much for the help!) [08:33:54] most certainly the sub job got canceled/interrupted and never had an opportunity to run [08:34:16] so it bubbles up as being in failure which in turns causes the calling build (shellcheck) to fail [08:34:34] ahhhh, okay [08:34:52] there is an issue in Jenkins itself, with a bunch of java code involved and I think I keep forgetting about finishing that patch for upstream [08:35:40] and the castor-save-workspace-cache job has its own issue, it is sometime way tooo slow and blocks everyone else, so eventually other builds end up waiting too long for job to start, a timeout is reached and the calling build get erroneously marked as failure [08:36:17] is there a way I might be able to see whether that's the problem with a particular failure? [08:36:19] we pushed some patches yesterday to address some issues which should improve it a bit and there are a few more steps we need to do [08:36:25] (sorry that is a very long brain dump) [08:36:42] i appreciate it, it's good to know :D [08:37:13] and yeah the symptom is the console logs I have pasted: [08:37:14] > Waiting for the completion of castor-save-workspace-cache [08:37:14] That is the sub job being triggered, once it triggers there should be some indication of the build having started/completed and a build number [08:38:09] if that message is immediately followed by `Build step 'Execute scripts' changed build result to FAILURE` after a minute or more, it means the `castor-save-workspace-cache` never had an opportunity to run and that is interpreted as a FAILURE by Jenkins [08:38:42] I think the patch I wrote for upstream makes Jenkins to emit an explanation message in that occurs [08:39:07] gotcha, that makes sense [08:39:15] yeah, an explanation message there would be really helpful [08:39:16] and properly handle interrupted/canceled/aborted jobs so one can decided whether the calling build should be a SUCCESS or a FAILURE (currently it always consider them to be FAILURE) [08:39:19] yeah [08:39:36] meanwhile the helm-lint has been updated, if you could test it and verify it is working it would be great :] [08:39:45] will do, thanks! [08:40:10] the rest is just CI madness, sorry if I have overwhelmed you with a ton of context that is entirely disconnected from helm/envoy πŸ™„ [08:41:03] it's all good, i'm happy to learn about these systems :) [08:41:41] i think we should be set, it looks like the rebuild last succeeded as expected, and we have "09:37:47 configuration '/etc/envoy/envoy.yaml' OK" [08:43:32] awesome thank you! [08:44:01] no worries, thanks for the help! [08:48:46] Somehow adding a test caused jenkins to say test coverage decreased (on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Scribunto/+/1325415 - PS1 had no tests, and coverage decreased by 1%. I added tests in PS2, and coverage decreased by 30 percentage points) [09:47:46] 10Phabricator (Upstream), 06Security-Team, 07SecTeam-Processed, 07Security, 07Upstream: Creating access-restricted tasks from email (e.g. security tasks) - https://phabricator.wikimedia.org/T434536#12211359 (10Aklapper) [09:48:17] 10Phabricator (Upstream), 06Security-Team, 07SecTeam-Processed, 07Security, 07Upstream: Creating access-restricted tasks from email (e.g. security tasks) - https://phabricator.wikimedia.org/T434536#12211360 (10Aklapper) If [validation of policy identifiers in upstream wasn't such a clusterfuck (basically... [10:30:56] 10Phabricator (Upstream), 06Security-Team, 07SecTeam-Processed, 07Security, 07Upstream: Creating access-restricted tasks from email (e.g. security tasks) - https://phabricator.wikimedia.org/T434536#12211516 (10Aklapper) I mean, we could put that code above into our downstream instance if you want to, it'... [10:34:34] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for gaiarajan - https://phabricator.wikimedia.org/T434626#12211525 (10Aklapper) @gaiarajan Hi, which related codebase for that task is located in Wikimedia GitLab? [10:36:51] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for WouterKo - https://phabricator.wikimedia.org/T434605#12211536 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Approved. Happy hacking! [10:39:09] 10GitLab (Account Approval), 06Release-Engineering-Team: GitLab Account Unlock Request - Srirae - https://phabricator.wikimedia.org/T431925#12211546 (10Aklapper) @Srirae: Could you please answer the last comment? Thanks in advance! [11:01:21] 10GitLab (Account Approval), 06Release-Engineering-Team: GitLab Account Unlock Request - Srirae - https://phabricator.wikimedia.org/T431925#12211592 (10Srirae) Hi Aklapper, I don't have a specific project in mind, it's just more of like starting off with simple systems and similar to good-first-issues. [11:33:53] (03open) 10phedenskog: Stop inserting duplicate quibble section rows [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/12 [11:55:59] 10Phabricator (Upstream), 07Upstream: Phabricator TOTP token should be evaluated against the currently valid one instead of expiring - https://phabricator.wikimedia.org/T434520#12211829 (10Michael) 05Openβ†’03Stalled Meh, now it works for me as well. If/when it happens to me again, I'll make a screenshot. Ma... [12:30:19] (03open) 10phedenskog: Add a Quibble dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/13 (https://phabricator.wikimedia.org/T434746) [12:47:33] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Improve situation with K3s cluster's cred/passwd error - https://phabricator.wikimedia.org/T434781 (10jnuche) 03NEW [12:47:50] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Improve situation with K3s cluster's cred/passwd error - https://phabricator.wikimedia.org/T434781#12212046 (10jnuche) [12:51:40] (03open) 10phedenskog: Graph castor-save run times without the no-op runs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/14 (https://phabricator.wikimedia.org/T432688) [13:32:29] (03close) 10lucaswerkmeister-wmde: Remove Lucas_WMDE from Wednesday backport window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/254 [13:49:41] 06Gerrit-Privilege-Requests, 10Codex: Requesting for +2 privillege in design/codex - https://phabricator.wikimedia.org/T434791 (10Les4353) 03NEW [13:51:54] 06Gerrit-Privilege-Requests, 10Codex: Requesting for +2 privillege in design/codex - https://phabricator.wikimedia.org/T434791#12212335 (10Les4353) [13:54:33] (03PS1) 10Hashar: Move .yamllint to .yamllint.yaml for auto syntax [integration/config] - 10https://gerrit.wikimedia.org/r/1325486 (https://phabricator.wikimedia.org/T429576) [13:54:35] (03PS1) 10Hashar: Revert "yaml: Add document-start marker to satisfy yamllint" [integration/config] - 10https://gerrit.wikimedia.org/r/1325487 (https://phabricator.wikimedia.org/T429576) [13:55:43] 06Gerrit-Privilege-Requests, 10Codex: Requesting for +2 privillege in design/codex - https://phabricator.wikimedia.org/T434791#12212348 (10Aklapper) Hi, please follow https://www.mediawiki.org/wiki/Gerrit/Privilege_policy - thanks. [13:56:09] 06Gerrit-Privilege-Requests, 10JSDoc WMF theme: Configure jsdoc/wmf-theme repo permissions - https://phabricator.wikimedia.org/T364361#12212352 (10taavi) 05Openβ†’03Invalid It seems like, from above, that there is no actual issue with the repository permissions. Closing. [13:56:25] (03CR) 10Hashar: "The `---` at the top annoys me and I am sure we will forget them if we ever have to create a new yaml file which leads to CI failure, edit" [integration/config] - 10https://gerrit.wikimedia.org/r/1325487 (https://phabricator.wikimedia.org/T429576) (owner: 10Hashar) [13:57:15] 06Gerrit-Privilege-Requests, 10Codex: Request membership in design/codex group for Les4353 - https://phabricator.wikimedia.org/T434791#12212357 (10Les4353) [13:57:37] 06Gerrit-Privilege-Requests, 10Codex: Request membership in design/codex group for Les4353 - https://phabricator.wikimedia.org/T434791#12212359 (10Les4353) >>! 在T434791#12212348中，@Aklapperε†™ι“οΌš > Hi, please follow https://www.mediawiki.org/wiki/Gerrit/Privilege_policy - thanks. Sorry, my bad! Fixed that! [14:07:40] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎), 10Striker, 06tools-platform-team: Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12212396 (10Aklapper) 05Openβ†’03Resolved https://gerr... [14:12:02] (03CR) 10Samtar: [C:03+1] "looks right to me :D" [integration/config] - 10https://gerrit.wikimedia.org/r/1324721 (https://phabricator.wikimedia.org/T432039) (owner: 10Jsn.sherman) [14:33:36] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Collaboration-Services, 13Patch-For-Review: Disable IO for diffusion repositories - https://phabricator.wikimedia.org/T405596#12212587 (10Aklapper) We were down to 9 URIs still enabled for pulling three repos from Diffusion: `... [14:34:07] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Collaboration-Services, 13Patch-For-Review: Disable IO for diffusion repositories - https://phabricator.wikimedia.org/T405596#12212591 (10Aklapper) 05Stalledβ†’03Open p:05Triageβ†’03Medium a:03Aklapper [14:34:43] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Collaboration-Services, 13Patch-For-Review: Disable IO for diffusion repositories - https://phabricator.wikimedia.org/T405596#12212596 (10Aklapper) **TODO:** Also note that we do still mirror //from Diffusion// (to Github etc)... [14:45:52] (03CR) 10Jforrester: [C:03+2] Move .yamllint to .yamllint.yaml for auto syntax [integration/config] - 10https://gerrit.wikimedia.org/r/1325486 (https://phabricator.wikimedia.org/T429576) (owner: 10Hashar) [14:46:09] (03CR) 10Jforrester: [C:03+2] Revert "yaml: Add document-start marker to satisfy yamllint" [integration/config] - 10https://gerrit.wikimedia.org/r/1325487 (https://phabricator.wikimedia.org/T429576) (owner: 10Hashar) [14:47:43] (03Merged) 10jenkins-bot: Move .yamllint to .yamllint.yaml for auto syntax [integration/config] - 10https://gerrit.wikimedia.org/r/1325486 (https://phabricator.wikimedia.org/T429576) (owner: 10Hashar) [14:47:53] (03Merged) 10jenkins-bot: Revert "yaml: Add document-start marker to satisfy yamllint" [integration/config] - 10https://gerrit.wikimedia.org/r/1325487 (https://phabricator.wikimedia.org/T429576) (owner: 10Hashar) [14:56:07] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for gaiarajan - https://phabricator.wikimedia.org/T434626#12212720 (10gaiarajan) @Aklapper Apologies for the confusion! That ticket is on Gerrit, which I mixed up. While looking into it, I realized that codepath (wiki... [15:42:06] (03open) 10dancy: lint: Include exit status and stderr when php lint fails [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1245 (https://phabricator.wikimedia.org/T272760) [15:42:40] (03update) 10dancy: lint: Include exit status and stderr when php lint fails [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1245 (https://phabricator.wikimedia.org/T272760) [16:06:09] hashar: Review request ping on https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1240 [17:08:39] 10Phabricator Translations Extension: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12213411 (10Pppery) After next update ` Translations are defined for the locale `kw`, which is not recognized. Translations are defined for the locale `ro`, which is not reco... [17:19:39] 10Phabricator Translations Extension, 13Patch-For-Review: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12213428 (10Pppery) Okay, added the two missing locale files. The rest of these look like upstream problems. [17:26:26] (03open) 10pppery: Remove unused downstream-only function [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/126 (https://phabricator.wikimedia.org/T412651) [17:26:32] (03update) 10pppery: Remove unused downstream-only function [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/126 (https://phabricator.wikimedia.org/T412651) [17:27:33] 10Phabricator Translations Extension, 13Patch-For-Review: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12213455 (10Pppery) Above patch deals with ` The locale `sk` defines a translation for the key `%d day(s)`, which varies on the plurality or gender of p... [17:35:45] 10Phabricator Translations Extension, 13Patch-For-Review: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12213494 (10Pppery) https://we.phorge.it/D27189 for the upstream problems. [17:49:30] (03open) 10dancy: kubernetes.py: Increase PROGRESS_GRACE_PERIOD to 30 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1251 (https://phabricator.wikimedia.org/T375514) [17:52:12] (03merge) 10dancy: kubernetes.py: Increase PROGRESS_GRACE_PERIOD to 30 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1251 (https://phabricator.wikimedia.org/T375514) [17:53:02] (03open) 10dancy: Release 4.280.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1252 [17:55:36] (03merge) 10dancy: Release 4.280.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1252 [18:13:37] (03update) 10brennen: Restore php-xhprof and add php-excimer [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 (https://phabricator.wikimedia.org/T401115) (owner: 10krinkle) [18:14:01] (03update) 10brennen: Restore php-xhprof and add php-excimer [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 (https://phabricator.wikimedia.org/T401115) (owner: 10krinkle) [18:14:14] (03update) 10brennen: Restore php-xhprof and add php-excimer [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 (https://phabricator.wikimedia.org/T401115) (owner: 10krinkle) [18:15:02] (03approved) 10brennen: Restore php-xhprof and add php-excimer [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 (https://phabricator.wikimedia.org/T401115) (owner: 10krinkle) [18:16:39] (03merge) 10brennen: Restore php-xhprof and add php-excimer [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 (https://phabricator.wikimedia.org/T401115) (owner: 10krinkle) [19:22:55] 10GitLab: Internal Server Error when publishing a maven package - https://phabricator.wikimedia.org/T434834 (10Don-vip) 03NEW [19:25:02] 10GitLab: Internal Server Error when publishing a maven package from GitLab CI - https://phabricator.wikimedia.org/T434834#12213952 (10Don-vip) [19:26:30] 10GitLab: Internal Server Error when publishing a maven package from GitLab CI - https://phabricator.wikimedia.org/T434834#12213953 (10Don-vip) [19:27:54] !log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 (T401115) [19:27:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:27:57] T401115: Replace php-tideways_xhprof with php-xhprof in MediaWiki-Docker - https://phabricator.wikimedia.org/T401115 [19:38:48] 10GitLab: Internal Server Error when publishing a maven package from GitLab CI - https://phabricator.wikimedia.org/T434834#12214003 (10Don-vip) @ABran-WMF could it be a side effect of T425441? [19:54:00] (03open) 10brennen: bookworm-php83: s/exicmer/excimer/ [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/116 (https://phabricator.wikimedia.org/T401115) [19:54:33] (03merge) 10brennen: bookworm-php83: s/exicmer/excimer/ [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/116 (https://phabricator.wikimedia.org/T401115) [19:55:04] !log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 (T401115) (redux for typo fix) [19:55:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:55:07] T401115: Replace php-tideways_xhprof with php-xhprof in MediaWiki-Docker - https://phabricator.wikimedia.org/T401115 [20:02:10] (03open) 10brennen: bookworm-php84: s/exicmer/excimer/ [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/117 (https://phabricator.wikimedia.org/T401115) [20:02:24] (03merge) 10brennen: bookworm-php84: s/exicmer/excimer/ [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/117 (https://phabricator.wikimedia.org/T401115)