[02:10:02] hi! could someone please clear the npm cache for mwext-node20-rundoc https://phabricator.wikimedia.org/T295351 [02:10:17] effected patch / build: https://gerrit.wikimedia.org/r/1176743 [02:10:20] https://integration.wikimedia.org/ci/job/mwext-node20-rundoc/15751/console [02:10:49] like so https://sal.toolforge.org/log/6eCgWZcB8tZ8Ohr0CC_n [04:10:09] 10GitLab (Project Migration), 06Community-Tech, 10WS Export, 13Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11084549 (10Samwilson) The other failure remaining is Phan running out of memory, but I think that can be ignored for now. I've moved the... [05:38:56] 10GitLab (Project Migration), 06Community-Tech, 10WS Export, 13Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11084575 (10Samwilson) [06:48:24] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11084631 (10Eileenmcnaughton) 05Resolved→03Open @hashar @dancy -sorry to re-open I just discovered there... [07:15:22] (03PS1) 10Hashar: Revert "dockerfiles: [civicrm] add php-mysqli" [integration/config] - 10https://gerrit.wikimedia.org/r/1178717 [07:17:02] (03PS2) 10Hashar: Revert "dockerfiles: [civicrm] add php-mysqli" [integration/config] - 10https://gerrit.wikimedia.org/r/1178717 (https://phabricator.wikimedia.org/T401451) [07:20:18] (03CR) 10Hashar: [C:03+2] Revert "dockerfiles: [civicrm] add php-mysqli" [integration/config] - 10https://gerrit.wikimedia.org/r/1178717 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [07:21:59] (03Merged) 10jenkins-bot: Revert "dockerfiles: [civicrm] add php-mysqli" [integration/config] - 10https://gerrit.wikimedia.org/r/1178717 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [07:31:34] 10Continuous-Integration-Config: Add `ext-mysqli` to the composer-test CI container - https://phabricator.wikimedia.org/T226585#11084725 (10hashar) 05Open→03Declined The composer jobs / Docker images do not have MySQL / MariaDB running and it will not be supported. When a project requires a MariaDB serv... [07:32:46] (03PS1) 10Hashar: dockerfiles: [civicrm] add composer [integration/config] - 10https://gerrit.wikimedia.org/r/1178721 (https://phabricator.wikimedia.org/T401451) [07:34:09] (03PS1) 10Hashar: jjb: update civicrm to 0.8 [integration/config] - 10https://gerrit.wikimedia.org/r/1178722 (https://phabricator.wikimedia.org/T401451) [07:34:24] (03CR) 10Hashar: [C:03+2] dockerfiles: [civicrm] add composer [integration/config] - 10https://gerrit.wikimedia.org/r/1178721 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [07:35:52] (03Merged) 10jenkins-bot: dockerfiles: [civicrm] add composer [integration/config] - 10https://gerrit.wikimedia.org/r/1178721 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [07:42:06] (03CR) 10Hashar: [C:03+2] jjb: update civicrm to 0.8 [integration/config] - 10https://gerrit.wikimedia.org/r/1178722 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [07:43:32] (03Merged) 10jenkins-bot: jjb: update civicrm to 0.8 [integration/config] - 10https://gerrit.wikimedia.org/r/1178722 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [08:15:28] (03PS1) 10Hashar: dockerfiles: rm /srv/composer from images [integration/config] - 10https://gerrit.wikimedia.org/r/1178807 [08:20:58] 10GitLab (Project Migration), 06Community-Tech, 10WS Export, 13Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11084845 (10Samwilson) The User-Agent issue was that we weren't configuring Guzzle correctly! Oops. Fixed now. [08:24:36] (03PS1) 10Hashar: dockerfiles: regroup composer test script in composer-scratch [integration/config] - 10https://gerrit.wikimedia.org/r/1178809 [08:24:59] (03CR) 10Hashar: [C:03+2] dockerfiles: rm /srv/composer from images [integration/config] - 10https://gerrit.wikimedia.org/r/1178807 (owner: 10Hashar) [08:26:37] (03Merged) 10jenkins-bot: dockerfiles: rm /srv/composer from images [integration/config] - 10https://gerrit.wikimedia.org/r/1178807 (owner: 10Hashar) [08:33:09] (03PS1) 10Hashar: dockerfiles: [civicrm] add /run-test from composer image [integration/config] - 10https://gerrit.wikimedia.org/r/1178811 (https://phabricator.wikimedia.org/T401451) [08:46:03] 10GitLab (Project Migration), 06Community-Tech, 06translatewiki.net, 10WS Export, 13Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11084943 (10Samwilson) [08:50:02] (03PS1) 10Hashar: Zuul: [wikimedia/fundraising/crm] run composer test first [integration/config] - 10https://gerrit.wikimedia.org/r/1178814 [08:50:02] (03PS1) 10Hashar: jjb: job to composer test with civicrm image [integration/config] - 10https://gerrit.wikimedia.org/r/1178815 (https://phabricator.wikimedia.org/T401451) [08:50:04] (03PS1) 10Hashar: Zuul: [wikimedia/fundraising/crm] composer test with crm env [integration/config] - 10https://gerrit.wikimedia.org/r/1178816 (https://phabricator.wikimedia.org/T401451) [09:08:04] (03CR) 10Hashar: [C:03+2] dockerfiles: regroup composer test script in composer-scratch [integration/config] - 10https://gerrit.wikimedia.org/r/1178809 (owner: 10Hashar) [09:09:49] (03Merged) 10jenkins-bot: dockerfiles: regroup composer test script in composer-scratch [integration/config] - 10https://gerrit.wikimedia.org/r/1178809 (owner: 10Hashar) [09:16:30] (03CR) 10Hashar: [C:03+2] dockerfiles: [civicrm] add /run-test from composer image [integration/config] - 10https://gerrit.wikimedia.org/r/1178811 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [09:17:52] (03Merged) 10jenkins-bot: dockerfiles: [civicrm] add /run-test from composer image [integration/config] - 10https://gerrit.wikimedia.org/r/1178811 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [09:21:21] (03CR) 10Hashar: [C:03+2] Zuul: [wikimedia/fundraising/crm] run composer test first [integration/config] - 10https://gerrit.wikimedia.org/r/1178814 (owner: 10Hashar) [09:21:47] (03CR) 10Hashar: [C:03+2] jjb: job to composer test with civicrm image [integration/config] - 10https://gerrit.wikimedia.org/r/1178815 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [09:21:51] (03CR) 10Hashar: [C:03+2] Zuul: [wikimedia/fundraising/crm] composer test with crm env [integration/config] - 10https://gerrit.wikimedia.org/r/1178816 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [09:22:47] (03Merged) 10jenkins-bot: Zuul: [wikimedia/fundraising/crm] run composer test first [integration/config] - 10https://gerrit.wikimedia.org/r/1178814 (owner: 10Hashar) [09:23:10] (03Merged) 10jenkins-bot: jjb: job to composer test with civicrm image [integration/config] - 10https://gerrit.wikimedia.org/r/1178815 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [09:23:17] (03Merged) 10jenkins-bot: Zuul: [wikimedia/fundraising/crm] composer test with crm env [integration/config] - 10https://gerrit.wikimedia.org/r/1178816 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [09:28:00] (03PS1) 10Arthur taylor: Remove composer timeout on wikibase phpunit jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1178820 (https://phabricator.wikimedia.org/T401888) [09:34:11] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11085098 (10hashar) I had to do a bit of refactoring. I have changed the composer test j... [09:36:29] Project beta-scap-sync-world build #219450: 04FAILURE in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219450/ [09:46:30] Project beta-scap-sync-world build #219451: 04STILL FAILING in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219451/ [09:56:34] Project beta-scap-sync-world build #219452: 04STILL FAILING in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219452/ [10:02:19] (03PS1) 10Hashar: jjb: update civicrm to 0.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1178826 (https://phabricator.wikimedia.org/T401451) [10:06:31] Project beta-scap-sync-world build #219453: 04STILL FAILING in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219453/ [10:14:12] (03update) 10sfaci: Added Metrics Platform Experimentation Lab deployment window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/186 (https://phabricator.wikimedia.org/T396742) [10:16:16] (03CR) 10Hashar: [C:03+2] jjb: update civicrm to 0.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1178826 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [10:16:39] Project beta-scap-sync-world build #219454: 04STILL FAILING in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219454/ [10:17:41] (03Merged) 10jenkins-bot: jjb: update civicrm to 0.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1178826 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [10:26:37] Project beta-scap-sync-world build #219455: 04STILL FAILING in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219455/ [10:29:16] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11085519 (10hashar) I have removed the `composer-php82` job which uses an image that does... [10:36:31] Project beta-scap-sync-world build #219456: 04STILL FAILING in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219456/ [10:46:28] Project beta-scap-sync-world build #219457: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219457/ [10:56:33] Project beta-scap-sync-world build #219458: 04STILL FAILING in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219458/ [11:00:13] 10Continuous-Integration-Config, 10Wikidata, 07ci-test-error, 13Patch-For-Review, 10Wikidata-Omega (The Board): wikibase-repo-php81 jobs starting to hit composer timeout (300 seconds) - https://phabricator.wikimedia.org/T401888#11085772 (10A_smart_kitten) [11:10:07] Project beta-scap-sync-world build #219459: 04STILL FAILING in 4 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219459/ [11:18:28] Project beta-scap-sync-world build #219460: 04STILL FAILING in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219460/ [11:26:33] Project beta-scap-sync-world build #219461: 04STILL FAILING in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219461/ [11:36:26] Project beta-scap-sync-world build #219462: 04STILL FAILING in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219462/ [11:46:33] Project beta-scap-sync-world build #219463: 04STILL FAILING in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219463/ [11:56:39] Project beta-scap-sync-world build #219464: 04STILL FAILING in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219464/ [12:06:30] Project beta-scap-sync-world build #219465: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219465/ [12:16:27] Project beta-scap-sync-world build #219466: 04STILL FAILING in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219466/ [12:26:34] Project beta-scap-sync-world build #219467: 04STILL FAILING in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219467/ [12:30:00] hey, happy to file a phab task in case it'd be better handled that way, but in case anyone knows the answer to this question off the top of their head - is gerrit meant to be relatively slow when `git clone`-ing a repo? it's not a new thing for me, but i've always been surprised by the relatively slow download speed when cloning from gerrit (which is much slower than e.g. cloning from github). is it like this for anyone else, o [12:30:16] testing just now over HTTPS, the download seemed to start off at a *relatively* fast speed (~5/6 MiB/s), but then quickly dropped down into speeds around 400-800 KiB/s (and seemed to mostly stay like that for the remainder of the clone, occasionally going up to ~1 MiB/s and then dropping back down again). [12:30:28] maybe i just have a relatively fast connection (that's faster than Gerrit can send the files to me)? :p [12:36:27] Project beta-scap-sync-world build #219468: 04STILL FAILING in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219468/ [12:41:26] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918#11086096 (10Lucas_We... [12:46:28] Project beta-scap-sync-world build #219469: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219469/ [12:56:32] Project beta-scap-sync-world build #219470: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219470/ [13:06:19] Project beta-scap-sync-world build #219471: 04STILL FAILING in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219471/ [13:11:19] failure caused by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Phonos/+/1176811 [13:16:37] Project beta-scap-sync-world build #219472: 04STILL FAILING in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219472/ [13:18:01] (03CR) 10Phuedx: [C:04-1] "If we can limit this to the Phan part of the change, then this LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/1178541 (https://phabricator.wikimedia.org/T397143) (owner: 10Santiago Faci) [13:26:32] Project beta-scap-sync-world build #219473: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219473/ [13:32:22] 10GitLab (Project Migration), 06Community-Tech, 06translatewiki.net, 10WS Export, 13Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11086234 (10Nikerabbit) We generally prefer separate tasks so that we can add them to our sprint. In this case you... [13:36:35] Project beta-scap-sync-world build #219474: 04STILL FAILING in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219474/ [13:36:54] 10Continuous-Integration-Config, 10Wikidata, 07ci-test-error, 13Patch-For-Review, 10Wikidata-Omega (The Board): wikibase-repo-php81 jobs starting to hit composer timeout (300 seconds) - https://phabricator.wikimedia.org/T401888#11086250 (10hashar) [13:41:27] (03PS1) 10Hashar: jjb: disable composer timeout on Wikibase PHPUnit jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1178881 (https://phabricator.wikimedia.org/T401888) [13:41:32] 10GitLab: GitLab Private Repository Request for: repos/sre/XCHEESESCORE - https://phabricator.wikimedia.org/T401921 (10CDanis) 03NEW [13:44:59] (03PS2) 10Hashar: Remove composer timeout on wikibase phpunit jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1178820 (https://phabricator.wikimedia.org/T401888) (owner: 10Arthur taylor) [13:45:05] (03Abandoned) 10Hashar: jjb: disable composer timeout on Wikibase PHPUnit jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1178881 (https://phabricator.wikimedia.org/T401888) (owner: 10Hashar) [13:45:39] (03CR) 10Hashar: [C:03+2] Remove composer timeout on wikibase phpunit jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1178820 (https://phabricator.wikimedia.org/T401888) (owner: 10Arthur taylor) [13:45:58] 10Continuous-Integration-Config, 10Wikidata, 07ci-test-error, 13Patch-For-Review, 10Wikidata-Omega (The Board): wikibase-repo-php81 jobs starting to hit composer timeout (300 seconds) - https://phabricator.wikimedia.org/T401888#11086295 (10hashar) 05Open→03Resolved a:03hashar [13:46:03] (03CR) 10Hashar: [C:03+2] "I have updated the jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/1178820 (https://phabricator.wikimedia.org/T401888) (owner: 10Arthur taylor) [13:46:32] Project beta-scap-sync-world build #219475: 04STILL FAILING in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219475/ [13:47:00] (03Merged) 10jenkins-bot: Remove composer timeout on wikibase phpunit jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1178820 (https://phabricator.wikimedia.org/T401888) (owner: 10Arthur taylor) [13:49:48] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Troubleshoot GitLab nftables throttling after switchover - https://phabricator.wikimedia.org/T400971#11086317 (10Jelto) a:05Jelto→03ABran-WMF I'll hand this task over to @ABran-WMF while I'm out. Arnaud already started with troubles... [13:53:37] A_smart_kitten: re slow git clone, I am pretty sure in all cases the issues were on the client side rather than on Gerrit server [13:53:58] there is one exception I remember of is that the git fetch negotiations would send every single refs known by the server [13:54:29] so that the server dumbly sends every single refs it knows about [13:54:57] and on Gerrit that means every single patches ever made for that repo (refs/changes/*/*/*) which is a sizeable amount of traffic [13:55:18] eventually Google engineers have faced the same issue we had and revisited git protocol [13:55:38] with `protocol.version = 2` that is no more an issue for fetches [13:55:39] https://phabricator.wikimedia.org/J199 [13:56:00] anyway for a full clone, I can almost guarantee it is on your side or somewhere in the network path [13:56:23] I have tried a full clone of mediawiki/core over https from a WMCS instance: [13:56:24] Receiving objects: 100% (1332592/1332592), 680.91 MiB | 33.63 MiB/s, done. [13:56:35] (which is MediaWiki 700MB is a different story :b ) [13:56:37] Project beta-scap-sync-world build #219476: 04STILL FAILING in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219476/ [13:56:40] s/which/why/ [13:57:01] You can take some traces / timing with `GIT_TRACE2=1 git clone` [13:59:07] try ipv4 vs ipv6: `git clone -4` `git clone -6` [14:00:44] beta-scap-sync-world will be fixed by a revert: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Phonos/+/1178885 [14:00:58] noticed by Reedy and the revert was made by MatmaRex `\o/` [14:01:22] i just saw the big red "STILL FAILING" above :) [14:02:51] hashar: hrm, my first bet from those options would be "somewhere in the network path" but i guess i can't immediately rule out a local issue somewhere :] [14:02:59] thanks for the info anyhow! i'll try and do more debugging [14:05:54] 10GitLab (Infrastructure), 06collaboration-services: Make sure GitLab scales with more usage - https://phabricator.wikimedia.org/T374448#11086387 (10Jelto) a:05Jelto→03None I'm currently not working on this task. [14:06:28] Project beta-scap-sync-world build #219477: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219477/ [14:11:59] `git clone -6` gives me "fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/core.git/': Failed to connect to gerrit.wikimedia.org port 443 after 14 ms: Couldn't connect to server", despite the fact that i *have* a public ipv6, so i guess that's something [14:16:31] Project beta-scap-sync-world build #219478: 04STILL FAILING in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219478/ [14:18:21] okay, the fact that ipv6 wasn't working seemed to be a WSL[0] thing. attempting to `git clone` outside of WSL doesn't show a meaningful difference in symptoms between using `-4` & `-6` AFAICS [14:18:23] [0] https://learn.microsoft.com/en-us/windows/wsl [14:25:52] (03PS2) 10Santiago Faci: Zuul: [mediawiki/extensions/WikimediaEvents]: Updated CI dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1178541 (https://phabricator.wikimedia.org/T397143) [14:26:30] Project beta-scap-sync-world build #219479: 04STILL FAILING in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219479/ [14:27:14] (03CR) 10Santiago Faci: Zuul: [mediawiki/extensions/WikimediaEvents]: Updated CI dependencies (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1178541 (https://phabricator.wikimedia.org/T397143) (owner: 10Santiago Faci) [14:39:19] A_smart_kitten: that could also be an issue on the server side, but I really doubt it :] [14:43:07] hashar: turns out, connecting to a VPN and then attempting to clone mediawiki/core improved my download speeds dramatically. so i guess it is something somewhere along the network path :/ [14:49:39] Yippee, build fixed! [14:49:39] Project beta-scap-sync-world build #219480: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/219480/ [15:36:40] !log Unblock 91.152.0.0/13 (T401898) [15:36:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:33:24] (03approved) 10dancy: make-container-image: php version metadata label [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/203 (https://phabricator.wikimedia.org/T401721) (owner: 10swfrench) [16:33:26] (03update) 10dancy: make-container-image: php version metadata label [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/203 (https://phabricator.wikimedia.org/T401721) (owner: 10swfrench) [16:40:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 07Zuul, 07Developer Productivity, 07Upstream: Abort a Zuul pipeline when one job completed with failures (change zuul scheduler's failure check from areAllJobsComplete to didAny... - https://phabricator.wikimedia.org/T248531#11086994 [16:50:50] (03approved) 10dancy: kubernetes: set php.version based on image labels [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/973 (https://phabricator.wikimedia.org/T401721) (owner: 10swfrench) [17:19:20] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11087106 (10Dzahn) [19:47:05] (03open) 10dancy: patches.py: Add "next" handling to update-patch/remove-patch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/978 (https://phabricator.wikimedia.org/T295925) [19:47:07] (03update) 10dancy: patches.py: Add "next" handling to update-patch/remove-patch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/978 (https://phabricator.wikimedia.org/T295925) [19:48:18] (03update) 10dancy: patches.py: Add "next" handling to update-patch/remove-patch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/978 (https://phabricator.wikimedia.org/T295925) [19:48:23] (03update) 10dancy: patches.py: Add "next" handling to update-patch/remove-patch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/978 (https://phabricator.wikimedia.org/T295925) [19:51:22] (03update) 10dancy: patches.py: Add "next" handling to update-patch/remove-patch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/978 (https://phabricator.wikimedia.org/T295925) [20:58:23] hi! I asked yesterday but got no reply. Could someone please clear the npm cache for mwext-node20-rundoc https://phabricator.wikimedia.org/T295351 [20:58:28] effected patch / build: https://gerrit.wikimedia.org/r/1176743 [20:58:36] https://integration.wikimedia.org/ci/job/mwext-node20-rundoc/15751/console [20:59:35] maybe James_F since he's helped me a few times in the past? 🙏 https://sal.toolforge.org/log/6eCgWZcB8tZ8Ohr0CC_n [21:23:15] musikanimal: I can help, lemme see [21:26:24] !log thcipriani@integration-castor05:~$ sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node20-rundoc/_cacache/ for https://gerrit.wikimedia.org/r/1176743 [21:26:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:26:48] musikanimal: wanna retry? [21:34:25] sorry, I was in a meeting. Retrying now! [21:35:30] by the way, this happens to the CodeMirror repo at least once or twice a month :( I do not know why, maybe because it has more NPM packages than other repos? In https://gerrit.wikimedia.org/r/1176743 we didn't even change the packages, though [21:36:23] yeah, I was just wondering: do you happen to know if npm has an exit code for ENOENT probems? [21:37:09] I was looking at the task, and I was wondering how to detect it deterministically (other than looking at stderr for ENOENT) [21:38:09] I'm not sure. I only ever see this in CI, and also only in post-merge [21:38:26] err, gate-and-submit, not post-merge. you know what i mean :) [21:38:54] yeah :) OK, I'll reply on the task, seems like something we should be able to detect. [21:38:55] anyway the mwext-node20-rundoc build passed, so I think we're good for now! Thank you :) [21:39:04] \o/ [21:46:10] thcipriani: sorry, maybe something else you can help with while I've got you… The selenium test failed (fluke, should pass with a rebuild), so I aborted the other builds to speed it up, and now it seems to be stuck!! :( https://integration.wikimedia.org/zuul/#q=1176743 [21:46:26] * thcipriani looks [21:49:08] hrm zuul doesn't seem too perturbed about it in the logs, 1176743,4> in gate-and-submit> is a failing item because ['a t least one job failed'] [21:49:53] it failing overall is expected, but I aborted all the builds so it should disappear from Zuul right? [21:50:08] oh, it's waiting on the thing in front of it in the queue [21:50:11] or is it actually waiting for the other CI job in the queue… [21:50:11] to report back [21:50:15] oh dear, lol [21:50:30] "fun" [21:50:50] okay, sorry to pester, then! I felt for sure it would abort early hehe [21:51:29] logs seem to indicate it would have [21:51:35] ...if you were the top of the queue [21:51:57] now it's waiting to see if the one in front of you fails to see if it has to re-run all your jobs :D [21:52:12] since maybe that change is making your change fail [21:52:17] Efficiency! [21:52:57] ohh I see! Well I did re-comment on the patch (to trigger a rebuild), so I guess this is expected [21:53:30] just the appearance is misleading, is all. It looks like it's waiting for its turn in the queue to immediately report back that the build failed [21:54:05] I guess gate-and-submit probably follows different rules [21:54:16] yeah, the link you sent me with just the one did confuse me, despite having stared at this system for years. [21:54:26] lol, my bad [21:55:09] it did not rebuild! [21:55:47] so it did in fact wait just to tell me it failed, which it should already know. I think that's only for gate-and-submit, I could have sworn this same tactic of killing the builds works fine for normal `test` builds [21:56:10] yeah, tests is not a dependant pipeline like gate-and-submit [21:56:26] I see [21:56:36] so the one in front of you passed, so the system now knows that the failures in your build were not the patch in front of you's fault [21:56:50] ahhh that makes sense [21:57:22] everything is just as it should be, then, hehe. Alright, thanks so much for the assistance and knowledge sharing! [21:58:36] it's weird, but it's got some nice side-effects. So, in some CI systems, your test might pass and another patch's tests might pass, but if you merge them together, they cause CI to fail. The gate-and-submit queue tries to catch this pre-merge. Which leads to funny-looking behaviors sometimes. [22:01:11] yeah I get it. It looks inefficient but really it's that way to save our asses! hehe. In that sense I think it's worth the wait, every time [22:01:13] but, in theory, the mainline branch of everything that has merged should pass tests together. https://graydon2.dreamwidth.org/1597.html is a good blog about a similar system [22:01:33] graydon2 == the rust guy [22:02:07] oh *that* guy, lol. (I don't actually know of him but I know how passionate Rust people are :) [22:02:16] :D [22:03:05] I felt the need to justify who this was since the blog looks like a long-abandoned dreamwidth thing :P [22:03:37] dreamwidth/livejournal/whatever [22:06:37] noooo!!! I guess I lied about the NPM cache problem only happening in gate-and-submit. https://integration.wikimedia.org/ci/job/mwext-node20-rundoc/15782/console [22:06:57] https://gerrit.wikimedia.org/r/1141495 *does* make NPM package changes, though, so that's probably why [22:07:42] need me to clear it again? [22:07:44] wait never mind… that might have been using the cache from earlier [22:07:55] https://integration.wikimedia.org/ci/job/mwext-node20-rundoc/15816/console passed :) false alarm! [22:08:04] phew, cool :) [22:15:30] 10Gerrit: Forbidden You don't have permission to access this resource In gerrit - https://phabricator.wikimedia.org/T401970 (10Gerges) 03NEW [22:16:47] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor: npm cache saved by castor get corrupted for unknown reason - https://phabricator.wikimedia.org/T295351#11088048 (10thcipriani) >>! In T295351#10906755, @MusikAnimal wrote: >> Can we have Quibble detect this automatically, trigger... [22:49:13] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor: npm cache saved by castor get corrupted for unknown reason - https://phabricator.wikimedia.org/T295351#11088118 (10thcipriani) >>! In T295351#11088048, @thcipriani wrote: > (also, does ENOENT exit `2` with npm?) No, that'd be too... [23:24:50] (03open) 10lmora: releases: Bump Codex to 2.3.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/88 (https://phabricator.wikimedia.org/T401953) [23:40:22] (03CR) 10Jeena Huneidi: [C:03+2] Revert "jjb: [wikilambda-catalyst-end-to-end] Don't set up with Castor, unused" [integration/config] - 10https://gerrit.wikimedia.org/r/1178588 (owner: 10Jforrester) [23:41:46] (03Merged) 10jenkins-bot: Revert "jjb: [wikilambda-catalyst-end-to-end] Don't set up with Castor, unused" [integration/config] - 10https://gerrit.wikimedia.org/r/1178588 (owner: 10Jforrester)