[00:03:22] 10Scap: scap.checks.CheckInvalid error during scap deploy - https://phabricator.wikimedia.org/T319472 (10dancy) [00:04:55] I'm not sure what the best approach would be here [00:05:28] I was proposing the composer-phpXX jobs as an intermediate step between phpXX and composer-package, but I do think the phpXX would be the right place in theory [00:05:46] moving it up one level unblocks it for a bit [00:06:01] I'm sure eventually we'll find the need to move it further... so if we can just go in one go... [00:06:09] I'm building locally to check it works as expected. [00:06:40] show off ;P [00:09:12] Daimona: Are we not using xdebug anywhere? If not, we should uninstall it I guess? [00:09:32] Oh, nice [00:09:55] Re xdebug: I think we can uninstall it from most images, yes. It's only useful if you actually have to debug some code, or maybe profile it [00:10:17] Which should be the job of the dev/ images not the releng/ ones. [00:10:29] So things like dev images for local development should keep it, but CI probably shouldn't [00:10:32] ^^ [00:11:15] And yes, it seems to be truly unused [00:12:25] 10Continuous-Integration-Infrastructure: Drop xdebug from all RelEng images - https://phabricator.wikimedia.org/T319495 (10Jdforrester-WMF) [00:16:54] (03CR) 10Jforrester: [C: 03+2] CoveragePage: Sort by mtime [integration/docroot] - 10https://gerrit.wikimedia.org/r/838929 (owner: 10Reedy) [00:17:23] (03CR) 10Jforrester: [C: 03+2] Minor cleanup [integration/docroot] - 10https://gerrit.wikimedia.org/r/838932 (owner: 10Reedy) [00:17:36] (03Merged) 10jenkins-bot: CoveragePage: Sort by mtime [integration/docroot] - 10https://gerrit.wikimedia.org/r/838929 (owner: 10Reedy) [00:18:03] (03Merged) 10jenkins-bot: Minor cleanup [integration/docroot] - 10https://gerrit.wikimedia.org/r/838932 (owner: 10Reedy) [00:19:24] (03CR) 10Jforrester: [C: 03+2] composer.json: Bump phpunit/phpunit to ^9.5 [integration/docroot] - 10https://gerrit.wikimedia.org/r/838936 (owner: 10Reedy) [00:19:47] (03CR) 10Jforrester: [C: 03+2] WmuiPageBase: Add modified timestamp to footer [integration/docroot] - 10https://gerrit.wikimedia.org/r/838937 (owner: 10Reedy) [00:20:05] (03Merged) 10jenkins-bot: composer.json: Bump phpunit/phpunit to ^9.5 [integration/docroot] - 10https://gerrit.wikimedia.org/r/838936 (owner: 10Reedy) [00:20:27] (03Merged) 10jenkins-bot: WmuiPageBase: Add modified timestamp to footer [integration/docroot] - 10https://gerrit.wikimedia.org/r/838937 (owner: 10Reedy) [00:25:17] (03PS1) 10Ahmon Dancy: perform-release: Fix old version handling [tools/scap] - 10https://gerrit.wikimedia.org/r/838941 [00:32:28] (03PS1) 10Jforrester: dockerfiles: Use the newer php-ast version (v1.0.14) in all images [integration/config] - 10https://gerrit.wikimedia.org/r/838942 [00:35:52] (03CR) 10Daimona Eaytoy: "Neat!" [integration/config] - 10https://gerrit.wikimedia.org/r/838942 (owner: 10Jforrester) [00:42:39] (03CR) 10Jforrester: [C: 03+2] dockerfiles: Use the newer php-ast version (v1.0.14) in all images [integration/config] - 10https://gerrit.wikimedia.org/r/838942 (owner: 10Jforrester) [00:44:38] (03Merged) 10jenkins-bot: dockerfiles: Use the newer php-ast version (v1.0.14) in all images [integration/config] - 10https://gerrit.wikimedia.org/r/838942 (owner: 10Jforrester) [00:48:35] (03PS1) 10Reedy: CoverPage: Fixup sort by mtime [integration/docroot] - 10https://gerrit.wikimedia.org/r/838944 [00:51:07] (03PS1) 10Jforrester: jjb: Update all PHP jobs to have php-ast v1.0.14 consistently [integration/config] - 10https://gerrit.wikimedia.org/r/838946 [00:51:19] I love how we're down to 192 PHP jobs now. [00:51:27] Used to be ~260. [00:51:39] Yay for scrapping PHP72 and some of PHP73. [00:52:00] wait for php 8.2 [00:52:03] and 8.3 [00:52:08] Hush. [00:52:11] :D [00:52:12] Let me celebrate where I can. [00:56:18] 10Continuous-Integration-Infrastructure: Add "last updated" timestamp to test coverage index pages - https://phabricator.wikimedia.org/T319461 (10Reedy) 05Open→03Resolved a:03Reedy Probably some styling/formatting to be added, but otherwise it works [00:57:40] 10Continuous-Integration-Infrastructure: http://localhost:4000/cover-skins/ doesn't work - https://phabricator.wikimedia.org/T319496 (10Reedy) [00:58:35] 10Continuous-Integration-Infrastructure: http://localhost:4000/cover-skins/ doesn't work - https://phabricator.wikimedia.org/T319496 (10Reedy) p:05Triage→03Lowest [00:59:33] Reedy: Oops, "sort by modified time" is perma-selected for me on https://doc.wikimedia.org/cover/ [00:59:35] (03PS1) 10Reedy: Add dev dir for cover-skins [integration/docroot] - 10https://gerrit.wikimedia.org/r/838947 (https://phabricator.wikimedia.org/T319496) [00:59:42] James_F: Yeah, https://gerrit.wikimedia.org/r/838944 :P [00:59:57] Ha, sorry, was heads-down with jjb. [01:00:00] :D [01:00:14] I didn't realise the local testing worked so well... So that one is actually tested [01:00:34] Ha. [01:00:51] (03CR) 10Jforrester: [C: 03+2] CoverPage: Fixup sort by mtime [integration/docroot] - 10https://gerrit.wikimedia.org/r/838944 (owner: 10Reedy) [01:00:52] Which found the cover-skins not working (And also now fixed) [01:01:34] (03Merged) 10jenkins-bot: CoverPage: Fixup sort by mtime [integration/docroot] - 10https://gerrit.wikimedia.org/r/838944 (owner: 10Reedy) [01:01:58] (03CR) 10Jforrester: [C: 03+2] Add dev dir for cover-skins [integration/docroot] - 10https://gerrit.wikimedia.org/r/838947 (https://phabricator.wikimedia.org/T319496) (owner: 10Reedy) [01:02:40] (03Merged) 10jenkins-bot: Add dev dir for cover-skins [integration/docroot] - 10https://gerrit.wikimedia.org/r/838947 (https://phabricator.wikimedia.org/T319496) (owner: 10Reedy) [01:06:54] (03PS1) 10Reedy: DocIndexPage: Exclude cover-skins [integration/docroot] - 10https://gerrit.wikimedia.org/r/838949 [01:06:56] (03PS1) 10Reedy: jquery: Update from v3.2.1 to v3.6.1 [integration/docroot] - 10https://gerrit.wikimedia.org/r/838950 [01:09:42] (03CR) 10Jforrester: [C: 03+2] DocIndexPage: Exclude cover-skins [integration/docroot] - 10https://gerrit.wikimedia.org/r/838949 (owner: 10Reedy) [01:09:48] (03CR) 10Jforrester: [C: 03+2] jquery: Update from v3.2.1 to v3.6.1 [integration/docroot] - 10https://gerrit.wikimedia.org/r/838950 (owner: 10Reedy) [01:10:28] (03Merged) 10jenkins-bot: DocIndexPage: Exclude cover-skins [integration/docroot] - 10https://gerrit.wikimedia.org/r/838949 (owner: 10Reedy) [01:10:49] (03Merged) 10jenkins-bot: jquery: Update from v3.2.1 to v3.6.1 [integration/docroot] - 10https://gerrit.wikimedia.org/r/838950 (owner: 10Reedy) [01:11:52] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: http://localhost:4000/cover-skins/ doesn't work - https://phabricator.wikimedia.org/T319496 (10Reedy) 05Open→03Resolved a:03Reedy [01:38:53] (03CR) 10Jforrester: "I'll fold this into the general upgrade." [integration/config] - 10https://gerrit.wikimedia.org/r/838928 (owner: 10Daimona Eaytoy) [01:38:57] (03Abandoned) 10Jforrester: jjb: Fix typo in image name [integration/config] - 10https://gerrit.wikimedia.org/r/838928 (owner: 10Daimona Eaytoy) [01:49:25] (03PS2) 10Jforrester: jjb: Update all PHP jobs to have php-ast v1.0.14 consistently [integration/config] - 10https://gerrit.wikimedia.org/r/838946 [01:49:33] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/838946 (owner: 10Jforrester) [01:51:24] (03Merged) 10jenkins-bot: jjb: Update all PHP jobs to have php-ast v1.0.14 consistently [integration/config] - 10https://gerrit.wikimedia.org/r/838946 (owner: 10Jforrester) [01:53:12] Oh dear. [01:54:25] Did something break? [01:54:31] Yes. [01:56:08] I relied on an existing image being correct in making the new php81 one. [01:56:14] And apparently it was always broken. [01:56:17] Oops. [01:57:27] (03PS1) 10Jforrester: dockerfiles: [php81] Correct location of php-ast extension [integration/config] - 10https://gerrit.wikimedia.org/r/838953 [02:06:40] 10Continuous-Integration-Infrastructure: https://doc.wikimedia.org/mediawiki-core/ should be sorted in a better way - https://phabricator.wikimedia.org/T319497 (10Reedy) [02:11:08] 22:10:22 + mkdir -m 2777 -p log [02:11:08] 22:10:22 mkdir: cannot create directory ‘log’: Permission denied [02:11:17] legoktm: Welcome to CI. [02:11:19] https://integration.wikimedia.org/ci/job/mwext-php74-phan-docker/1537/console [02:11:30] :< [02:13:31] 22:13:00 + mkdir -m 2777 -p src [02:13:31] 22:13:00 mkdir: cannot create directory ‘src’: Permission denied [02:13:33] it hates me [02:13:35] https://integration.wikimedia.org/ci/job/mwgate-node14-docker/66506/console [02:13:42] (this is on the "..." retry of course) [02:13:59] It's generally a sign that CI is over-loaded and the jobs are colliding, AIUI. [02:14:23] hrmph. *glares in direction of person who is running a massive patch submission bot* [02:14:32] Indeed. :-) [02:17:49] 10Continuous-Integration-Infrastructure: https://doc.wikimedia.org/mediawiki-core/ should be sorted in a better way - https://phabricator.wikimedia.org/T319497 (10Reedy) [02:19:20] ok, I'm throttling libup a bit harder (whenever this patch goes through) [02:19:31] Cool. [02:19:33] <3 legoktm for getting it working again though [02:19:40] +1 [02:20:04] gonna be useful for the phan and phpcs bumps to come pretty soon [02:20:16] :D [02:22:18] (03PS2) 10Jforrester: dockerfiles: [php81] Correct location of php-ast extension [integration/config] - 10https://gerrit.wikimedia.org/r/838953 [02:22:20] (03PS1) 10Jforrester: dockerfiles: [php-ast] Drop support for PHP 7.2/7.3, add 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/838982 [02:22:38] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [php-ast] Drop support for PHP 7.2/7.3, add 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/838982 (owner: 10Jforrester) [02:22:45] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [php81] Correct location of php-ast extension [integration/config] - 10https://gerrit.wikimedia.org/r/838953 (owner: 10Jforrester) [02:24:31] (03Merged) 10jenkins-bot: dockerfiles: [php-ast] Drop support for PHP 7.2/7.3, add 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/838982 (owner: 10Jforrester) [02:24:33] (03Merged) 10jenkins-bot: dockerfiles: [php81] Correct location of php-ast extension [integration/config] - 10https://gerrit.wikimedia.org/r/838953 (owner: 10Jforrester) [02:30:46] (03PS1) 10Jforrester: jjb: Upgrade PHP jobs for php-ast fiddles and PHP 8.1 ast fix [integration/config] - 10https://gerrit.wikimedia.org/r/838985 [02:40:56] Heya. [02:41:15] bawolff: I'm re-building all the PHP 8.1 images right now, in case that's why you're here. :-( [02:41:51] nope, i didn't notice that [02:42:01] But keep up the good work :) [02:42:03] Ha, OK, never mind then, everything is perfect. [02:42:14] Definitely no fails on all PHP 8.1 jobs right now. [02:42:16] * James_F coughs. [03:07:05] OK, I've deployed, now I'm dashing. [03:07:12] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/838985 (owner: 10Jforrester) [03:09:27] (03Merged) 10jenkins-bot: jjb: Upgrade PHP jobs for php-ast fiddles and PHP 8.1 ast fix [integration/config] - 10https://gerrit.wikimedia.org/r/838985 (owner: 10Jforrester) [03:20:31] (03PS5) 10Jforrester: [WIP] dockerfiles: Provide PHP 8.2 CI test images [integration/config] - 10https://gerrit.wikimedia.org/r/818239 (https://phabricator.wikimedia.org/T314093) [03:20:33] (03PS1) 10Jforrester: dockerfiles: [php-ast] Add support for PHP 8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/839005 [03:20:35] (03PS1) 10Jforrester: dockerfiles: [php-ast] Upgrade php-ast to v1.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/839026 [05:42:41] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Deployment-prep should host its own statsd/graphite server - https://phabricator.wikimedia.org/T241285 (10taavi) a:05taavi→03None [05:46:19] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops: Undeploy patch to use old PHP serialization in PHP 7.4 - https://phabricator.wikimedia.org/T318918 (10Joe) a:03Joe [07:26:05] 10Continuous-Integration-Infrastructure, 10Doxygen, 10MediaWiki-Documentation: Upgrade Doxygen to 1.9.1 - https://phabricator.wikimedia.org/T291950 (10hashar) a:05hashar→03Krinkle Typo, the Doxygen 1.9.1 was done by @Krinkle to address T295111 -:] [07:31:32] 10Continuous-Integration-Infrastructure: Add "last updated" timestamp to test coverage index pages - https://phabricator.wikimedia.org/T319461 (10hashar) Result from https://doc.wikimedia.org/cover/ {F35549755 size=full} Thanks for the fix! [07:48:06] Reedy: James_F: when deploying for docroot, scap should invoke a a php fpm restart script which is version agnostic. The actual php version to act on is provisioned by Puppet and should match whatever OS/php version is on the host :] [07:49:01] Reedy: James_F: one can also try out doc.wm.o / integration.wm.o locally by running `composer start:doc` and `composer start:integration` then head to http://localhost:4000 and http://localhost:4001 [07:49:09] anyway, kudos on the sprint yesterday! [08:41:03] 10Phabricator: Have a task creation form which only shows Title, Description, Tags fields - https://phabricator.wikimedia.org/T278287 (10valerio.bozzolan) [09:10:41] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.4): Upgrade Gerrit to 3.4.6 - https://phabricator.wikimedia.org/T319513 (10hashar) [09:10:57] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.4): Upgrade Gerrit to 3.4.6 - https://phabricator.wikimedia.org/T319513 (10hashar) [09:10:59] 10Gerrit, 10Data³, 10Patch-For-Review: Investigate sending Gerrit events to our data lake - https://phabricator.wikimedia.org/T304947 (10hashar) [09:12:11] (03CR) 10Hashar: [C: 04-2] "Gerrit 3.4.6 has been released and includes my patch to add a public getter \o/ I will get our instance upgraded via T319513." [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/830654 (https://phabricator.wikimedia.org/T304947) (owner: 10Hashar) [09:13:23] 10Gerrit, 10Data³, 10Patch-For-Review: Investigate sending Gerrit events to our data lake - https://phabricator.wikimedia.org/T304947 (10hashar) Gerrit 3.4.6 has been released and includes a change I have made in order to access the list of events currently registered in a Gerrit instance. I filed T319513 to... [10:09:12] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [10:09:27] (03PS1) 10Samtar: layout.yaml: Add chlod to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/839472 [10:10:50] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) Thank you @brennen! GitLab is now OK. Do you know who can help with the things on Diffusion and Gerrit? [10:11:31] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [10:35:51] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10Peachey88) >>! In T318342#8289818, @valerio.bozzolan wrote: > Thank you @brennen! GitLab is now OK. > > Do you know who can help with the things on Diffusion and Ge... [11:21:19] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops: Undeploy patch to use old PHP serialization in PHP 7.4 - https://phabricator.wikimedia.org/T318918 (10Joe) 05Open→03Resolved [11:27:29] 10Diffusion: Update wikimedia CH Diffusion repositories - https://phabricator.wikimedia.org/T319529 (10valerio.bozzolan) [11:30:11] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops: Undeploy patch to use old PHP serialization in PHP 7.4 - https://phabricator.wikimedia.org/T318918 (10Joe) [11:30:35] 10MediaWiki-Releasing, 10serviceops, 10MW-1.40-notes (1.40.0-wmf.4; 2022-10-03): Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4 - https://phabricator.wikimedia.org/T261872 (10Joe) [11:32:24] 10Diffusion: Update wikimedia CH Diffusion repositories - https://phabricator.wikimedia.org/T319529 (10valerio.bozzolan) [11:35:17] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 (10TheresNoTime) I apologise for all the back and forth @MatthewVernon — on 172.16.7.115 (`deployment-ms-be06`), swift appears to be running on **6001/6002**: ` samtar@deployment-ms-... [11:36:02] 10Diffusion: Update Wikimedia CH Diffusion repositories - https://phabricator.wikimedia.org/T319529 (10valerio.bozzolan) [12:00:12] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 (10TheresNoTime) Got to be almost there now... running `swift_ring_manager -v`: ` samtar@deployment-ms-fe03:~$ sudo swift_ring_manager -v Checking dispersion for policy standard Chec... [12:28:29] (03CR) 10CI reject: [V: 04-1] Merge tag 'v3.4.6' into wmf/stable-3.4 [software/gerrit] (wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839504 (https://phabricator.wikimedia.org/T319513) (owner: 10Hashar) [12:28:38] New version created yesterday https://gerrit.wikimedia.org/g/integration/config/+/e1f4d1ab2022ac9023087296bae25e6a6b55f251/dockerfiles/mediawiki-phan-testrun/changelog [12:29:01] hashar: Bonjour, do you have any idea why it can't find the image? [12:30:29] Hmmm maybe because 0.3.0-s2 does not seem to exist indeed https://docker-registry.wikimedia.org/releng/mediawiki-phan-testrun/tags/ [12:30:42] I guess it did not get build [12:30:49] or the related changelog miss the version bump [12:31:36] hashar: Yeah, oops. [12:31:48] I'm digging. [12:32:37] Huh. [12:32:46] 2022-10-06 08:32:28,774 [docker-pkg-build] INFO - E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied) [12:33:28] How come apt isn't ready yet? [12:33:51] Oh, duh. [12:34:46] Hmmmm SNAFU, I guess? [12:35:06] (03PS1) 10Jforrester: dockerfiles: [mediawiki-phan-testrun] Add missing 'USER root' so apt works [integration/config] - 10https://gerrit.wikimedia.org/r/839513 [12:35:13] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [mediawiki-phan-testrun] Add missing 'USER root' so apt works [integration/config] - 10https://gerrit.wikimedia.org/r/839513 (owner: 10Jforrester) [12:39:32] (03Merged) 10jenkins-bot: dockerfiles: [mediawiki-phan-testrun] Add missing 'USER root' so apt works [integration/config] - 10https://gerrit.wikimedia.org/r/839513 (owner: 10Jforrester) [12:39:53] (03Abandoned) 10Hoo man: ArticlePlaceholder depends on Scribunto [integration/config] - 10https://gerrit.wikimedia.org/r/837075 (owner: 10Hoo man) [12:39:57] Daimona: Building now. [12:40:04] Thank you [12:40:44] BTW, now that code coverage should be much faster everywhere, the next goal could be to speed up codehealt as well, since it takes ages to complete... [12:41:11] But that's for another day [12:41:18] I think I'd like to move the phan job into the regular composer test jobs first before fiddling with the secondary pipelines. [12:41:20] But yes! [12:42:12] (In terms of speeding up CI.) [12:46:59] (03PS3) 10Hashar: Merge tag 'v3.4.6' into wmf/stable-3.4 [software/gerrit] (wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839504 (https://phabricator.wikimedia.org/T319513) [12:50:18] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [php-ast] Add support for PHP 8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/839005 (owner: 10Jforrester) [12:50:59] https://integration.wikimedia.org/ci/job/wikibase-client-docker/29687/console failed with an npm error “Your cache folder contains root-owned files” [12:51:28] I’ll retry the gate-and-submit, but that sounds like an issue that might need a manual fix on integration-agent-docker-1033? if the cache is shared between builds [12:51:32] so I’m mentioning it here ^^ [12:52:10] (03Merged) 10jenkins-bot: dockerfiles: [php-ast] Add support for PHP 8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/839005 (owner: 10Jforrester) [12:52:54] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops: Undeploy patch to use old PHP serialization in PHP 7.4 - https://phabricator.wikimedia.org/T318918 (10Lucas_Werkmeister_WMDE) 05Resolved→03Open As far as I can tell, this is done in production (thanks Joe!), but not yet in CI – a change I just... [12:53:58] (03PS1) 10Hashar: Update Gerrit to v3.4.6 [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839515 (https://phabricator.wikimedia.org/T319513) [12:54:19] (03CR) 10CI reject: [V: 04-1] Update Gerrit to v3.4.6 [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839515 (https://phabricator.wikimedia.org/T319513) (owner: 10Hashar) [12:55:53] !log update gitlab-settings to enable admin_mode on gitlab test instance - T316419 [12:55:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:56:17] (03PS1) 10Jforrester: dockerfiles: [php74] Re-build to pick up newer version of Wikimedia's PHP [integration/config] - 10https://gerrit.wikimedia.org/r/839517 (https://phabricator.wikimedia.org/T318918) [12:56:58] \o/ [12:57:35] Lucas_WMDE: Should be simple enough. Hopefully. [12:57:43] thanks! [12:58:59] (03CR) 10Hashar: [C: 03+2] Merge tag 'v3.4.6' into wmf/stable-3.4 [software/gerrit] (wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839504 (https://phabricator.wikimedia.org/T319513) (owner: 10Hashar) [12:59:05] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [php74] Re-build to pick up newer version of Wikimedia's PHP [integration/config] - 10https://gerrit.wikimedia.org/r/839517 (https://phabricator.wikimedia.org/T318918) (owner: 10Jforrester) [12:59:18] (03CR) 10Jforrester: [C: 03+2] "After locally building:" [integration/config] - 10https://gerrit.wikimedia.org/r/839517 (https://phabricator.wikimedia.org/T318918) (owner: 10Jforrester) [12:59:30] !log update gitlab-settings to enable admin_mode on gitlab replica instances - T316419 [12:59:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:59:45] hashar: Can you do the jjb update for the new php74? I have to run. :-( [12:59:54] YES [13:00:09] James_F: so no coffee break? :D [13:00:47] (03Merged) 10jenkins-bot: dockerfiles: [php74] Re-build to pick up newer version of Wikimedia's PHP [integration/config] - 10https://gerrit.wikimedia.org/r/839517 (https://phabricator.wikimedia.org/T318918) (owner: 10Jforrester) [13:00:53] !log Docker: Building and publishing php74:0.3.2 and cascade for T318918 [13:00:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:00:55] T318918: Undeploy patch to use old PHP serialization in PHP 7.4 - https://phabricator.wikimedia.org/T318918 [13:00:58] hashar: Yeah, have to go. :-( [13:01:09] no worries ) [13:01:57] I will take care of crafting the jjb change and updating the jobs [13:02:11] !log update gitlab-settings to enable admin_mode on gitlab production instances - T316419 [13:02:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:07:51] (03Merged) 10jenkins-bot: Merge tag 'v3.4.6' into wmf/stable-3.4 [software/gerrit] (wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839504 (https://phabricator.wikimedia.org/T319513) (owner: 10Hashar) [13:08:31] (03CR) 10Hashar: "recheck" [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839515 (https://phabricator.wikimedia.org/T319513) (owner: 10Hashar) [13:08:50] oh joy so many jjb updates needed [13:08:53] hashar: James_F: can I quickly deploy a Zuul config patch or are you working on something? [13:09:08] go for it ) [13:09:27] (03PS2) 10Majavah: layout.yaml: Add chlod to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/839472 (owner: 10Samtar) [13:09:37] (03CR) 10Majavah: [C: 03+2] "deploying" [integration/config] - 10https://gerrit.wikimedia.org/r/839472 (owner: 10Samtar) [13:11:49] (03Merged) 10jenkins-bot: layout.yaml: Add chlod to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/839472 (owner: 10Samtar) [13:12:13] !log reloading zuul for https://gerrit.wikimedia.org/r/839472 [13:12:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:12:46] * taavi done [13:15:06] (03PS1) 10Hashar: jjb: update jobs for newer Wikimedia's php 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/839520 (https://phabricator.wikimedia.org/T318918) [13:17:33] (03CR) 10Hashar: [C: 03+2] jjb: update jobs for newer Wikimedia's php 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/839520 (https://phabricator.wikimedia.org/T318918) (owner: 10Hashar) [13:17:47] !log Mass updating Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/839520 [13:17:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:18:12] (03CR) 10Hashar: [C: 03+2] Update Gerrit to v3.4.6 [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839515 (https://phabricator.wikimedia.org/T319513) (owner: 10Hashar) [13:18:34] (03Merged) 10jenkins-bot: Update Gerrit to v3.4.6 [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839515 (https://phabricator.wikimedia.org/T319513) (owner: 10Hashar) [13:21:02] (03Merged) 10jenkins-bot: jjb: update jobs for newer Wikimedia's php 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/839520 (https://phabricator.wikimedia.org/T318918) (owner: 10Hashar) [13:28:58] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) 05Open→03Resolved p:05Triage→03Medium a:03brennen [13:29:25] 10Diffusion: Update Wikimedia CH Diffusion repositories - https://phabricator.wikimedia.org/T319529 (10valerio.bozzolan) [13:29:29] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [13:29:38] 10Gerrit: Drop Gerrit repositories related to Wikimedia CH (they migrated to GitLab) - https://phabricator.wikimedia.org/T319534 (10valerio.bozzolan) [13:29:42] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [13:41:24] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10User-brennen: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token - https://phabricator.wikimedia.org/T308501 (10Jelto) The findings of yesterdays troubleshooting s... [13:44:39] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops, 10serviceops-collab, and 2 others: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10Jelto) Cross-post from T308501#8290690: it seems we have our first successful [buildkitd... [13:54:42] 10Continuous-Integration-Config, 10MediaWiki-Codesniffer, 10MediaWiki-Core-Tests: Consider enforcing usage of MediaWiki(Unit|Integration)TestCase in MW PHPUnit Test - https://phabricator.wikimedia.org/T319536 (10Legoktm) > I'm not sure about the motivation for that, perhaps it's because the developer thought... [14:00:32] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops: Undeploy patch to use old PHP serialization in PHP 7.4 - https://phabricator.wikimedia.org/T318918 (10hashar) James has made the necessary CI updates and I have deployed them. [14:04:21] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops: Undeploy patch to use old PHP serialization in PHP 7.4 - https://phabricator.wikimedia.org/T318918 (10Joe) 05Open→03Resolved Sorry @LucasWerkmeister I assumed this task was about updating production. Re-resolving then :) [14:04:39] (03PS1) 10Hashar: Remove gerrit2001 from deployment targets [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839551 (https://phabricator.wikimedia.org/T243027) [14:04:55] (03CR) 10Hashar: [C: 03+2] Remove gerrit2001 from deployment targets [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839551 (https://phabricator.wikimedia.org/T243027) (owner: 10Hashar) [14:05:19] (03Merged) 10jenkins-bot: Remove gerrit2001 from deployment targets [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839551 (https://phabricator.wikimedia.org/T243027) (owner: 10Hashar) [14:08:33] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops: Undeploy patch to use old PHP serialization in PHP 7.4 - https://phabricator.wikimedia.org/T318918 (10Lucas_Werkmeister_WMDE) The change for T316923 is passing in CI now (currently going through test-and-submit), so I think this is indeed done. Th... [14:13:02] Project beta-code-update-eqiad build #412113: 04FAILURE in 0.74 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/412113/ [14:21:26] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 (10MatthewVernon) I think this is likely a mismatch between the devices present on the host and the devices declared in the hosts.yaml file - the code can add or remove hosts, but no... [14:23:45] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.4): Upgrade Gerrit to 3.4.6 - https://phabricator.wikimedia.org/T319513 (10hashar) 05Open→03Resolved a:03hashar Both Gerrit instances are now running 3.4.6 [14:23:49] 10Gerrit, 10Data³, 10Patch-For-Review: Investigate sending Gerrit events to our data lake - https://phabricator.wikimedia.org/T304947 (10hashar) [14:24:54] Yippee, build fixed! [14:24:55] Project beta-code-update-eqiad build #412114: 09FIXED in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/412114/ [14:33:14] 10Gerrit: org.apache.http.client.protocol.ResponseProcessCookies : Invalid cookie header for WMF-Last-Access - https://phabricator.wikimedia.org/T273605 (10hashar) [14:33:41] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10WMF-General-or-Unknown, 10PHP 7.4 support: Make Wikimedia Production MediaWiki compatible with PHP 7.4 - https://phabricator.wikimedia.org/T247658 (10Lucas_Werkmeister_WMDE) [14:34:09] apparently I just closed the last two subtasks of “Make Wikimedia Production MediaWiki compatible with PHP 7.4” :D [14:34:34] 10Gerrit: org.apache.http.client.protocol.ResponseProcessCookies : Invalid cookie header for WMF-Last-Access - https://phabricator.wikimedia.org/T273605 (10hashar) [15:05:50] 10Continuous-Integration-Config: Upgrade, release, and use new PHP 7.4+ versions of all our home-grown CI libraries - https://phabricator.wikimedia.org/T319596 (10Daimona) [15:06:30] legoktm: Hi, do you think we could have LibUp update phan everywhere, or would it be too much given it's still catching up? [15:08:17] Daimona: AFAIK you can add things to its backlog and it'll get to it... [15:08:22] 10Continuous-Integration-Config: Upgrade, release, and use new PHP 7.4+ versions of all our home-grown CI libraries - https://phabricator.wikimedia.org/T319596 (10Jdforrester-WMF) [15:08:26] Or it'll start doing it as it touches relevant repos? [15:09:17] That was kind of my question. https://gerrit.wikimedia.org/r/c/labs/libraryupgrader/config/+/839590/ was merged, so LibUp should know about it already, I was just wondering if it should also be told to update it everywhere *now* [15:11:36] (To be precise, it still doesn't know about the new version because packagist also doesn't seem to know about it yet) [15:11:47] * Reedy looks at packagist [15:12:02] https://packagist.org/packages/mediawiki/phan-taint-check-plugin has 4.0.0. ? [15:12:40] https://packagist.org/packages/mediawiki/mediawiki-phan-config [15:12:48] 0.12.0 not there [15:13:29] https://github.com/wikimedia/mediawiki-phan-config/tags 0.12.0 is not there either [15:14:16] Whoa, and https://gerrit.wikimedia.org/r/c/mediawiki/tools/phan/+/839518 missing on github as well [15:14:25] I wonder if it coincides with the gerrit updates? [15:14:31] We can do a force replication [15:14:47] Ah, right, the gerrit updates... [15:17:30] * Reedy tries [15:19:38] it's taking a while to come back [15:19:44] so gerrit is possibly backlogged indeed [15:22:10] 10Gerrit, 10Release-Engineering-Team (Seen), 10Machine-Learning-Team: gerrit: scoring/ores/editquality takes a long time to git gc - https://phabricator.wikimedia.org/T237807 (10hashar) I ran the script today and it is still a thing: ` $ python /home/thcipriani/elapsed_gc_time.py |sort -rn|head 7:14:43 TOTAL... [15:22:19] hashar: is gerrit a bit slow/backlogged? [15:23:03] (03CR) 10Alexandros Kosiaris: [C: 03+1] Allow SRE to send annotated and signed tags [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/836711 (owner: 10Hashar) [15:23:44] 10Gerrit, 10Release-Engineering-Team (Seen), 10Machine-Learning-Team: gerrit: scoring/ores/editquality takes a long time to git gc - https://phabricator.wikimedia.org/T237807 (10hashar) [15:24:35] looking at monitoring, gerrit looks healthy, maybe this is rebuilding all its caches...or apache workers overload (rather than gerrit overload) [15:26:03] aha, that would make some sense [15:26:38] the replicaiton with --wait usually comes back within a minute or two, and it's not, so it's obviously doing something :) [15:26:46] oh [15:27:02] yeah, when we restart it re-runs all replication with a couple threads [15:27:17] so if you manual trigger it with --wait...you might be in the queue a while :) [15:27:36] thcipriani: Gerrit does a full replication on restart [15:27:46] hmm yeah what you said sorry :D [15:28:19] heh [15:28:21] LibUp should be hitting gerrit-replica [15:28:23] Daimona: Patience then :) [15:28:31] hashar: We're more waiting for gerrit -> github -> packagist [15:28:55] Sure :) [15:28:59] ssh -p 29418 gerrit.wikimedia.org -- gerrit show-queue --wide | grep -c push => 3504 [15:29:05] things are Happening™ [15:29:08] and https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream?orgId=1 [15:29:10] heh [15:29:16] gerrit-replica is the `replica_codfw` target [15:31:18] (03PS1) 10Ahmon Dancy: build-debug-image: Pass -y to apt-get install [tools/release] - 10https://gerrit.wikimedia.org/r/839607 [15:36:27] (03CR) 10Ahmon Dancy: [C: 03+2] build-debug-image: Pass -y to apt-get install [tools/release] - 10https://gerrit.wikimedia.org/r/839607 (owner: 10Ahmon Dancy) [15:36:31] 10Gerrit, 10Release-Engineering-Team (Seen), 10Machine-Learning-Team: gerrit: scoring/ores/editquality takes a long time to git gc - https://phabricator.wikimedia.org/T237807 (10hashar) [15:37:21] Daimona: Is there a nice way to tell PHPUnit to not whine about the deprecated at() calls for now, or do we have to re-write all the tests immediately to upgrade PHPUnit? [15:37:52] I'm not sure honestly... I don't think there's a nice way though. Let me see if I can rewrite them, the migration seems easy enough [15:38:03] Awesome. [15:38:03] (03Merged) 10jenkins-bot: build-debug-image: Pass -y to apt-get install [tools/release] - 10https://gerrit.wikimedia.org/r/839607 (owner: 10Ahmon Dancy) [15:38:20] 10Gerrit, 10Release-Engineering-Team (Seen), 10Machine-Learning-Team: gerrit: scoring/ores/editquality takes a long time to git gc - https://phabricator.wikimedia.org/T237807 (10elukey) Really sorry for the issue :( We are actively working on Lift Wing and we hope to deprecate any git-lfs usage in favor of s... [15:43:49] 10Gerrit, 10Release-Engineering-Team (Seen), 10Machine-Learning-Team: gerrit: scoring/ores/editquality takes a long time to git gc - https://phabricator.wikimedia.org/T237807 (10hashar) @elukey `scoring/ores/editquality.git` has a bunch of a large objects which somehow cause jGit gc to take four hours. I th... [16:03:32] 10Gerrit, 10Release-Engineering-Team (Seen), 10Machine-Learning-Team: gerrit: scoring/ores/editquality takes a long time to git gc - https://phabricator.wikimedia.org/T237807 (10hashar) From Luca: > You don't need to run the aggressive GC, ever. Unless the repo has massively changed its structure The gc ca... [16:14:32] Daimona: throwing it into the pile is fine, things are still somewhat broken, but it's always like that [17:11:43] replication delay upto 3 hours... [17:11:58] (for github) [17:13:26] but codfw caught up 40-45 mins ago... so progress [17:27:18] (03CR) 10Dzahn: [C: 03+1] Remove gerrit2001 from deployment targets [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/839551 (https://phabricator.wikimedia.org/T243027) (owner: 10Hashar) [18:03:51] legoktm: Thank you, that works [18:04:50] Aaaaand 0.12.0 now on packagist [18:06:07] yay [18:25:53] 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) [18:27:35] 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) [18:28:55] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) 05Open→03In progress p:05Triage→03High a:03dduvall [18:30:04] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) [18:51:46] Hey all I need some input on https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/838945 regarding the yaml files in our configuration [18:52:10] Is my understanding that the yaml files in wmf-config/config/ define configuration taht can be overriden in wmf-config/InitialiseSettings.php ? [18:52:21] If so I want to move a bunch of definitions into the yaml files [18:53:01] Second question I have is around how we redirect static images in mediawiki config - what's the best way to do that? I need to rename a few files, but presumably need to not break the internet by removing the old URL :) [18:54:57] For the second question, you can probably just use symlinks [18:54:58] e.g. I need https://en.m.wikipedia.org/static/images/mobile/copyright/meta.svg to redirect to https://en.m.wikipedia.org/static/images/mobile/metawiki.svg when the latter is added [18:55:20] Reedy: okay cool! That's what I was hoping. I saw a few symlinks in that folder but wasn't 100% sure. [18:55:48] It seems easier than creating a load of webserver side rewrites that we probably don't need [18:56:52] And we shouldn't need to keep the symlinks around for more than a few weeks (or a couple of months), presumably... [18:57:18] The cache length for static assets is quite long IIRC. [18:57:49] Jdlrobson: For the YAML files, I've lost track but I don't think they're properly used yet, sadly. [18:58:41] I guess T.imo would be an expert at this point? [18:59:55] https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream?orgId=1 [19:00:00] heh, that dropped off with a bump :) [19:00:19] (03PS1) 10Ahmon Dancy: blubber.yaml: Decouple the *-test and *-test images [tools/scap] - 10https://gerrit.wikimedia.org/r/839670 [19:00:48] I assume we finally synced the ORES mega-repo or something and it went from hrs to secs? [19:02:01] Wouldn't be surprised [19:02:04] (03PS2) 10Ahmon Dancy: blubber.yaml: Decouple the *-build and *-test images [tools/scap] - 10https://gerrit.wikimedia.org/r/839670 [19:02:40] I feel sorry for Microsoft's bandwidth bill when we re-sync. ;-) [19:03:00] Isn't git clever enough to not need to actually sync everything? [19:03:10] If only they let us programmatically set the 'mirror' flag rather than it being a thing only GH support can manually set on a per-repo basis. [19:03:20] I'd have thought so, but given it took hours not seconds… [19:03:32] Java, being efficient? [19:03:38] In this economy? [19:03:45] I guess, it's entirely possible jgit does something completely bizarre... [19:03:54] It's entirely guaranteed. [19:04:16] here, have ALL OF THE OBJECTS [19:28:31] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T314193 (10brennen) [19:33:34] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T314193 (10brennen) [20:07:54] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops, 10serviceops-collab, and 2 others: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall) [20:08:15] James_F: Reedy: the replication delay is reported when a replication task has completed. So the last one that got enqueued on gerrit restart took 5h00 to be processed and thus report a 5 hours delay [20:08:51] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10User-brennen: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token - https://phabricator.wikimedia.org/T308501 (10dduvall) 05In progress→03Resolved >>! In T30850... [20:09:01] maybe we can add more threads for the replication [20:13:18] Oooh. [20:13:36] Yeah, though I'd not worry too much about it given the move to GitLab. [20:24:51] James_F: https://gerrit.wikimedia.org/r/c/operations/puppet/+/839694 gerrit: use 2 threads to replicate to GitHub [20:27:45] (03PS1) 10Ahmon Dancy: Set up /srv/deployment/testrepo in the deploy container [tools/train-dev] - 10https://gerrit.wikimedia.org/r/839697 [21:38:06] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) [23:19:29] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T314193 (10brennen) End-of-workday status: Still blocked but some progress on blocker tickets. We'll make a judgment call tomorrow, but... [23:35:56] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) [23:51:11] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/1 [23:51:23] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) [23:52:19] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall)