[00:15:43] (03PS1) 10Jforrester: Zuul: [mediawiki/services/function-evaluator] Use direct coverage job [integration/config] - 10https://gerrit.wikimedia.org/r/835716 (https://phabricator.wikimedia.org/T302608) [00:18:13] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/services/function-evaluator] Use direct coverage job [integration/config] - 10https://gerrit.wikimedia.org/r/835716 (https://phabricator.wikimedia.org/T302608) (owner: 10Jforrester) [00:20:05] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/function-evaluator] Use direct coverage job [integration/config] - 10https://gerrit.wikimedia.org/r/835716 (https://phabricator.wikimedia.org/T302608) (owner: 10Jforrester) [00:20:19] !log Zuul: [mediawiki/services/function-evaluator] Use direct coverage job, for T302608 [00:20:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:20:21] T302608: Publish function-evaluator's test code coverage - https://phabricator.wikimedia.org/T302608 [00:27:30] (03PS1) 10Jforrester: Zuul: [mediawiki/services/function-orchestrator] Use direct coverage job [integration/config] - 10https://gerrit.wikimedia.org/r/835718 [00:27:49] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/services/function-orchestrator] Use direct coverage job [integration/config] - 10https://gerrit.wikimedia.org/r/835718 (owner: 10Jforrester) [00:29:41] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/function-orchestrator] Use direct coverage job [integration/config] - 10https://gerrit.wikimedia.org/r/835718 (owner: 10Jforrester) [00:30:39] !log Zuul: [mediawiki/services/function-orchestrator] Use direct coverage job here too [00:30:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:01:24] (03CR) 10Hashar: "recheck Blubberoid got updated by https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/835660 and if I understand it properly " [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [08:07:12] (03CR) 10Hashar: Define requirements.txt exhaustively (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [08:09:08] (03CR) 10Hashar: Define requirements.txt exhaustively (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [08:10:51] (03PS3) 10Hashar: Define requirements.txt exhaustively [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) [08:14:16] (03CR) 10CI reject: [V: 04-1] Define requirements.txt exhaustively [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [08:20:55] 10GitLab, 10Release-Engineering-Team, 10serviceops, 10serviceops-collab, 10Patch-For-Review: Disable email notifications from GitLab replicas - https://phabricator.wikimedia.org/T318682 (10Jelto) 05Open→03Resolved Should be fixed now. Feel free to re-open if you find any more mails from the replica. [08:35:44] (03PS4) 10Hashar: Define requirements.txt exhaustively [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) [08:53:34] (03PS1) 10Hashar: (DO NOT MERGE) test with incomplete requirements.txt [tools/scap] - 10https://gerrit.wikimedia.org/r/836129 (https://phabricator.wikimedia.org/T316755) [08:55:29] (03CR) 10CI reject: [V: 04-1] (DO NOT MERGE) test with incomplete requirements.txt [tools/scap] - 10https://gerrit.wikimedia.org/r/836129 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [08:55:32] (03CR) 10Hashar: [C: 03+1] "I have made some further adjustments:" [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [08:56:19] (03CR) 10Hashar: [C: 04-2] "This is solely for asserting the parent change (https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/835193/) is behaving as expected:" [tools/scap] - 10https://gerrit.wikimedia.org/r/836129 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [08:57:10] (03CR) 10Jaime Nuche: [C: 03+2] Add release-scripts/perform-release [tools/scap] - 10https://gerrit.wikimedia.org/r/835238 (owner: 10Ahmon Dancy) [08:58:19] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Patch-For-Review: Strictly lock scap requirements dependencies - https://phabricator.wikimedia.org/T316755 (10hashar) a:03hashar >>! In T316755#8202296, @Lucas_Werkmeister_WMDE wrote: >> We would probably want to add the packages checksum as well,... [09:01:24] (03Merged) 10jenkins-bot: Add release-scripts/perform-release [tools/scap] - 10https://gerrit.wikimedia.org/r/835238 (owner: 10Ahmon Dancy) [10:32:23] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10User-brennen: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token - https://phabricator.wikimedia.org/T308501 (10Jelto) I tried to troubleshoot what's happening on... [11:47:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review: The python-build images regenerate wheels even when matching ones are already available - https://phabricator.wikimedia.org/T259611 (10hashar) 05Declin... [11:47:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Zuul, 10Patch-For-Review, and 2 others: Improve scheduling of CI jobs invoked by zuul - https://phabricator.wikimedia.org/T258630 (10hashar) [12:28:55] 10Project-Admins: Migrate mwbot-rs tasks to Phabricator - https://phabricator.wikimedia.org/T318069 (10taavi) > And tasks can either be in the top level "mwbot-rs" project, or in *one* of the subprojects (but not multiple), correct? AIUI a task can be in the parent project, or *multiple* subprojects, but not bot... [14:56:33] (03CR) 10Ahmon Dancy: "Nice work" [tools/scap] - 10https://gerrit.wikimedia.org/r/836129 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [15:28:37] (03CR) 10Hashar: [C: 03+1] "+ Dan for the PipelineLib magic :]" [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [15:28:50] (03PS1) 10Giuseppe Lavagetto: Revert "Revert "make-container-image: switch to php7.4"" [tools/release] - 10https://gerrit.wikimedia.org/r/836195 [15:32:23] (03PS2) 10Giuseppe Lavagetto: Revert "Revert "make-container-image: switch to php7.4"" [tools/release] - 10https://gerrit.wikimedia.org/r/836195 [15:35:13] (03PS3) 10Giuseppe Lavagetto: Revert "Revert "make-container-image: switch to php7.4"" [tools/release] - 10https://gerrit.wikimedia.org/r/836195 [15:44:51] (03CR) 10Ahmon Dancy: "On my system when I run `make test-buster` I get:" [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [15:46:55] (03PS1) 10Ahmon Dancy: Release 4.24.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/836246 [15:46:57] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.24.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/836246 (owner: 10Ahmon Dancy) [15:48:14] (03Abandoned) 10Ahmon Dancy: Release 4.24.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/836246 (owner: 10Ahmon Dancy) [15:49:21] (03PS1) 10Ahmon Dancy: perform-release bugfix [tools/scap] - 10https://gerrit.wikimedia.org/r/836249 [15:49:40] (03CR) 10Ahmon Dancy: [C: 03+2] perform-release bugfix [tools/scap] - 10https://gerrit.wikimedia.org/r/836249 (owner: 10Ahmon Dancy) [15:52:47] (03Merged) 10jenkins-bot: perform-release bugfix [tools/scap] - 10https://gerrit.wikimedia.org/r/836249 (owner: 10Ahmon Dancy) [15:53:21] (03PS1) 10Ahmon Dancy: Release 4.24.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/836250 [15:53:23] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.24.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/836250 (owner: 10Ahmon Dancy) [15:56:42] (03Merged) 10jenkins-bot: Release 4.24.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/836250 (owner: 10Ahmon Dancy) [15:58:48] 10Release-Engineering-Team, 10Scap: `scap backport` should gracefully handle a single error when polling for patch merge - https://phabricator.wikimedia.org/T318713 (10dancy) 05Open→03Resolved Included in scap 4.24.0 which was deployed today. [15:59:05] 10Release-Engineering-Team, 10Scap: `scap backport` should display branch in the patch table - https://phabricator.wikimedia.org/T318711 (10dancy) 05Open→03Resolved Included in scap 4.24.0 which was deployed today. [16:08:50] (03PS1) 10Ahmon Dancy: RELEASE.md: Use release-scripts/perform-release [tools/scap] - 10https://gerrit.wikimedia.org/r/836257 [16:17:21] (03CR) 10Hashar: [C: 03+1] Define requirements.txt exhaustively (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [16:18:54] (03CR) 10Hashar: [C: 03+1] Define requirements.txt exhaustively (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [16:19:39] (03PS5) 10Hashar: Define requirements.txt exhaustively [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) [16:27:23] (03PS6) 10Hashar: Define requirements.txt exhaustively [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) [16:27:50] (03CR) 10Hashar: "I forgot a --quiet when building the test image. Sorry about the spam :-\" [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [16:28:12] (03PS2) 10Hashar: (DO NOT MERGE) test with incomplete requirements.txt [tools/scap] - 10https://gerrit.wikimedia.org/r/836129 (https://phabricator.wikimedia.org/T316755) [16:32:14] (03CR) 10CI reject: [V: 04-1] (DO NOT MERGE) test with incomplete requirements.txt [tools/scap] - 10https://gerrit.wikimedia.org/r/836129 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [16:34:44] (03CR) 10Ahmon Dancy: "Image build test passed: https://releases-jenkins.wikimedia.org/view/incremental%20image%20build/job/build-mw-container-image/3513/console" [tools/release] - 10https://gerrit.wikimedia.org/r/836195 (owner: 10Giuseppe Lavagetto) [17:01:34] dduvall: I have a "fun" blubber-buildkit bug. https://gerrit.wikimedia.org/r/c/wikimedia/developer-portal/+/835289 causes a "qemu: uncaught target signal 11 (Segmentation fault) - core dumped" crash during a COPY instruction on two different M1 Macs. The static Dockerfile removed by the same patch works fine on those exact same macs. [17:01:35] M1: MediaWiki Userpage - https://phabricator.wikimedia.org/M1 [17:02:01] dduvall: any idea where/how I should try to figure out WTF is causing that? [17:07:45] Sounds like you found a bug in qemu [17:07:54] bd808: oh that does sound "fun" [17:08:06] we got a Phabricator tag dedicated to Apple M1 / ARM issues https://phabricator.wikimedia.org/project/view/6112/ [17:08:06] M1: MediaWiki Userpage - https://phabricator.wikimedia.org/M1 [17:08:16] and yeah that sounds like a qemu maybe you get a core file to be inspected? [17:08:50] yeah, it seems to be similar to https://github.com/docker/for-mac/issues/5123 which was basically closed as "upstream" [17:09:35] on x86 we had issue with nodejs/qemu as well [17:10:02] emulating amd64 on arm64 is going to keep being a pita [17:10:09] nod [17:11:05] (03CR) 10Ahmon Dancy: [C: 03+2] Revert "Revert "make-container-image: switch to php7.4"" [tools/release] - 10https://gerrit.wikimedia.org/r/836195 (owner: 10Giuseppe Lavagetto) [17:11:20] "Please encourage the author of this container to supply an arm64 or multi-arch image, not just an Intel one. Now that M1 is a mainstream platform, we think that most container authors will be keen to do this." -- https://github.com/docker/for-mac/issues/5123#issuecomment-784992589 [17:11:51] * bd808 has seen the wontfix close on this question in phab [17:11:54] (03Merged) 10jenkins-bot: Revert "Revert "make-container-image: switch to php7.4"" [tools/release] - 10https://gerrit.wikimedia.org/r/836195 (owner: 10Giuseppe Lavagetto) [17:12:05] and the parent is https://phabricator.wikimedia.org/T272500 [17:12:45] which boils down to if we have some of our custom packages required in the images we need to build them for arm as well [17:12:52] so the status quo is requiring x86 [17:13:42] anyway it might be worth filing a task for us with the qemu version used and some details [17:13:53] and if there is a core file maybe a stack trace can be retrieved from it? [17:14:07] or attempt to use a newer qemu if that is at all possible, maybe the issue got fixed [17:14:11] (03CR) 10Ahmon Dancy: [C: 03+2] Define requirements.txt exhaustively [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [17:14:22] dancy: <3 [17:15:14] patching Docker Desktop sounds un-fun [17:15:19] haha [17:15:48] https://phabricator.wikimedia.org/T284696 was Qemu preventing us from updating from nodejs 10 to 12 [17:15:51] solved by upgrading Qemu [17:15:59] but yeah sorry for Docker Desktop :-\ [17:17:12] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥): Establish image trust system for GitLab/Blubber - https://phabricator.wikimedia.org/T307541 (10dduvall) 05Open→03Declined Out of scope for GitLab migration. We have trusted runners. They will be able to publish to the registry u... [17:17:18] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10dduvall) [17:17:22] (03Merged) 10jenkins-bot: Define requirements.txt exhaustively [tools/scap] - 10https://gerrit.wikimedia.org/r/835193 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [17:26:38] Thanks for the ideas about chasing into qemu folks. I'll see if I can make time to poke that deeply. I might also try building an arm64 native blubber-buildkit image just to see if that makes any difference. [17:28:21] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10dduvall) [17:30:04] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥): Build Blubber test variant on GitLab untrusted runners - https://phabricator.wikimedia.org/T307536 (10dduvall) [17:32:03] !log trying a re-publish of dev-images in case https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/eb82162b4bf443df20998a53bfb06460bfc6a365 didn't get picked up [17:32:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:54:58] bd808: oh, so we'd need an arm64 blubber-buildkit image? [17:55:20] that might not be so difficult since it's based on a scratch image [17:55:43] so no dependency on wmf having an arm64 debian base image or anything like that [18:11:39] (03PS1) 10Dduvall: [EXPERIMENTAL] Build darwin/arm64 buildkit frontend image [blubber] - 10https://gerrit.wikimedia.org/r/836264 [18:11:52] bd808: ^ [18:52:37] (03CR) 10Jeena Huneidi: [C: 03+2] Parallelize gerrit's git repo import [tools/train-dev] - 10https://gerrit.wikimedia.org/r/835708 (owner: 10Dduvall) [18:53:10] (03Merged) 10jenkins-bot: Parallelize gerrit's git repo import [tools/train-dev] - 10https://gerrit.wikimedia.org/r/835708 (owner: 10Dduvall) [18:59:13] (03CR) 10Jeena Huneidi: [C: 03+1] scap deploy-promote should retry when verifying Special:Version [tools/scap] - 10https://gerrit.wikimedia.org/r/835675 (https://phabricator.wikimedia.org/T316692) (owner: 10Ahmon Dancy) [19:34:57] (03PS2) 10Dduvall: [EXPERIMENTAL] Support Docker cross-platform builds [blubber] - 10https://gerrit.wikimedia.org/r/836264 [19:40:55] maintenance-disconnect-full-disks build 425481 integration-agent-docker-1024 (/: 28%, /srv: 99%, /var/lib/docker: 69%): OFFLINE due to disk space [19:45:37] maintenance-disconnect-full-disks build 425482 integration-agent-docker-1024 (/: 28%, /srv: 74%, /var/lib/docker: 69%): RECOVERY disk space OK [20:11:31] (03CR) 10Jforrester: "Tag against T272500 or T274140?" [blubber] - 10https://gerrit.wikimedia.org/r/836264 (owner: 10Dduvall) [20:25:34] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Upstream, 10ci-test-error, 10php-segfault: CI job mediawiki-quibble-composer-mysql-php80-docker on mediawiki/core gate-and-submit is flaky failing with Segmentation fault - https://phabricator.wikimedia.org/T315167 (10Jdforrester-WMF) 05... [20:25:36] 10Continuous-Integration-Config, 10PHP 8.0 support: Make PHP 8.0 voting on MW master - https://phabricator.wikimedia.org/T300463 (10Jdforrester-WMF) [20:30:43] (03CR) 10BryanDavis: [EXPERIMENTAL] Support Docker cross-platform builds (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/836264 (owner: 10Dduvall) [20:40:07] (03PS1) 10Hashar: install-world: ensure a clear venv [tools/scap] - 10https://gerrit.wikimedia.org/r/836288 [20:40:09] (03PS1) 10Hashar: install-world: install only declared requirements [tools/scap] - 10https://gerrit.wikimedia.org/r/836289 (https://phabricator.wikimedia.org/T316755) [20:42:21] (03CR) 10Hashar: "When running install_local_version.sh we pill requirements in the scap virtualenv so that removed requirements are still present in the ta" [tools/scap] - 10https://gerrit.wikimedia.org/r/836288 (owner: 10Hashar) [20:46:30] (03CR) 10Hashar: "The continuity of not processing transient dependencies." [tools/scap] - 10https://gerrit.wikimedia.org/r/836289 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [21:42:13] 10Phabricator (Upstream): Deleting a file server-side results in exception: Unknown edge constant "26"! at PhabricatorEdgeType.php:214 - https://phabricator.wikimedia.org/T318547 (10Dylsss) [21:51:17] maintenance-disconnect-full-disks build 425507 integration-agent-docker-1024 (/: 28%, /srv: 96%, /var/lib/docker: 69%): OFFLINE due to disk space [21:55:54] maintenance-disconnect-full-disks build 425508 integration-agent-docker-1024 (/: 28%, /srv: 81%, /var/lib/docker: 69%): RECOVERY disk space OK [21:57:29] (03CR) 10Ahmon Dancy: install-world: ensure a clear venv (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/836288 (owner: 10Hashar) [22:03:05] (03CR) 10Ahmon Dancy: [C: 03+1] "OK w/ me. I'm leaving final approval to Jaime." [tools/scap] - 10https://gerrit.wikimedia.org/r/836289 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [22:21:33] (03CR) 10Dduvall: "I didn't have time to do as exhaustive a review as I would have liked before taking my vacation, but I hope my comments so far will be hel" [blubber] - 10https://gerrit.wikimedia.org/r/825708 (owner: 10Jaime Nuche) [22:29:18] 10MediaWiki-Releasing, 10MW-1.39-notes, 10MW-1.39-release: Release MW 1.39.0 - https://phabricator.wikimedia.org/T313916 (10Jdforrester-WMF) [22:29:54] 10MediaWiki-Releasing, 10Documentation, 10MW-1.39-notes, 10MW-1.39-release, 10Patch-For-Review: Finish up RELEASE-NOTES for 1.39.0 - https://phabricator.wikimedia.org/T313922 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [23:04:49] (03PS1) 10Dduvall: buildkit: Support builds for specific target platforms [blubber] - 10https://gerrit.wikimedia.org/r/836307 (https://phabricator.wikimedia.org/T318866) [23:11:16] (03PS2) 10Dduvall: buildkit: Support builds for specific target platforms [blubber] - 10https://gerrit.wikimedia.org/r/836307 (https://phabricator.wikimedia.org/T318866) [23:11:18] (03PS3) 10Dduvall: buildkit: Support target platform in Makefile [blubber] - 10https://gerrit.wikimedia.org/r/836264 (https://phabricator.wikimedia.org/T318866)