[00:28:31] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10SRE, and 2 others: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183 (10Quiddity) >>! In T191183#8250010, @kostajh wrote: >>>! In T191183#8249977, @hashar wrote: >> Maybe if one day MediaWiki... [00:33:20] looks like zuul is still not working for Echo's for some reason [00:46:11] Jdlrobson: you may need to recheck the patch [01:56:26] 10Project-Admins: Create project tag for pwangai - https://phabricator.wikimedia.org/T318087 (10Peachey88) 05Open→03Resolved a:03Peachey88 @pwangai, Personal user tag/project has been created has been created #User-pwangai. [06:01:21] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TThoabala - https://phabricator.wikimedia.org/T317734 (10ArielGlenn) This training on the 15th was done. We hope to see our new deployer at many other windows (and trainings too if they want)! [08:06:31] (03CR) 10Hashar: "Next patch addresses the easy feedbacks. I will create another patch to fully cover DeployLocal._valid_chk() and rebase on top of it, this" [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [08:10:12] (03PS6) 10Hashar: Support checks running before a stage [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 [10:38:13] !log deployment-db10: start replication # T318126 [10:38:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:38:16] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 [11:00:02] (03PS1) 10Hashar: Make DeployLocal._valid_chk() static + test coverage [tools/scap] - 10https://gerrit.wikimedia.org/r/834012 [11:04:11] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Quota-requests): Request increased quota for deployment-prep Cloud VPS project - https://phabricator.wikimedia.org/T318287 (10rook) Hello! What volume count would be desired? [12:40:55] (03CR) 10Hashar: Support checks running before a stage (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [12:41:02] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Quota-requests): Request increased quota for deployment-prep Cloud VPS project - https://phabricator.wikimedia.org/T318287 (10Zabe) >>! In T318287#8252789, @rook wrote: > Hello! What volume count would be desired? very rough lists of hosts that likely need a volume... [12:41:06] (03PS7) 10Hashar: Support checks running before a stage [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 [12:42:49] (03CR) 10Hashar: "This should fully cover the boolean logic in DeployLocal._valid_chk() which I refactor in the child change "Support checks running before " [tools/scap] - 10https://gerrit.wikimedia.org/r/834012 (owner: 10Hashar) [12:55:01] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Quota-requests): Request increased quota for deployment-prep Cloud VPS project - https://phabricator.wikimedia.org/T318287 (10rook) Done! `root@cloudcontrol1005:~# wmcs-openstack quota set --volumes 22 deployment-prep` [12:55:20] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Quota-requests): Request increased quota for deployment-prep Cloud VPS project - https://phabricator.wikimedia.org/T318287 (10rook) 05Open→03Resolved a:03rook [12:57:05] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Quota-requests): Request increased quota for deployment-prep Cloud VPS project - https://phabricator.wikimedia.org/T318287 (10Zabe) Thanks! [13:53:32] 10GitLab (Project Migration): Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [14:02:27] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [14:13:40] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [14:16:37] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [14:54:30] https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/170687/consoleFull on integration-agent-docker-1024 failed with ENOSPC, reporting it in case it needs manual cleanup or something [14:56:18] I'll take a look [15:00:29] Looks like there's decent free space now. [15:07:20] ok, thanks [15:30:50] (03CR) 10Jaime Nuche: [C: 04-1] "Verified in traindev using a slightly modified version. Overall LGTM, a few smaller comments" [tools/scap] - 10https://gerrit.wikimedia.org/r/832338 (https://phabricator.wikimedia.org/T318181) (owner: 10Ahmon Dancy) [15:50:37] Hey all there's some issue with Jenkins bot on this patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Echo/+/832380 [15:50:55] it's not responding to new patches/+2s or recheck command [15:51:03] Does anyone know what might be happening here? [15:51:15] let me check Jdlrobson ;) [15:51:16] I'll take a stab at investigating [15:51:21] oh, hashar will .. even better [15:52:33] oh I am not selfish, we can do it all together ;] [15:54:14] hmm [15:54:25] 2022-09-22 15:50:23,351 DEBUG zuul.source.Gerrit: Updating : Running query message:I9fe8c921956ede164d93cb1f42647cf4b80ca976 to find changes needed-by [15:54:25] 2022-09-22 15:50:23,538 DEBUG zuul.source.Gerrit: Updating : Found change 832381,3 needs I9fe8c921956ede164d93cb1f42647cf4b80ca976 from commit [15:54:55] so when Zuul process that Echo patchset 832380,7 it does a query to gerrit with message: [15:55:34] which yields https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/832381/ which has a `Depends-On: I9fe8c921956ede164d93cb1f42647cf4b80ca976` aka the change id of the Echo patchset [15:56:08] Sounds reasonable so far. [15:56:41] that Vector change is open and has a parent change which is opened [15:56:46] so maybe that somehow confuses Zuul [15:57:12] I don't know why Zuul queries for `message:` [15:57:28] I was just wondering that.. why does it care about what depends on the commit. [15:57:38] (unless it is looking for loops) [15:57:41] OH [15:57:44] GOT IT [15:57:47] I do see some loop complaints in error.log [15:57:50] Jdlrobson: there is a dependency cycle [15:58:01] and Zuul does not report anything in this case [15:58:05] let me paste [15:58:36] https://phabricator.wikimedia.org/P34899 [16:02:41] hmm. [16:04:08] I pasted the explanation [16:04:20] in short that Echo change is needed by a change to Vector which itself is in a dependency cycle [16:04:28] Zuul should be smart enough to process the Echo change [16:04:40] nod.. that seems like a bug to me. [16:04:45] but part of the initial processing involves looking for any change that might depends-on that Echo change [16:05:07] and in this case, the Vector change is in a cycle which erroneously prevent the Echo change from entering [16:05:09] it is [16:05:18] Let's fix it! [16:05:27] it should catch the dependency loop processing error, log it and keep moving [16:06:08] Jdlrobson: in short there is a dependency loop between 3 changes I have replied on the paste with the loop and links to each of the changes https://phabricator.wikimedia.org/P34899 ;) [16:06:38] dancy: that is where I have a funky rabit hole :) [16:06:45] Zuul is still deployed using a Debian package [16:06:56] but surely it could be converted to use scap for deployment, exactly like we are doing for scap [16:07:16] the stop over so far has always been "how do we ship dependencies" but that apparently got solved for scap [16:07:25] err s/apparently// [16:08:23] nod [16:08:40] jeena: about the Zuul restart yesterday, the Echo change could not be processed due to a dependency loop between changes [16:09:13] jeena: the loop is silently ignored by Zuul so everyone ends up wondering why nothing happens on the change [16:18:14] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Scap: Refactor Scap to use TimeoutLock as the sole locking mechanism - https://phabricator.wikimedia.org/T315531 (10dancy) 05In progress→03Resolved [16:21:24] (03CR) 10Dduvall: Use Blubber BuildKit frontend instead of blubberoid (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833863 (https://phabricator.wikimedia.org/T318290) (owner: 10Dduvall) [16:23:02] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Zuul: Interpret extension dependencies as mutual - https://phabricator.wikimedia.org/T318107 (10hashar) James told me about this task though I can't currently dig into it :-\ A related case we found today i... [16:23:14] (03PS5) 10Ahmon Dancy: Perform k8s deployments in parallel [tools/scap] - 10https://gerrit.wikimedia.org/r/832338 (https://phabricator.wikimedia.org/T318181) [16:24:05] (03CR) 10Ahmon Dancy: Perform k8s deployments in parallel (036 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/832338 (https://phabricator.wikimedia.org/T318181) (owner: 10Ahmon Dancy) [16:26:32] hashar: wowser! [16:26:43] !log Upgrading scap to latest code revision in beta cluster [16:26:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:27:24] (03PS2) 10Dduvall: Use Blubber BuildKit frontend instead of blubberoid [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833863 (https://phabricator.wikimedia.org/T318290) [16:27:26] (03CR) 10Hashar: [C: 04-1] "I ran out of time today and did not manage to do the tests above. The whole series is not blocker anything anyway so it is not a rush. I " [blubber] - 10https://gerrit.wikimedia.org/r/833778 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [16:27:48] (03CR) 10Dduvall: Use Blubber BuildKit frontend instead of blubberoid (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833863 (https://phabricator.wikimedia.org/T318290) (owner: 10Dduvall) [16:28:21] I am off, thanks for the reviews talks and such [16:28:32] Have a good night! [16:28:48] (03CR) 10Ahmon Dancy: [C: 03+2] Use Blubber BuildKit frontend instead of blubberoid [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833863 (https://phabricator.wikimedia.org/T318290) (owner: 10Dduvall) [16:30:05] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Epic, 10Release Pipeline (Blubber): Deprecate Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) [16:30:23] (03Merged) 10jenkins-bot: Use Blubber BuildKit frontend instead of blubberoid [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833863 (https://phabricator.wikimedia.org/T318290) (owner: 10Dduvall) [16:41:46] (03PS1) 10Ahmon Dancy: Release 4.22.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/834354 [16:41:48] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.22.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/834354 (owner: 10Ahmon Dancy) [16:44:28] (03CR) 10CI reject: [V: 04-1] Release 4.22.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/834354 (owner: 10Ahmon Dancy) [16:50:30] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Epic, 10Release Pipeline (Blubber): Deprecate Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) I've outlined what it would take to deprecate the CLI and blubberoid, but it's important to note this... [17:00:11] (03PS1) 10Dduvall: Revert "Use Blubber BuildKit frontend instead of blubberoid" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833820 [17:00:55] (03CR) 10CI reject: [V: 04-1] Revert "Use Blubber BuildKit frontend instead of blubberoid" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833820 (owner: 10Dduvall) [17:01:07] (03CR) 10Ahmon Dancy: [C: 03+2] Revert "Use Blubber BuildKit frontend instead of blubberoid" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833820 (owner: 10Dduvall) [17:02:05] (03CR) 10CI reject: [V: 04-1] Revert "Use Blubber BuildKit frontend instead of blubberoid" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833820 (owner: 10Dduvall) [17:02:30] (03CR) 10Dduvall: "We'll have to manually submit this due to the self-blocking failure." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833820 (owner: 10Dduvall) [17:02:42] (03CR) 10Dduvall: [V: 03+2 C: 03+2] Revert "Use Blubber BuildKit frontend instead of blubberoid" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833820 (owner: 10Dduvall) [17:02:58] 10MediaWiki-Releasing, 10Documentation, 10MW-1.39-release: Write release announcement for 1.39.0-rc.0 - https://phabricator.wikimedia.org/T313924 (10Reedy) 05Open→03In progress p:05Triage→03Medium a:03Reedy [17:03:02] 10MediaWiki-Releasing, 10MW-1.39-notes, 10MW-1.39-release: Release 1.39.0-rc.0 - https://phabricator.wikimedia.org/T313923 (10Reedy) [17:04:18] 10MediaWiki-Releasing, 10Documentation, 10MW-1.39-release: Write release announcement for 1.39.0-rc.0 - https://phabricator.wikimedia.org/T313924 (10Reedy) [17:05:51] 10MediaWiki-Releasing, 10Documentation, 10MW-1.39-release: Write release announcement for 1.39.0-rc.0 - https://phabricator.wikimedia.org/T313924 (10Reedy) [17:33:02] 10MediaWiki-Releasing, 10Documentation, 10MW-1.39-release: Write release announcement for 1.39.0-rc.0 - https://phabricator.wikimedia.org/T313924 (10Reedy) Need to get the number of commits since REL1_38 was branched [17:35:52] 10MediaWiki-Releasing, 10Documentation, 10MW-1.39-release: Write release announcement for 1.39.0-rc.0 - https://phabricator.wikimedia.org/T313924 (10Jdforrester-WMF) git log --oneline --no-merges --topo-order origin/REL1_38..origin/REL1_39 | wc -l 1851 [18:13:28] (03PS1) 10Dduvall: Avoid BuildKit build failure due to non-headless credential helpers [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/834363 [18:13:35] (03CR) 10CI reject: [V: 04-1] Avoid BuildKit build failure due to non-headless credential helpers [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/834363 (owner: 10Dduvall) [18:13:52] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.22.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/834354 (owner: 10Ahmon Dancy) [18:14:59] (03CR) 10Dduvall: "Note the old base is intentional to test this fix against the introduction of the Blubber buildkit frontend." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/834363 (owner: 10Dduvall) [18:17:51] (03Merged) 10jenkins-bot: Release 4.22.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/834354 (owner: 10Ahmon Dancy) [18:33:10] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Scap: Refactor Scap to use TimeoutLock as the sole locking mechanism - https://phabricator.wikimedia.org/T315531 (10dancy) [20:25:04] !log set deployment-prep as read-only # T318126 [20:25:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:25:06] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 [20:28:33] 10Release-Engineering-Team (Seen), 10Scap: Make "scap backport" skip syncing steps for labs-only changes - https://phabricator.wikimedia.org/T317242 (10dancy) 05Open→03Resolved a:03dancy Included in scap 4.22.0 which was deployed today and the beta-only workflow was verified. [20:32:38] !log failover deployment-prep master from deployment-db07 to deployment-db09 # T318126 [20:32:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:32:40] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 [20:36:35] !log take deployment-prep out of read-only # T318126 [20:36:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:36:53] Project beta-update-databases-eqiad build #61685: 04FAILURE in 16 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/61685/ [20:40:27] !log shutoff deployment-db07 # T318126 [20:40:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:40:29] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 [20:46:55] Yippee, build fixed! [20:46:55] Project beta-update-databases-eqiad build #61686: 09FIXED in 8 min 26 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/61686/ [21:22:15] 10Beta-Cluster-Infrastructure, 10SRE, 10Technical-Debt, 10Tracking-Neverending: Minimize infrastructure differences between Beta Cluster and production - https://phabricator.wikimedia.org/T87220 (10Zabe) [21:22:55] 10Beta-Cluster-Infrastructure: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 (10Zabe) 05Open→03Resolved a:03Zabe Both hosts are now running MariaDB 10.6 on Bullseye. [22:47:54] (03CR) 10Dduvall: [C: 04-2] "I can confirm that this fixes the credential helper related issue. Replaying a recent pipelinelib-pipeline-test job on contint1001 against" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/834363 (owner: 10Dduvall) [22:48:44] (03PS2) 10Dduvall: Avoid BuildKit build failure due to non-headless credential helpers [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/834363 (https://phabricator.wikimedia.org/T318290) [22:48:52] (03CR) 10CI reject: [V: 04-1] Avoid BuildKit build failure due to non-headless credential helpers [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/834363 (https://phabricator.wikimedia.org/T318290) (owner: 10Dduvall) [22:56:38] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Epic, 10Release Pipeline (Blubber): Upgrade docker on integration hosts for fixes to BuildKit builder - https://phabricator.wikimedia.org/T318382 (10dduvall) [22:56:55] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Epic, 10Release Pipeline (Blubber): Deprecate Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) [22:56:57] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Epic, 10Patch-For-Review, 10Release Pipeline (Blubber): Refactor PipelineLib to use blubber's buildkit frontend - https://phabricator.wikimedia.org/T318290 (10dduvall) 05In progress→03Stalled [22:57:10] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Epic, 10Release Pipeline (Blubber): Upgrade docker on integration hosts for fixes to BuildKit builder - https://phabricator.wikimedia.org/T318382 (10dduvall) p:05Triage→03Medium