[05:18:00] dancy: I really like your reviews :] [07:15:42] (03CR) 10Jaime Nuche: RELEASE.md: Use release-scripts/perform-release (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/836257 (owner: 10Ahmon Dancy) [07:23:06] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Patch-For-Review, 10Release Pipeline (Blubber): Upgrade docker on integration hosts for fixes to BuildKit builder - https://phabricator.wikimedia.org/T318382 (10MoritzMuehlenhoff) I've merged the patch and imported the following to thirdparty/ci fo... [07:34:37] (03Abandoned) 10Hashar: (DO NOT MERGE) test with incomplete requirements.txt [tools/scap] - 10https://gerrit.wikimedia.org/r/836129 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [07:53:21] (03CR) 10Hashar: install-world: ensure a clear venv (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/836288 (owner: 10Hashar) [07:56:21] (03PS1) 10Matthias Mullie: Remove extension-coverage for SearchVue [integration/config] - 10https://gerrit.wikimedia.org/r/836696 [07:57:50] (03CR) 10Hashar: install-world: ensure a clear venv (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/836288 (owner: 10Hashar) [07:57:56] (03PS2) 10Hashar: install-world: ensure a clear venv [tools/scap] - 10https://gerrit.wikimedia.org/r/836288 [08:22:56] (03CR) 10Jaime Nuche: [C: 03+2] "Verified in traindev with a slightly modified version" [tools/scap] - 10https://gerrit.wikimedia.org/r/835675 (https://phabricator.wikimedia.org/T316692) (owner: 10Ahmon Dancy) [08:27:56] (03Merged) 10jenkins-bot: scap deploy-promote should retry when verifying Special:Version [tools/scap] - 10https://gerrit.wikimedia.org/r/835675 (https://phabricator.wikimedia.org/T316692) (owner: 10Ahmon Dancy) [08:44:09] (03PS1) 10Hashar: Review access change [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/836711 [08:46:34] (03CR) 10Hashar: "operations/puppet is the parent repository of most if not all SRE maintained repositories. It lacked the rights for SRE to push annotated" [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/836711 (owner: 10Hashar) [08:48:02] (03CR) 10Clément Goubert: "We saw that there was no way to push annotated or signed tags to our repos, can you check this out?" [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/836711 (owner: 10Hashar) [08:58:52] 10Release-Engineering-Team, 10docker-pkg, 10serviceops: docker-pkg / docker downloads all versions of parent image upon building - https://phabricator.wikimedia.org/T310458 (10Clement_Goubert) Actions needed : [X] Push 3.0.3 tag pointing to 66b22ed50 Release 3.0.3 [] Update submodule in docker-pkg-deploy [10:03:37] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10DBA, 10serviceops, and 2 others: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Marostegui) @Dzahn I think this is now sorted. It was a big mess, so if you can do some good double checking I would... [10:08:05] (03PS1) 10Jbond: dockerfiles/operations-puppet: bump changelog to pick up new changes [integration/config] - 10https://gerrit.wikimedia.org/r/836744 [10:08:35] hashar: can you shepered ^^ to pick up your recent puppet-strings change [10:27:00] 10MediaWiki-Releasing, 10serviceops, 10PHP 7.2 support, 10PHP 7.3 support, 10Patch-For-Review: Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4 - https://phabricator.wikimedia.org/T261872 (10taavi) Policy-required wikitech-l message: https://lists.wikimedia.org... [10:47:54] (03CR) 10Jaime Nuche: install-world: ensure a clear venv (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/836288 (owner: 10Hashar) [10:59:13] (03CR) 10Jaime Nuche: [C: 03+2] "LGTM" [tools/scap] - 10https://gerrit.wikimedia.org/r/836289 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [11:02:29] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops: Undeploy patch to use old PHP serialization in PHP 7.4 - https://phabricator.wikimedia.org/T318918 (10Lucas_Werkmeister_WMDE) Adding #continuous-integration-infrastructure (or should it be #continuous-integration-config?) since the patched PHP is... [11:02:38] (03CR) 10Jaime Nuche: install-world: ensure a clear venv (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/836288 (owner: 10Hashar) [11:28:37] (03CR) 10Seddon: [C: 03+1] "Can't +2 but this is good to go!" [integration/config] - 10https://gerrit.wikimedia.org/r/836696 (owner: 10Matthias Mullie) [12:14:07] jbond: doing the puppet image rebuild, sorry I had a long lunch :D [12:14:22] (03CR) 10Hashar: [C: 03+2] dockerfiles/operations-puppet: bump changelog to pick up new changes [integration/config] - 10https://gerrit.wikimedia.org/r/836744 (owner: 10Jbond) [12:15:34] jnuche: thanks for the review for bin/install_local_version.sh [12:16:04] jnuche: I went with an `ERR` trap cause there is already a trap set for `EXIT` to restore the git repo when a tag is passed as an argument [12:16:13] and setting a trap a second time replaces the previously one [12:16:14] but [12:16:31] I missed that ERR trap does not triggers on script completion and thus the temp dir is left behind :\ [12:16:48] I so miss a `try` `finally` in bash [12:17:00] hashar: I think you can use `trap -p` to get the defined callback and make a compound [12:17:23] yeah I felt it was going to be a lot more codes, but I guess there is not much choice [12:17:24] or [12:17:29] [12:17:30] (03Merged) 10jenkins-bot: dockerfiles/operations-puppet: bump changelog to pick up new changes [integration/config] - 10https://gerrit.wikimedia.org/r/836744 (owner: 10Jbond) [12:17:38] port the script to python! :D [12:19:46] hehehe, dunno if it's worth it [12:21:42] (03CR) 10Hashar: [C: 03+2] Remove extension-coverage for SearchVue [integration/config] - 10https://gerrit.wikimedia.org/r/836696 (owner: 10Matthias Mullie) [12:22:23] I guess I could add the commands to an array [12:22:31] and have an EXIT trap which executes each members of the array [12:24:51] (03Merged) 10jenkins-bot: Remove extension-coverage for SearchVue [integration/config] - 10https://gerrit.wikimedia.org/r/836696 (owner: 10Matthias Mullie) [12:25:36] I think you should be able to just compose the behavior, something like: `trap "echo blah; echo bleh EXIT"` [12:25:46] ohia hashar :) Is there a way to set a user to be in a lower priority queue for "checks" in ci ? [12:25:58] you can retrieve the existing recorded callback with (I think) `trap -p` [12:27:34] yeah, just tried, that's the one [12:28:04] addshore: nop [12:28:12] :( okaies! [12:28:28] addshore: the priority is set on a per pipeline basis such as `gate-and-submit` has a higher precedence than the other pipelines [12:28:32] I wanted to set "Addbot" as automatically having CI run for them, but I don't want it to clog anything up [12:28:41] or `postmerge` having a lower precedence [12:29:03] what is Addbot about? [12:29:57] (03CR) 10Hashar: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/836696 (owner: 10Matthias Mullie) [12:30:02] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/836696 [12:30:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:30:20] hashar, Addbot copies dependabot PRs from github for some repos, putting them on gerrit [12:30:33] eg, https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseManifest/+/819155 [12:30:45] but [12:30:47] but right now anyone wanting to merge the things needs to first "recheck" and then +2 if it is green [12:30:53] we already have LibUpgrader? [12:31:15] Libupgrader doesn't upgrade as many things as dependeabot [12:31:18] which has logic to throttle sending updates and +2ing them to prevent overloading the CI [12:33:05] well, addbot isn’t on the CI allowlist, so in a way it’s naturally throttled by humans pressing +2 ^^ [12:33:30] well when I look at https://gerrit.wikimedia.org/r/q/owner:Addbot it bumps a bunch of nodejs libraries which LibUp definitely does [12:33:43] but I imagine it does not cover all points and for Wikibase* repos you have some unique use cases ;) [12:34:18] what LibraryUpgarder is doing is that it analyzes https://integration.wikimedia.org/zuul/status.json [12:34:37] and I guess prevent itself from having more than X changes [12:34:59] same it +2, it looks at the gate-and-submit queue and waits for changes to be processed [12:35:34] one thing libup doesn’t cover is nested package.json files, I don’t remember if there were other things too [12:36:45] there are a few proposed updates that are redundant with LibUpgrader for sure such as the eslint-config-wikimedia / grunt-banana-checker updates [12:36:59] they are typically run on all repos from time to time to get everything more or less aligned at the same version [12:37:13] aka that is managed centrally [12:38:02] those can surely be updated for a subset of repositories managed by WMDE, but eventually that will be done as part of a campaign to update all repositories [12:40:47] addshore: the libup logic to watch Zuul is at https://gerrit.wikimedia.org/r/plugins/gitiles/labs/libraryupgrader/+/refs/heads/master/libup/gerrit.py#68 [12:42:35] and the throttler is called when attempting a push (waiting for less than 3 changes to be processed) https://gerrit.wikimedia.org/r/plugins/gitiles/labs/libraryupgrader/+/refs/heads/master/libup/tasks.py#137 [12:42:49] so you could do something similar to prevent AddBot from flooding CI ;-] [12:43:35] hashar: np, thanks [12:44:30] (03PS1) 10Hashar: jjb: update image operations-puppet job [integration/config] - 10https://gerrit.wikimedia.org/r/836797 [12:44:45] taavi: Next time please ask first before sending an announcement mailing list post. :-( [12:44:47] jbond: I am switching the job to the new image ;) [12:45:04] 10MediaWiki-Releasing, 10serviceops, 10PHP 7.2 support, 10PHP 7.3 support, 10Patch-For-Review: Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4 - https://phabricator.wikimedia.org/T261872 (10Jdforrester-WMF) >>! In T261872#8271964, @taavi wrote: > Policy-requir... [12:45:31] sorry :( [12:45:51] taavi: Yeah, I was going to land the CI changes in the next few minutes, but… [12:45:54] (03CR) 10Hashar: [C: 03+2] "I have updated the job:" [integration/config] - 10https://gerrit.wikimedia.org/r/836797 (owner: 10Hashar) [12:46:28] * James_F is not sure what to do. [12:47:24] thanks [12:47:49] (03Merged) 10jenkins-bot: jjb: update image operations-puppet job [integration/config] - 10https://gerrit.wikimedia.org/r/836797 (owner: 10Hashar) [12:52:07] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T314192 (10Ladsgroup) [12:54:31] (03CR) 10Hashar: [C: 04-1] install-world: ensure a clear venv (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/836288 (owner: 10Hashar) [13:17:44] (03PS8) 10Jforrester: Zuul: [mediawiki/vendor] Drop PHP72 jobs, use only PHP74 ones [integration/config] - 10https://gerrit.wikimedia.org/r/833024 [13:17:46] (03PS9) 10Jforrester: Zuul: [mediawiki/core] Drop PHP 7.2 and PHP 7.3 testing for master and wmf [integration/config] - 10https://gerrit.wikimedia.org/r/827496 (https://phabricator.wikimedia.org/T261872) [13:17:58] (03CR) 10Alexandros Kosiaris: "What's the use case for this? Any specific reason we need this at this repo/level?" [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/836711 (owner: 10Hashar) [13:23:28] (03PS5) 10Jforrester: Zuul: Drop PHP72 jobs everywhere, and PHP73 everywhere except old branches [integration/config] - 10https://gerrit.wikimedia.org/r/833051 [13:25:58] (03CR) 10CI reject: [V: 04-1] Zuul: Drop PHP72 jobs everywhere, and PHP73 everywhere except old branches [integration/config] - 10https://gerrit.wikimedia.org/r/833051 (owner: 10Jforrester) [13:32:14] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/vendor] Drop PHP72 jobs, use only PHP74 ones [integration/config] - 10https://gerrit.wikimedia.org/r/833024 (owner: 10Jforrester) [13:34:19] (03Merged) 10jenkins-bot: Zuul: [mediawiki/vendor] Drop PHP72 jobs, use only PHP74 ones [integration/config] - 10https://gerrit.wikimedia.org/r/833024 (owner: 10Jforrester) [13:34:35] !log Zuul: [mediawiki/vendor] Drop PHP72 jobs, use only PHP74 ones [13:34:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:35:12] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/core] Drop PHP 7.2 and PHP 7.3 testing for master and wmf [integration/config] - 10https://gerrit.wikimedia.org/r/827496 (https://phabricator.wikimedia.org/T261872) (owner: 10Jforrester) [13:35:19] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) Hi @brennen, what do you think about? I think that, at the moment, it could be nice to just create the GitLab group. The migration for Gerrit can... [13:37:33] (03Merged) 10jenkins-bot: Zuul: [mediawiki/core] Drop PHP 7.2 and PHP 7.3 testing for master and wmf [integration/config] - 10https://gerrit.wikimedia.org/r/827496 (https://phabricator.wikimedia.org/T261872) (owner: 10Jforrester) [13:41:29] !log Zuul: [mediawiki/core] Drop PHP 7.2 and PHP 7.3 testing for master and wmf for T261872 [13:41:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:41:32] T261872: Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4 - https://phabricator.wikimedia.org/T261872 [13:49:49] (03PS4) 10Jforrester: Zuul: Fold extension-quibble-composer-php73-or-later into extension-quibble-composer [integration/config] - 10https://gerrit.wikimedia.org/r/834570 [13:49:51] (03PS6) 10Jforrester: Zuul: Drop PHP72 jobs everywhere, and PHP73 everywhere except old branches [integration/config] - 10https://gerrit.wikimedia.org/r/833051 [13:51:54] (03PS5) 10Jforrester: Zuul: Fold extension-quibble-composer-php73-or-later into main job [integration/config] - 10https://gerrit.wikimedia.org/r/834570 [13:51:56] (03PS7) 10Jforrester: Zuul: Drop PHP72 jobs everywhere, and PHP73 everywhere except old branches [integration/config] - 10https://gerrit.wikimedia.org/r/833051 [13:53:50] (03CR) 10Jforrester: [C: 03+2] Zuul: Fold extension-quibble-composer-php73-or-later into main job [integration/config] - 10https://gerrit.wikimedia.org/r/834570 (owner: 10Jforrester) [13:53:59] (03CR) 10Jforrester: [C: 03+2] Zuul: Drop PHP72 jobs everywhere, and PHP73 everywhere except old branches [integration/config] - 10https://gerrit.wikimedia.org/r/833051 (owner: 10Jforrester) [13:55:43] (03Merged) 10jenkins-bot: Zuul: Fold extension-quibble-composer-php73-or-later into main job [integration/config] - 10https://gerrit.wikimedia.org/r/834570 (owner: 10Jforrester) [13:55:51] (03Merged) 10jenkins-bot: Zuul: Drop PHP72 jobs everywhere, and PHP73 everywhere except old branches [integration/config] - 10https://gerrit.wikimedia.org/r/833051 (owner: 10Jforrester) [13:56:19] !log Zuul: Drop PHP72 jobs everywhere, and PHP73 everywhere except old branches [13:56:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:56:26] \o/ \o/ \o/ [13:57:36] does the releng SAL also have a twitter account? ;) [13:58:06] (03PS4) 10Jforrester: Zuul: [integration/docroot] Migrate tests from php73+ to php74+ [integration/config] - 10https://gerrit.wikimedia.org/r/834574 [13:58:08] (03PS4) 10Jforrester: jjb: Drop unused jobs for PHP 7.2 and 7.3 [integration/config] - 10https://gerrit.wikimedia.org/r/834576 [13:58:10] (03PS4) 10Jforrester: jjb: Drop quibble-daily-* for for php72 [integration/config] - 10https://gerrit.wikimedia.org/r/834581 [13:58:12] (03PS4) 10Jforrester: jjb: Drop quibble php72 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/834582 [13:58:14] (03PS4) 10Jforrester: dockerfiles: Drop php72 images, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/834583 [13:58:24] Lucas_WMDE: Ha. No, but I can re-!log in the main ops channel if you want? :-) [13:58:33] :D [13:58:38] It is a bit momentous, after all. [13:59:24] maybe it could be a summary message [13:59:35] Yeah, but let's check that nothing has exploded first. :-) [13:59:39] ^^ [13:59:48] you haven’t added PHP80 yet, right? [13:59:51] (or made it voting at least) [13:59:57] I note that Wikibase's gate pipeline is a lot shorter now. [14:00:17] Lucas_WMDE: Sadly we've not been able to make it voting whilst having php72 voting, as some of our libraries won't support both. [14:00:25] * Lucas_WMDE nods [14:00:26] So now the work to actually finally support 80 can happen. [14:02:16] (03PS5) 10Jforrester: Zuul: [integration/docroot] Use main template with custom jobs for php73 [integration/config] - 10https://gerrit.wikimedia.org/r/834574 [14:02:18] (03PS5) 10Jforrester: jjb: Drop unused jobs for PHP 7.2 and 7.3 [integration/config] - 10https://gerrit.wikimedia.org/r/834576 [14:02:20] (03PS5) 10Jforrester: jjb: Drop quibble-daily-* for for php72 [integration/config] - 10https://gerrit.wikimedia.org/r/834581 [14:02:22] (03PS5) 10Jforrester: jjb: Drop quibble php72 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/834582 [14:02:24] (03PS5) 10Jforrester: dockerfiles: Drop php72 images, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/834583 [14:03:32] (03CR) 10Jforrester: [C: 03+2] Zuul: [integration/docroot] Use main template with custom jobs for php73 [integration/config] - 10https://gerrit.wikimedia.org/r/834574 (owner: 10Jforrester) [14:03:42] we’re allowed to use PHP 7.4 features in code now, right? since as you pointed out the decision was made months ago [14:03:57] Lucas_WMDE: Yes, go for it. There's a bunch of clean-up to do. [14:04:05] 🥳 [14:04:38] Personally I'm really looking forward to JSON_THROW_ON_ERROR [14:05:40] I think some of us were looking forward to typed class properties [14:05:40] (03Merged) 10jenkins-bot: Zuul: [integration/docroot] Use main template with custom jobs for php73 [integration/config] - 10https://gerrit.wikimedia.org/r/834574 (owner: 10Jforrester) [14:06:06] I like the sound of trailing commas in calls, but that probably needs to be an agreed-upon coding style change with accompanying PHPCS fixup, if we want to do it [14:06:21] Definitely. [14:06:28] codesniffer update time. [14:06:51] (also IIRC it’s not allowed in function *definitions* yet) [14:10:13] Lucas_WMDE: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/619457 is the symbolic change if you want to land it. :_) [14:11:29] wowee, r.eedy’s -2 goes all the way back to 2020 [14:11:46] Yup. [14:11:51] Been working on this for three years. [14:16:01] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/834576 (owner: 10Jforrester) [14:16:17] (03CR) 10Jforrester: [C: 03+2] jjb: Drop quibble-daily-* for for php72 [integration/config] - 10https://gerrit.wikimedia.org/r/834581 (owner: 10Jforrester) [14:16:20] I’ve +1ed it, but if you want to get it over with, I can also raise that to a +2 and remove the -2 ^^ [14:16:32] (ETA 11 more minutes for the regular test build) [14:16:39] Eh. Merge party is fine. It doesn't actually block anything. :-) [14:18:05] (03CR) 10Jforrester: [C: 03+2] jjb: Drop quibble php72 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/834582 (owner: 10Jforrester) [14:18:49] (03Merged) 10jenkins-bot: jjb: Drop unused jobs for PHP 7.2 and 7.3 [integration/config] - 10https://gerrit.wikimedia.org/r/834576 (owner: 10Jforrester) [14:19:04] (03CR) 10Jforrester: [C: 03+2] dockerfiles: Drop php72 images, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/834583 (owner: 10Jforrester) [14:19:08] (03Merged) 10jenkins-bot: jjb: Drop quibble-daily-* for for php72 [integration/config] - 10https://gerrit.wikimedia.org/r/834581 (owner: 10Jforrester) [14:20:15] (03Merged) 10jenkins-bot: jjb: Drop quibble php72 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/834582 (owner: 10Jforrester) [14:20:56] 10MediaWiki-Releasing, 10serviceops, 10PHP 7.2 support, 10PHP 7.3 support, 10Patch-For-Review: Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4 - https://phabricator.wikimedia.org/T261872 (10Jdforrester-WMF) 05Stalled→03Resolved a:03Jdforrester-WMF [14:21:08] (03Merged) 10jenkins-bot: dockerfiles: Drop php72 images, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/834583 (owner: 10Jforrester) [14:42:44] James_F: Enjoy the moment :) [14:42:47] nice work [14:43:01] Krinkle: A shared celebration, much like the work. [14:43:12] Next step, PHP 8.2 in prod. [14:43:27] Though I'd settle for now for just getting the 8.0 tests passing. :-) [14:58:47] :) [15:03:30] (03CR) 10Thcipriani: [C: 03+2] Start branching SearchVue for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/830606 (https://phabricator.wikimedia.org/T310367) (owner: 10Matthias Mullie) [15:04:35] (03Merged) 10jenkins-bot: Start branching SearchVue for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/830606 (https://phabricator.wikimedia.org/T310367) (owner: 10Matthias Mullie) [15:20:07] (03CR) 10Ahmon Dancy: install-world: ensure a clear venv (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/836288 (owner: 10Hashar) [15:24:22] (03PS2) 10Ahmon Dancy: RELEASE.md: Use release-scripts/perform-release [tools/scap] - 10https://gerrit.wikimedia.org/r/836257 [15:27:39] (03PS3) 10Ahmon Dancy: RELEASE.md: Use release-scripts/perform-release [tools/scap] - 10https://gerrit.wikimedia.org/r/836257 [15:28:18] (03CR) 10Ahmon Dancy: RELEASE.md: Use release-scripts/perform-release (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/836257 (owner: 10Ahmon Dancy) [15:31:40] (03PS1) 10Ahmon Dancy: Release 4.25.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/836862 [15:31:42] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.25.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/836862 (owner: 10Ahmon Dancy) [15:34:48] (03Merged) 10jenkins-bot: Release 4.25.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/836862 (owner: 10Ahmon Dancy) [15:39:07] 10Release-Engineering-Team (Bonus Level 🕹️), 10Scap: scap deploy-promote should retry when verifying Special:Version - https://phabricator.wikimedia.org/T316692 (10dancy) 05Open→03Resolved Deployed in scap 4.25.0 [15:44:40] (03CR) 10Jaime Nuche: [C: 03+2] RELEASE.md: Use release-scripts/perform-release [tools/scap] - 10https://gerrit.wikimedia.org/r/836257 (owner: 10Ahmon Dancy) [15:46:50] (03CR) 10Jforrester: buildkit: Support target platform in Makefile (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/836264 (https://phabricator.wikimedia.org/T318866) (owner: 10Dduvall) [15:47:17] (03Merged) 10jenkins-bot: RELEASE.md: Use release-scripts/perform-release [tools/scap] - 10https://gerrit.wikimedia.org/r/836257 (owner: 10Ahmon Dancy) [15:55:15] 10Release-Engineering-Team (Seen), 10Scap, 10Developer Productivity: Integrate mwdebug staging as part of `scap sync-dir` - https://phabricator.wikimedia.org/T239373 (10Krinkle) [18:10:01] James_F: I believe most node jobs are still 14 is that right? In any event, I went bold and fliipped 16 to default for local fresh-node given irregular releases. Might as well lead this time instead of trail. [18:10:22] probably means on average it'll be more in sync than not [18:10:40] Yeah, I should switch us over to 16 some time soon. [18:10:49] I'll keep 22.09.1 unanonunced for now [18:10:50] E_TOOMUCHGOINGON [18:10:55] ack. [18:11:06] it's there for people who early adopt or first install from readme today, the rest I'll keep it quiet for now. [18:11:23] either way, fresh-node14 is there to use directly [18:11:47] James_F: E_TOOMUCHGOINGON is a good problem to have though :) relatively speaking [18:11:59] Fair. [19:12:44] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [19:24:37] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [19:37:44] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [19:56:27] 10MediaWiki-Releasing, 10Epic: Distribute official container images of MediaWiki alongside the tarballs and git links - https://phabricator.wikimedia.org/T318977 (10Jdforrester-WMF) [19:59:00] thcipriani: Hey, in https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=2014572&oldid=2014570 you added a deployment calendar for two weeks' time but not next week. [20:02:44] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [20:04:12] I made a very quick one for now: https://wikitech.wikimedia.org/wiki/Deployments#Week_of_October_3 [20:16:27] James_F: whoops. Bot problems. [20:16:47] thank you for catching that, I'll take a look after the backport window here [20:18:32] (updated) [20:18:43] <3 [20:24:50] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [20:54:51] Project beta-scap-sync-world build #70748: 04FAILURE in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/70748/ [21:04:17] ^ is known [21:06:22] Yippee, build fixed! [21:06:22] Project beta-scap-sync-world build #70749: 09FIXED in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/70749/ [21:10:58] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Upstream, 10ci-test-error, 10php-segfault: CI job mediawiki-quibble-composer-mysql-php80-docker on mediawiki/core gate-and-submit is flaky failing with Segmentation fault - https://phabricator.wikimedia.org/T315167 (10Jdforrester-WMF) Sit... [21:32:29] (03PS2) 10Catrope: Zuul: [design/codex] Run special doc command for release tags [integration/config] - 10https://gerrit.wikimedia.org/r/822144 (https://phabricator.wikimedia.org/T313767) [21:34:29] (03CR) 10CI reject: [V: 04-1] Zuul: [design/codex] Run special doc command for release tags [integration/config] - 10https://gerrit.wikimedia.org/r/822144 (https://phabricator.wikimedia.org/T313767) (owner: 10Catrope) [21:35:20] (03PS1) 10Ahmon Dancy: auto-stage: Use PHP 7.4 [tools/release] - 10https://gerrit.wikimedia.org/r/836933 (https://phabricator.wikimedia.org/T271736) [21:38:53] (03CR) 10Ahmon Dancy: [C: 03+2] "Tested at https://releases-jenkins.wikimedia.org/view/incremental%20image%20build/job/build-mw-container-image/3536/console" [tools/release] - 10https://gerrit.wikimedia.org/r/836933 (https://phabricator.wikimedia.org/T271736) (owner: 10Ahmon Dancy) [21:39:56] (03Merged) 10jenkins-bot: auto-stage: Use PHP 7.4 [tools/release] - 10https://gerrit.wikimedia.org/r/836933 (https://phabricator.wikimedia.org/T271736) (owner: 10Ahmon Dancy) [22:25:25] Project beta-scap-sync-world build #70757: 04FAILURE in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/70757/ [22:29:25] Project beta-update-databases-eqiad build #61856: 04FAILURE in 9 min 25 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/61856/ [22:30:34] Looks like we need to change beta CLI to default to PHP 7.4... [22:31:29] Bleh. [22:31:35] Didn't we land this? [22:33:39] https://gerrit.wikimedia.org/r/c/operations/puppet/+/824216 [22:34:02] But we no longer read default_php_version but instead it goes by the order of php_versions or something? [22:34:15] reedy@deployment-deploy03:~$ php -v [22:34:15] PHP 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf5 (cli) (built: Dec 15 2021 23:28:21) ( NTS ) [22:34:20] Yeah. [22:35:41] Project beta-scap-sync-world build #70758: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/70758/ [22:35:49] https://gerrit.wikimedia.org/r/c/operations/puppet/+/836945 [22:35:59] Someone want to cherry-pick it? [22:37:16] leeeroy? [22:37:33] I'm going for my evening constitutional. :-) [22:38:42] * Reedy breaks more stuff [22:39:29] That won't actually remove it though, right? Just stop it being installed again... [22:39:54] reedy@deployment-deploy03:~$ php -v [22:39:54] PHP 7.4.30 (cli) (built: Aug 31 2022 09:45:57) ( NTS ) [22:40:02] lets see what the next run does then :) [22:45:10] Project beta-scap-sync-world build #70759: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/70759/ [22:52:52] just got pinged for the overdue `beta-scap-sync-world` ^, that y'all switching around the PHP versions? [22:54:16] (read above, yes) [22:55:08] Project beta-scap-sync-world build #70760: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/70760/ [22:58:17] TheresNoTime: We're breaking shit and going for the weekend [22:58:31] Project beta-scap-sync-world build #70761: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/70761/ [22:58:32] and no one told me?? :D [22:58:33] :p [22:59:04] Looks like it's still running PHP 7.2 somewhere? [22:59:18] * Reedy ponders purging 7.2 from the host [23:00:09] are the appservers also meant to be on 7.4, because https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version shows .2 still? [23:00:15] Or is that the scap patches I saw recently... and we need a new release? dancy [23:00:28] I think people assumed the production patches would filter down [23:03:23] !log ran `sudo puppet agent -tv` on deployment -deploy03, -mediawiki11, -mediawiki12 [23:03:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:04:49] https://phabricator.wikimedia.org/P35194 - that'll do it [23:05:09] Project beta-scap-sync-world build #70762: 04STILL FAILING in 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/70762/ [23:05:13] trying to bind to what 7.2 is bound to? [23:05:39] probably anything that relies on hiera settings will not apply to beta from prod settings. the role() hiera classifier system doesn't work in cloud vps instances [23:05:42] https://gerrit.wikimedia.org/r/c/operations/puppet/+/836932 [23:05:46] (this feels dejavu) [23:06:08] ^ that is what scap needs to run php7.4 instead of php7.2 [23:06:31] Thought I'd seen something :) [23:07:01] (I could just manually `sudo systemctl stop php7.2-fpm.service` and see if 7.4 binds?) [23:07:21] yeah [23:07:55] okay that's working on deploy03 [23:08:19] !log deployment-deploy03, `sudo systemctl stop php7.2-fpm.service`, `sudo systemctl start php7.4-fpm.service` [23:08:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:08:56] TheresNoTime: you might do a forced puppet run there again too just to double check that it remains stable [23:08:58] Project beta-scap-sync-world build #70763: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/70763/ [23:09:28] !log [samtar@deployment-deploy03 ~]$ sudo puppet agent -tv [23:09:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:10:05] yeah that's looking okay.. I'll do the same for the appservers too as they have the same bind issue [23:11:04] https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version shows 7.4 now [23:12:00] \o/ [23:12:12] we still need 836932 merged for scap to work though yeah? [23:12:34] * bd808 dreams of a world where beta is updated before prod for all things [23:12:37] Can cherry pick it to the puppetmaster [23:12:49] Reedy: ack, will do [23:12:56] I've just done :P [23:13:04] I already had it open :) [23:13:22] * Reedy runs puppet on the deployment host.. [23:13:41] :D [23:13:44] Notice: /Stage[main]/Scap/File[/etc/scap.cfg]/content: content changed '{md5}18db53fc324aa28c944ac27e4f3f07b6' to '{md5}a1a11f791127822e16882017d10b4fc0' [23:14:04] Wanna kick another one off? [23:14:47] `beta-code-update-eqiad` is running, so well timed [23:15:22] 23:15:07 23:15:07 Updating ExtensionMessages-master.php [23:15:22] 23:15:07 23:15:07 Updating LocalisationCache for master using 2 thread(s) [23:15:22] 23:15:07 23:15:07 Running rebuildLocalisationCache.php as www-data [23:15:23] yay [23:15:33] teamwork makes the dream (of no more php 7.2) work [23:16:17] welcome to 2019! [23:16:43] BC? Before Covid? [23:17:07] :p so 8.0 next week then? [23:17:26] I think we've some cleanup to do before ;) [23:17:33] 23:17:04 470 languages rebuilt out of 470 [23:24:15] Yippee, build fixed! [23:24:16] Project beta-scap-sync-world build #70764: 09FIXED in 9 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/70764/ [23:24:30] wooo [23:24:40] party [23:25:07] now, have we broken everything? :D [23:26:58] thanks TheresNoTime :) [23:27:03] so far so good.. https://beta-logs.wmcloud.org/goto/f61e40f8413bad332767ee6f4bb1e1c3 seems to look okayish? [23:27:17] oh no [23:27:23] you're gonna make me find that password again? :D [23:27:34] (I know it's something silly) [23:28:04] :P so that's it then, production will go to 7.4 soon(tm)? [23:28:39] I think production is almost all on 7.4? [23:28:43] if not all... [23:28:47] Just 7.2 removal and cleanup to go [23:28:47] oh! [23:29:08] even wikitech is on 7.4 [23:29:15] (oh yeah, so it is.. why did beta go last then? o.o) [23:29:19] Yippee, build fixed! [23:29:19] Project beta-update-databases-eqiad build #61857: 09FIXED in 9 min 18 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/61857/ [23:29:30] [00:05:39] probably anything that relies on hiera settings will not apply to beta from prod settings. the role() hiera classifier system doesn't work in cloud vps instances [23:29:33] Probably some of this... [23:29:43] And maybe no one stopping the php7.2 services [23:29:54] ahh [23:30:01] IIRC, I saw something about playing with ports in production config [23:30:15] so that port binding issue wasn't one [23:47:52] !log cherry pick 836953 to deployment-prep [23:47:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:51:12] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 (10TheresNoTime) Cherry-picked [[ https://gerrit.wikimedia.org/r/836953 | 836953 ]] to deployment-prep and puppet now runs successfully [23:51:40] didn't fix T317417 though (: (: (: [23:51:41] T317417: Phonos links to unroutable domain/URL for the MP3 file - https://phabricator.wikimedia.org/T317417 [23:54:28] !log samtar@deployment-jobrunner04:~$ sudo systemctl stop php7.2-fpm.service && sudo systemctl start php7.4-fpm.service [23:54:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL