[00:02:57] TimStarling: Reedy: I am in this channel, just was running an errand - taking a look [00:03:26] I also reported this at https://phabricator.wikimedia.org/T386006#10581145 [00:03:29] my autocomplete didn't find you :( [00:04:21] TimStarling: ack, thank you - I'll follow up in the task! [00:04:25] Reedy: no worries :) [00:22:47] swfrench-wmf: I'm tracking this error text down in the PHP and PCRE source. I guess it could happen if PHP was compiled with headers not matching the deployed PCRE library [00:23:23] TimStarling: that's exactly what's happening [00:23:55] it looks like somehow the wrong PCRE2 is installed in at least a subset of the CI images [00:24:03] (production images are consistent) [00:33:15] sounds like it's going to take a bit longer -- I'll just force-submit this UBN fix https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CodeMirror/+/1122696 [00:33:27] CodeMirror is fully broken right now [00:33:33] 10 lines of JS is pretty low risk [00:35:08] actually 1 line of JS and 9 lines of test [00:39:24] interesting, the release branch backport isn't running PHP 8.1 tests anyway, so it's going to be conventionally merged, just the master patch will need to be forced [00:39:57] oh, interesting! are just the 7.4 tests run? [00:40:24] yes, judging by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CodeMirror/+/1122697?tab=checks [00:41:22] yeah, for wmf/ branches we only run the version of PHP we use in prod [00:42:14] I suspect now that we're also running PHP 8.1 as part of the rollout... that probably should change [00:50:25] I think https://gerrit.wikimedia.org/r/c/integration/config/+/1020477 wants reverting [00:50:41] Ah, which James has done in https://gerrit.wikimedia.org/r/c/integration/config/+/1115147 but not merged [00:51:34] (03PS6) 10Jforrester: Zuul: Switch all wmf-quibble gate jobs from 7.4 to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1115147 (https://phabricator.wikimedia.org/T320818) [00:52:17] (03CR) 10Reedy: [C:03+1] "Probably should be merged soon... After the CI images have been fixed (T386006#10581145)" [integration/config] - 10https://gerrit.wikimedia.org/r/1115147 (https://phabricator.wikimedia.org/T320818) (owner: 10Jforrester) [00:52:30] ah, thanks for the digging, R.eedy! that makes sense [00:57:47] (03PS1) 10Scott French: Docker: [quibble-bullseye-php81] Ensure correct PCRE2 (10.42) is used [integration/config] - 10https://gerrit.wikimedia.org/r/1122700 (https://phabricator.wikimedia.org/T386006) [00:57:47] (03CR) 10Scott French: "I think this patch and, once quibble-bullseye-php81 is rebuilt using this, the next patch should resolve the issues @tstarling@wikimedia.o" [integration/config] - 10https://gerrit.wikimedia.org/r/1122700 (https://phabricator.wikimedia.org/T386006) (owner: 10Scott French) [00:57:57] (03PS1) 10Scott French: jjb: Update all instances of quibble-bullseye-php81 to 1.12.0-s3 [integration/config] - 10https://gerrit.wikimedia.org/r/1122701 (https://phabricator.wikimedia.org/T386006) [01:00:53] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T382369#10581270 (10MusikAnimal) [01:07:22] alright, to recap: [01:07:22] * CI jobs using quibble-bullseye-php81 may fail with an error like the one reported by T.imStarling on use of preg_* functions. [01:07:23] * I believe this is due to [0] and have posted patches to the task that I believe should fix it [1]. [01:07:23] * if someone from releng who has familiarity with the CI image build process could drive this forward, that would be greatly appreciated. [01:07:23] [0] https://phabricator.wikimedia.org/T386006#10581178 [01:07:23] [1] https://phabricator.wikimedia.org/T386006#10581278 [01:07:48] (03CR) 10Reedy: Docker: [quibble-bullseye-php81] Ensure correct PCRE2 (10.42) is used (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1122700 (https://phabricator.wikimedia.org/T386006) (owner: 10Scott French) [01:10:13] (03PS2) 10Scott French: Docker: [quibble-bullseye-php81] Ensure correct PCRE2 (10.42) is used [integration/config] - 10https://gerrit.wikimedia.org/r/1122700 (https://phabricator.wikimedia.org/T386006) [01:10:13] (03PS2) 10Scott French: jjb: Update all instances of quibble-bullseye-php81 to 1.12.0-s3 [integration/config] - 10https://gerrit.wikimedia.org/r/1122701 (https://phabricator.wikimedia.org/T386006) [01:11:19] (03CR) 10Scott French: Docker: [quibble-bullseye-php81] Ensure correct PCRE2 (10.42) is used (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1122700 (https://phabricator.wikimedia.org/T386006) (owner: 10Scott French) [01:11:25] Was it just that image that missed the pcre2 stuff... [01:11:26] I0491c38b52a67ad9423a74b960c4d7a736bf6ca2 [01:11:41] ^ added dockerfiles/php81/wikimedia-pcre2.list [01:12:23] (how all the different images are merged together is a bit of a mystery) [01:12:39] I believe so, yeah - I was looking at the `FROM` statements on 8.1-related Dockerfile.template files, and it looks like everything else picks up from `php81` either directly or transitively [01:13:10] yay consistency [01:14:19] (03CR) 10Reedy: [C:03+2] Docker: [quibble-bullseye-php81] Ensure correct PCRE2 (10.42) is used [integration/config] - 10https://gerrit.wikimedia.org/r/1122700 (https://phabricator.wikimedia.org/T386006) (owner: 10Scott French) [01:15:51] (03Merged) 10jenkins-bot: Docker: [quibble-bullseye-php81] Ensure correct PCRE2 (10.42) is used [integration/config] - 10https://gerrit.wikimedia.org/r/1122700 (https://phabricator.wikimedia.org/T386006) (owner: 10Scott French) [01:16:44] !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1122700 T386006 [01:16:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:16:47] T386006: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006 [01:17:41] Reedy: wow, thank you! let me know if you want me to spot-check the -s3 builds once they're done [01:18:26] Successfully published image docker-registry.discovery.wmnet/releng/quibble-bullseye-php81:1.12.0-s3 [01:18:40] swfrench-wmf: Might aswell if you don't mind [01:19:11] 2025-02-26 01:17:16 [docker-pkg-build] INFO - Unpacking libpcre2-8-0:amd64 (10.42-1~wmf11+1) over (10.36-2+deb11u1) ... (drivers.py:106) [01:19:11] 2025-02-26 01:17:16 [docker-pkg-build] INFO - Setting up libpcre2-8-0:amd64 (10.42-1~wmf11+1) ... (drivers.py:106) [01:19:21] It definitely would look right from that log entry [01:19:25] LGTM - just pulled the image and checked [01:20:12] just catching up on scrollback here [01:20:21] (03CR) 10Reedy: [C:03+2] jjb: Update all instances of quibble-bullseye-php81 to 1.12.0-s3 [integration/config] - 10https://gerrit.wikimedia.org/r/1122701 (https://phabricator.wikimedia.org/T386006) (owner: 10Scott French) [01:20:51] i touch these images rarely, but i'm around, let me know if there is other stuff to be done. [01:21:18] see if jenkins_jobs wants to play ball with me [01:21:39] (03Merged) 10jenkins-bot: jjb: Update all instances of quibble-bullseye-php81 to 1.12.0-s3 [integration/config] - 10https://gerrit.wikimedia.org/r/1122701 (https://phabricator.wikimedia.org/T386006) (owner: 10Scott French) [01:28:47] brennen: thanks! I _think_ we might be good now? [01:30:18] I've not deployed the jobs yet [01:30:19] Reedy: would it make sense to trigger a rebuild on Tim's failed job from earlier, or are those hermetic in such a way that the updated image wouldn't be used? [01:30:19] 10Continuous-Integration-Config: jenkins-jobs can error but says congratulations? - https://phabricator.wikimedia.org/T387264 (10Reedy) 03NEW [01:30:28] ah, got it [01:30:43] upgrading the os I use for dev stuff has broken a few python-y things :) [01:37:27] ah, yeah I'd imagine - I see that's documented as a 3.12 incompatibility in https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder#ERROR:stevedore.extension:_module_'pkgutil'_has_no_attribute_'ImpImporter' [01:38:06] yeah [01:38:20] the linked task is marked as dupe in a comment which links to https://bugs.launchpad.net/python-jenkins/+bug/2038277 [01:38:27] > [01:38:27] thanks for reporting the bug. I think this got fixed with python-jenkins 1.8.2 (commit 876a2a928cf97f105e805ea2901e697e27eb833b). So closing this. Feel free to reopen if you think this is still a problem. [01:38:44] yet... with 1.8.2 installed, seemingly not fixed [01:39:53] and yet ... [01:40:20] `pip3 install python-jenkins --break-system-packages` [01:40:47] still shows the same error (if the comment that that fixes it is indeed correct) [01:40:56] got worried for a minute, then realized the CI failure on https://gerrit.wikimedia.org/r/1122710 actually happened at 01:09 [01:41:44] but yeah, any thoughts on potentially rebuild a previously failed CI job? [01:42:10] a recheck should be fine when the update is deployed... [01:44:29] it's been long enough since i've done this that stuff has changed and i'm also untangling python dependencies [01:47:56] heh [01:48:24] > Used 107 time(s), last time was 1,682 day(s) ago [01:48:33] on my jjb token tells you, uh, exactly how long [01:48:43] :) [01:54:39] * bd808 is trying to run it [01:55:28] `jenkins-jobs: OK (53.61=setup[0.01]+cmd[53.59] seconds)` [01:55:35] bd808: i've just gotten to a running jenkins-jobs now, if that doesn't work but you have the incantation to hand [01:55:39] well that sounds promising [01:55:41] it thinks it worked [01:55:43] 10Continuous-Integration-Config: jjb-update output shouldn't be in red - https://phabricator.wikimedia.org/T387265 (10Reedy) 03NEW [01:55:57] !log `./jjb-update 'integration-quibble-fullrun-*-php81' '*-php81-phan' '*php81*'` [01:55:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:56:43] heh, thank you all for fighting with this! [01:57:11] it's running! [01:57:15] https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php81/5090/console [01:57:18] whether it passes :P [01:57:19] swfrench-wmf: apparently it takes a village ;) [01:57:53] how many engineers does it take to `$THING` [01:58:00] * bd808 sneaks off for dinner [01:58:48] thanks everybody. i apologize for arriving to the party late and basically useless. [02:02:36] given that we're well past the point where we failed in the first run (early in `Install composer dev-requires for vendor.git`), I am _cautiously optimistic_ [02:03:18] the only valid kind of optimistic [02:03:25] :) [02:12:51] aaand mediawiki-quibble-vendor-mysql-php81 is green [02:12:57] beauty [02:14:12] thank you all for moving this forward! [02:21:17] * swfrench-wmf heads off to go cook some food [02:28:25] * brennen departs in search of a burrito [02:38:39] 10Continuous-Integration-Config, 13Patch-For-Review, 07PHP 8.1 support: Configure CI to test the Wikimedia production gate code against PHP 8.1 instead of PHP 7.4 - https://phabricator.wikimedia.org/T320818#10581362 (10Reedy) [02:58:08] 10Continuous-Integration-Config: jjb-update output shouldn't be in red - https://phabricator.wikimedia.org/T387265#10581372 (10Reedy) p:05Triage→03Low [02:58:16] 10Continuous-Integration-Config: jenkins-jobs can error but says congratulations? - https://phabricator.wikimedia.org/T387264#10581374 (10Reedy) p:05Triage→03Low [03:04:00] I've got python11 on my mw dev vm now, so jenkins-jobs shold work fine again for me too [06:21:43] Project beta-update-databases-eqiad build #82856: 04FAILURE in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/82856/ [07:20:03] Project beta-update-databases-eqiad build #82857: 04STILL FAILING in 2.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/82857/ [08:20:03] Project beta-update-databases-eqiad build #82858: 04STILL FAILING in 2.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/82858/ [08:46:17] grrr [08:46:38] so yeah same as T386006 [08:46:39] T386006: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006 [08:46:46] php and pcre are mismatching on beta [08:52:02] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: PHP on Beta cluster fails due to mismatching PCRE - https://phabricator.wikimedia.org/T387276 (10hashar) 03NEW [08:53:51] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: PHP on Beta cluster fails due to mismatching PCRE - https://phabricator.wikimedia.org/T387276#10581787 (10hashar) p:05Triage→03Unbreak! That is due to T386006 and new php 8.1 packages that have been uploaded and automatically installed: ` Start-Dat... [09:04:39] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: PHP on Beta cluster fails due to mismatching PCRE - https://phabricator.wikimedia.org/T387276#10581805 (10hashar) `deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud` has: ` name=/etc/apt/sources.list.d/wikimedia-php81.list deb http://apt.wikim... [09:11:37] !log deployment-prep: cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/1122901 "php: use component/pcre2 when using Php 8.1" to fix php # T387276 [09:11:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:11:40] T387276: PHP on Beta cluster fails due to mismatching PCRE - https://phabricator.wikimedia.org/T387276 [09:20:03] Project beta-update-databases-eqiad build #82859: 04STILL FAILING in 2.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/82859/ [09:51:29] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: PHP on Beta cluster fails due to mismatching PCRE - https://phabricator.wikimedia.org/T387276#10581912 (10hashar) 05Open→03Resolved I have manually upgraded `libpcre2-8-0` on: deployment-deploy04.deployment-prep.eqiad1.wi... [09:52:08] 10GitLab, 06Release-Engineering-Team: GitLab Private Repository Request for: Trust and Safety Product team - https://phabricator.wikimedia.org/T382405#10581914 (10kostajh) Thanks! [09:58:13] hashar: do you happen to know by chance if CI/Jenkins does still deploy things to the wikikube staging cluster? [09:58:32] like temporarily for testing, I suppose [09:59:45] There is definitely an access token on contint hosts and we're dragging a snowflake config for CI around for quite some time, but I *think* it's not actually working (and has not for some time) - so it might as well be unused [10:01:38] Yippee, build fixed! [10:01:38] Project beta-update-databases-eqiad build #82860: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/82860/ [10:16:50] (03CR) 10Hashar: [C:03+2] Skip execution upon a success cache hit [integration/quibble] - 10https://gerrit.wikimedia.org/r/1121689 (https://phabricator.wikimedia.org/T383243) (owner: 10Dduvall) [10:17:23] jayme: hi, I don't know anything about a wikikube staging cluster :) [10:18:21] if there is a credential, I guess it would be in the Jenkins credential store at https://integration.wikimedia.org/ci/credentials/ [10:19:58] or maybe that uses an other mechanism :/ [10:32:19] hashar: finding the credentials is not the problem, they reside in /etc/kubernetes on the contint hosts...I'm just not sure how to verify if something is still making use of them (partly because I don't know what was using them in first place) [10:34:03] jayme: that might be used by PipelineLib which builds MediaWiki services and has support for deploying with helm [10:34:07] but [10:34:08] ah, something around https://phabricator.wikimedia.org/T288629 I guess ... maybe it was never used at all [10:34:10] I don't know much about that [10:34:32] ah yeah there is that as well [10:34:52] I think at some point we had each change merged to mediawiki/core to create a multiversion image and have tests running against it [10:35:01] (03Merged) 10jenkins-bot: Skip execution upon a success cache hit [integration/quibble] - 10https://gerrit.wikimedia.org/r/1121689 (https://phabricator.wikimedia.org/T383243) (owner: 10Dduvall) [10:36:30] jayme: you can ask in #developer-experience on Slack for my US based colleagues which would know for sure :) [10:36:38] https://app.slack.com/client/E012JBDTTHA/C05H0JYT85V [10:38:33] ok, thanks [10:40:37] (03PS1) 10Hashar: release: Quibble 1.13.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1122915 (https://phabricator.wikimedia.org/T383243) [10:41:13] (03PS1) 10Hashar: release: Start 1.13.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1122916 [10:42:18] (03PS2) 10Hashar: release: Quibble 1.13.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1122915 (https://phabricator.wikimedia.org/T383243) [10:42:18] (03PS2) 10Hashar: release: Start 1.13.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1122916 [10:42:29] (03CR) 10Hashar: [C:03+2] release: Quibble 1.13.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1122915 (https://phabricator.wikimedia.org/T383243) (owner: 10Hashar) [10:42:35] (03CR) 10Hashar: [C:03+2] release: Start 1.13.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1122916 (owner: 10Hashar) [10:45:33] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove explicit enablement of G1 garbage collector for Gerrit - https://phabricator.wikimedia.org/T387223#10582062 (10hashar) [10:55:50] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 06SRE: Automated validation of mediawiki-multiversion images - https://phabricator.wikimedia.org/T288629#10582102 (10JMeybohm) I stumbled upon this again recently and I think the current configuration does not allow pod creation at... [10:57:48] (03Merged) 10jenkins-bot: release: Quibble 1.13.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1122915 (https://phabricator.wikimedia.org/T383243) (owner: 10Hashar) [10:58:30] (03Merged) 10jenkins-bot: release: Start 1.13.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1122916 (owner: 10Hashar) [10:59:04] error: gpg failed to sign the data [10:59:04] error: unable to sign the tag [10:59:05] LE SIGH [10:59:55] that is "OBVIOUSLY" because my key has expired some days ago [11:06:07] !log Tag Quibble 1.13.0 @ 0ac128f7bc060c82f11317aabaf78a10b24aeeec # T383243 [11:06:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:06:10] T383243: Zuul/Jenkins: Investigate caching of build results for MediaWiki testsuite jobs - https://phabricator.wikimedia.org/T383243 [11:07:24] (03PS1) 10Hashar: dockerfiles: update Quibble to 1.13.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1122921 [11:07:24] ahah [11:07:31] (03CR) 10CI reject: [V:04-1] dockerfiles: update Quibble to 1.13.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1122921 (owner: 10Hashar) [11:07:36] I am so happy to have written that `./utils/update-quibble` script [11:07:37] well [11:07:43] except it fails to pass CI bah [11:08:51] (03PS2) 10Hashar: dockerfiles: update Quibble to 1.13.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1122921 [12:30:29] (03CR) 10Hashar: [C:03+2] dockerfiles: update Quibble to 1.13.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1122921 (owner: 10Hashar) [12:31:46] (03Merged) 10jenkins-bot: dockerfiles: update Quibble to 1.13.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1122921 (owner: 10Hashar) [13:08:35] (03PS1) 10Hashar: jjb: switch jobs to Quibble 1.13.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1122935 (https://phabricator.wikimedia.org/T383243) [13:18:16] (03PS1) 10Hashar: utils: teach docker-updates to update files [integration/config] - 10https://gerrit.wikimedia.org/r/1122936 [13:18:57] (03CR) 10Hashar: [C:03+2] jjb: switch jobs to Quibble 1.13.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1122935 (https://phabricator.wikimedia.org/T383243) (owner: 10Hashar) [13:20:09] (03Merged) 10jenkins-bot: jjb: switch jobs to Quibble 1.13.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1122935 (https://phabricator.wikimedia.org/T383243) (owner: 10Hashar) [13:20:10] !log Updating Quibble jobs to 1.13.0. "Skip execution upon a success cache hit" which would make some jobs to skip tests entirely when a set of commits/image is known to have previously passed # T383243 | dduvall [13:20:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:20:12] T383243: Zuul/Jenkins: Investigate caching of build results for MediaWiki testsuite jobs - https://phabricator.wikimedia.org/T383243 [14:12:22] (03CR) 10Jforrester: "Aha, oops, thank you." [integration/config] - 10https://gerrit.wikimedia.org/r/1122700 (https://phabricator.wikimedia.org/T386006) (owner: 10Scott French) [14:14:57] (03PS1) 10Daimona Eaytoy: zuul: Make ContentTranslation and GrowthExperiments depend on WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/1122954 (https://phabricator.wikimedia.org/T386018) [14:15:46] (03CR) 10Jforrester: "Should this also print out the jobs changed?" [integration/config] - 10https://gerrit.wikimedia.org/r/1122936 (owner: 10Hashar) [14:16:27] (03CR) 10CI reject: [V:04-1] zuul: Make ContentTranslation and GrowthExperiments depend on WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/1122954 (https://phabricator.wikimedia.org/T386018) (owner: 10Daimona Eaytoy) [15:05:24] (03PS2) 10Daimona Eaytoy: Make ContentTranslation and GrowthExperiments depend on WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/1122954 (https://phabricator.wikimedia.org/T386018) [15:05:37] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: PHP on Beta cluster fails due to mismatching PCRE - https://phabricator.wikimedia.org/T387276#10583374 (10Scott_French) Ah, thanks for the fix, @hashar! It did not occur to me that these same packages were being used in Beta. [15:10:31] (03PS2) 10Hashar: utils: teach docker-updates to update files [integration/config] - 10https://gerrit.wikimedia.org/r/1122936 [15:10:51] (03CR) 10Hashar: "That can then be done with `./utils/jjb-diff.sh` :]" [integration/config] - 10https://gerrit.wikimedia.org/r/1122936 (owner: 10Hashar) [15:11:18] (03CR) 10Hashar: [C:03+2] Make ContentTranslation and GrowthExperiments depend on WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/1122954 (https://phabricator.wikimedia.org/T386018) (owner: 10Daimona Eaytoy) [15:12:38] (03Merged) 10jenkins-bot: Make ContentTranslation and GrowthExperiments depend on WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/1122954 (https://phabricator.wikimedia.org/T386018) (owner: 10Daimona Eaytoy) [15:12:59] (03CR) 10Jforrester: [C:03+2] "Let's do it!" [integration/config] - 10https://gerrit.wikimedia.org/r/1122936 (owner: 10Hashar) [15:15:21] (03Merged) 10jenkins-bot: utils: teach docker-updates to update files [integration/config] - 10https://gerrit.wikimedia.org/r/1122936 (owner: 10Hashar) [15:15:45] (03PS1) 10Jforrester: [Docker] Refresh PHP base images to latest upstreams [integration/config] - 10https://gerrit.wikimedia.org/r/1122971 [15:16:02] hashar: ^^^ Because I like to make contint cry. [15:17:52] (03CR) 10CI reject: [V:04-1] [Docker] Refresh PHP base images to latest upstreams [integration/config] - 10https://gerrit.wikimedia.org/r/1122971 (owner: 10Jforrester) [15:19:11] (03PS2) 10Jforrester: [Docker] Refresh PHP base images to latest upstreams [integration/config] - 10https://gerrit.wikimedia.org/r/1122971 [15:19:16] Boo, merge conflict whitespace. [15:29:38] (03CR) 10Hashar: [C:04-1] "I don't think that is needed for:" [integration/config] - 10https://gerrit.wikimedia.org/r/1122971 (owner: 10Jforrester) [15:29:55] too many updates :d [15:30:49] oh [15:31:34] we have the PHPUnit coverage being run with quibble-buster-php74-coverage [15:31:46] there is an unused quibble-bullseye-php74-coverage [15:31:56] and give coverage is generated from the master branch [15:32:13] I felt like we could switch to 8.1 :) [15:51:36] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [15:56:51] RESOLVED: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [15:59:23] (03CR) 10Jforrester: "They're needed because you're using the PHP binaries in the Quibble images. We could skip the 7.4 ones if you want, assuming we can drop i" [integration/config] - 10https://gerrit.wikimedia.org/r/1122971 (owner: 10Jforrester) [16:02:34] James_F: I don't get why the quibble images need to be rebuilt? I did it some hours ago? [16:02:50] Your ones will have the old binaries with the old versions of PHP. [16:02:51] or i am missing the intent of your patch :] [16:03:01] This upgrades them to the latest upstream PHP version. [16:03:16] I think? [16:04:38] docker-registry.wikimedia.org/releng/quibble-bullseye-php84 has 8.4.4-1+0~20250215.23+debian11~1.gbp9a7680 [16:04:41] and that is the latest [16:04:46] cause I rebuilt it this morning [16:05:00] Hmm. I thought it didn't bypass the version from our 'php' image, but it does? [16:05:05] Interesting. [16:05:36] the quibble images do their own install of the php packages [16:05:47] with the `apt_install` docker-pkg macro which does run `apt update` [16:05:57] Hmm. Should they? [16:05:59] so on rebuild the packages are the latest .That is why I got confused :] [16:06:06] They depend on the php images. [16:06:22] Or we could just have them not depend on those images at all. [16:06:23] they don't :) [16:06:53] `docker-pkg` thinks they do. [16:08:20] oh [16:08:23] yes but now [16:08:37] so quibble-bullseye-php84 is a multibuild [16:08:50] Either split them fully, or merge them? [16:08:57] it uses the sury-php image to only copy the gpg and apt config [16:09:32] but otherwhise I don't see a dependency on php [16:09:39] so I don't know what the hell docker-pkg might be doing [16:10:17] I'll have a look after my endless meetings. [16:11:04] which command have you used to do the update? [16:11:07] (03update) 10brennen: update submodules for 2025-02-24 release [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/60 (https://phabricator.wikimedia.org/T387172) [16:12:01] `docker-pkg -c dockerfiles/config.yaml update --reason "Update PHP to 8.4.4" -v "8.4.4" php84 ./dockerfiles` [16:14:03] (03merge) 10brennen: update submodules for 2025-02-24 release [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/60 (https://phabricator.wikimedia.org/T387172) [16:17:32] (03CR) 10Hashar: [C:04-1] "The Quibble images do not depend on the php ones, they do their own `apt_install` of the PHP packages." [integration/config] - 10https://gerrit.wikimedia.org/r/1122971 (owner: 10Jforrester) [16:17:55] I pasted the outcome of running it locally and it only lists the php84 images :-] [16:18:02] so either docker-pkg is broken somehow [16:18:08] or you have an image introducing the dep? [16:18:33] Maybe I have an older/newer version? [16:18:41] maybe [16:19:30] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/LdapAuthentication] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1122982 (https://phabricator.wikimedia.org/T376097) [16:19:33] I run it straight from a clone of https://gerrit.wikimedia.org/r/operations/docker-images/docker-pkg [16:19:55] installed with `pip install -e path/to/checkout/of/docker-pkg` [16:21:06] Aha, I've updated docker-pkg from 4.0.0 to 4.0.3, and it still bumps quibble-bullseye-php84 [16:21:40] try against a clean checkout of https://gerrit.wikimedia.org/r/integration/config.git ? [16:21:59] I did. [16:22:12] Well, clean except for r1122982, but that's irrelevant. [16:22:31] {__name__="proc_jvm_gc_count_G1_Young_Generation", cluster="misc", instance="gerrit.wikimedia.org:443", job="gerrit-metrics", prometheus="ops", site="eqiad"} [16:22:33] err [16:26:24] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2025-02-25 - https://phabricator.wikimedia.org/T387172#10583705 (10Malyacko) Test comment. [16:43:52] 10Phabricator (phabricator-next), 10Phabricator Antivandalism Extension, 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review, 07Performance Issue: More AVA performance improvements - https://phabricator.wikimedia.org/T386704#10583810 (10Aklapper) [16:45:04] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove explicit enablement of G1 garbage collector for Gerrit - https://phabricator.wikimedia.org/T387223#10583812 (10thcipriani) > We also introduced -XX:G1NewSizePercent=15 back in October 2019 (dd13903f1c98ff3a1fc904fe6b... [16:45:21] 10Gerrit, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Remove explicit enablement of G1 garbage collector for Gerrit - https://phabricator.wikimedia.org/T387223#10583813 (10thcipriani) [16:46:14] 10Phabricator (2025-02-26), 10Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phabricator/Phorge 2025-02-25 - https://phabricator.wikimedia.org/T387172#10583821 (10thcipriani) [16:47:14] 10Release-Engineering-Team (Priority Backlog 📥), 10wikimedia.biterg.io: Automatically export and publish a list of WMF deployed code repositories in Bitergia's JSON format - https://phabricator.wikimedia.org/T385529#10583826 (10thcipriani) [16:48:09] 10Phabricator (2025-02-26), 10Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phabricator/Phorge 2025-02-25 - https://phabricator.wikimedia.org/T387172#10583830 (10Aklapper) 05Open→03Resolved a:03Aklapper Thank you! Lots of [AVA changes](https://gerrit.wikimedia.org/r/plugins/g... [16:59:00] 10Deployments, 10Release-Engineering-Team (Radar), 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10583913 (10thcipriani) [16:59:53] 10Gerrit (Gerrit 3.11), 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#10583918 (10thcipriani) [17:00:58] 10Release-Engineering-Team (Priority Backlog 📥), 10WikimediaDebug, 13Patch-For-Review: Support WebExtensions Manifest v3 - https://phabricator.wikimedia.org/T312694#10583930 (10thcipriani) [17:01:48] 10Release-Engineering-Team (Radar), 10Scap, 10MW-on-K8s, 06serviceops: Pushing mediawiki-multiversion Docker image from deploy server takes 4 minutes - https://phabricator.wikimedia.org/T341441#10583934 (10thcipriani) [17:21:38] 10Deployments, 10Release-Engineering-Team (Radar), 06serviceops, 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10584067 (10Aklapper) [17:31:58] 10Continuous-Integration-Config, 06Abstract Wikipedia team, 10function-evaluator, 03Abstract Wikipedia Fix-It tasks: Find a way to re-use the non-post-merge jobs when merging patches in FF-only GitLab repos' CI - https://phabricator.wikimedia.org/T386936#10584117 (10Jdforrester-WMF) p:05Triage→03Low [17:35:03] maintenance-disconnect-full-disks build 679446 integration-agent-docker-1041 (/: 28%, /srv: 98%, /var/lib/docker: 43%): OFFLINE due to disk space [17:38:59] it looks like i just got a few emails from phabricator that were delayed by about ~15minutes - the emails were for comments in tasks i'm subscribed to, and (in one case) an email about a comment made at 17:26 UTC arrived before an email about a comment made at 17:15 UTC. (these might have been a one-off, but i thought i'd mention it just in case as i don't know if i've seen something like this happening before) [17:40:03] maintenance-disconnect-full-disks build 679447 integration-agent-docker-1041 (/: 28%, /srv: 71%, /var/lib/docker: 43%): RECOVERY disk space OK [17:51:43] (03CR) 10Hashar: "I will review this since I do have some experience with an extension written by Google to assist with Gerrit development :)" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis) [17:57:29] 06Release-Engineering-Team, 10Scap, 10RESTBase: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T361608#10584318 (10thcipriani) 05Open→03Resolved a:03dancy I think the core issue here was resolved in scap with this commit: [[https://gitlab.wikimedia.org/repos/releng/scap/-/merg... [18:08:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing 😎): Zuul/Jenkins: Investigate caching of build results for MediaWiki testsuite jobs - https://phabricator.wikimedia.org/T383243#10584349 (10dduvall) Attempting a better summary before closing this: === Implementation === From the comm... [18:09:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing 😎): Zuul/Jenkins: Investigate caching of build results for MediaWiki testsuite jobs - https://phabricator.wikimedia.org/T383243#10584350 (10dduvall) 05Open→03Resolved [18:19:30] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T382369#10584364 (10DLynch) [19:21:31] A_smart_kitten: It shouldn't happen all the time, but a 15 minute mail queue delay isn't unheard of, especially when a someone is doing bulk changes somewhere (not sure this was the case this time). [20:11:30] (03open) 10jforrester: Add an extra newline after 'The following rules are failing and were disabled' blobs [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/69 [20:16:55] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/LdapAuthentication] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1122982 (https://phabricator.wikimedia.org/T376097) (owner: 10Jforrester) [20:18:16] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/LdapAuthentication] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1122982 (https://phabricator.wikimedia.org/T376097) (owner: 10Jforrester) [20:18:46] !log Zuul: [mediawiki/extensions/LdapAuthentication] Mark as archived, for T376097 [20:18:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:18:48] T376097: Archive the LdapAuthentication extension - https://phabricator.wikimedia.org/T376097 [20:20:43] (03approved) 10reedy: Add an extra newline after 'The following rules are failing and were disabled' blobs [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/69 (owner: 10jforrester) [20:20:46] (03merge) 10reedy: Add an extra newline after 'The following rules are failing and were disabled' blobs [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/69 (owner: 10jforrester) [20:21:22] !log jforrester@doc1003:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/LdapAuthentication/ #T376097 [20:21:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:23:06] (03PS3) 10Jforrester: [Docker] Refresh PHP base images to latest upstreams [integration/config] - 10https://gerrit.wikimedia.org/r/1122971 [20:23:06] (03CR) 10Jforrester: [Docker] Refresh PHP base images to latest upstreams (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1122971 (owner: 10Jforrester) [20:23:28] (03CR) 10Jforrester: [C:03+2] [Docker] Refresh PHP base images to latest upstreams [integration/config] - 10https://gerrit.wikimedia.org/r/1122971 (owner: 10Jforrester) [20:24:39] (03Merged) 10jenkins-bot: [Docker] Refresh PHP base images to latest upstreams [integration/config] - 10https://gerrit.wikimedia.org/r/1122971 (owner: 10Jforrester) [20:25:23] (03open) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 [20:26:47] (03PS1) 10Jforrester: jjb: Update bare PHP jobs (non-Quibble) to latest versions [integration/config] - 10https://gerrit.wikimedia.org/r/1123027 [20:29:32] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [20:31:14] bd808: ah okay, ty for the info :) [20:39:32] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [21:32:58] (03CR) 10Hashar: "Nice that is great, I haven't looked at a browser extension in a while and migrating to Manifest v3 surely has some fun oddities. You did " [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis) [21:33:17] bd808: here you got a review :] [21:33:20] I am off though [21:35:26] thanks hashar [21:35:40] beware there is at least one nitpick! :-] [21:38:19] * hashar ZzzZzzz [21:51:26] (03update) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 [21:51:27] (03update) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 [21:51:30] (03update) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 [21:57:42] (03update) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 [21:58:08] (03update) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 [22:02:43] 10GitLab (CI & Job Runners): https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts buildkit failure - https://phabricator.wikimedia.org/T386947#10584920 (10dduvall) This error seems to emanate from Blubber's BuildKit frontend, not buildkitd. The buildkit dependency is woefully out of date in Blubb... [22:02:52] (03update) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 (https://phabricator.wikimedia.org/T386947) [22:02:52] (03update) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 (https://phabricator.wikimedia.org/T386947) [22:02:54] (03update) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 (https://phabricator.wikimedia.org/T386947) [22:03:02] (03update) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 (https://phabricator.wikimedia.org/T386947) [23:21:07] (03update) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 (https://phabricator.wikimedia.org/T386947) [23:21:10] (03update) 10dduvall: buildkit: Update BuildKit dependency to v0.20 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/120 (https://phabricator.wikimedia.org/T386947) [23:21:13] (03open) 10dduvall: llbtest: Moved llbtest to its own project/module [repos/releng/blubber] (review/buildkit-v0.20-3a8f) - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/121 [23:22:48] (03update) 10dduvall: llbtest: Moved llbtest to its own project/module [repos/releng/blubber] (review/buildkit-v0.20-3a8f) - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/121 [23:22:49] (03update) 10dduvall: llbtest: Moved llbtest to its own project/module [repos/releng/blubber] (review/buildkit-v0.20-3a8f) - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/121