[07:27:42] !log Refreshing all Jenkins jobs [07:27:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:31:03] !log Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # T300340 T291549 T225730 [07:31:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:31:07] T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 [07:31:08] T291549: Evaluate phpbench for benchmarking and guarding against performance regressions - https://phabricator.wikimedia.org/T291549 [07:31:08] T300340: Use Memcached with Quibble - https://phabricator.wikimedia.org/T300340 [07:32:23] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/763559 (https://phabricator.wikimedia.org/T300340) (owner: 10Hashar) [07:34:08] (03Merged) 10jenkins-bot: jjb: update Quibble jobs from 1.3.0 to 1.4.0 [integration/config] - 10https://gerrit.wikimedia.org/r/763559 (https://phabricator.wikimedia.org/T300340) (owner: 10Hashar) [08:14:22] (03CR) 10Arturo Borrero Gonzalez: [V: 03+2 C: 03+2] Edit Repo Config [cloud/toolforge/jobs-framework-cli] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/719077 (owner: 10Arturo Borrero Gonzalez) [08:15:53] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests: PHP notices should fail CI tests - https://phabricator.wikimedia.org/T302033 (10hashar) 05Openβ†’03Resolved a:03hashar I don't think there is anything to do here. As highlighted notices are caught when running unit tests. The issue was in bootstrap.p... [09:16:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Zuul, 10ci-test-error (WMF-deployed Build Failure): fatal: remote error: access denied or repository not exported: /mediawiki/extensions/ReadingLists - https://phabricator.wikimedia.org/T187897 (10hashar) Went to grep all the CI... [09:22:40] hashar: are you around? [09:23:38] arturo: yes :) [09:23:58] ok :-) [09:24:11] would like to help me understand this problem? https://gerrit.wikimedia.org/r/c/cloud/toolforge/jobs-framework-api/+/764307 [09:24:29] I'm not sure if there is something wrong with tox, or the zuul config is missing something [09:24:30] ? [09:26:29] hashar: ok I think it uses a tox-buster docker image, I see `-env-file /dev/fd/63 docker-registry.wikimedia.org/releng/tox-buster:0.5.1` [09:28:00] 00:00:05.628 ERROR: flake8: InterpreterNotFound: python3.9 [09:28:00] 00:00:05.628 ERROR: black: InterpreterNotFound: python3.9 [09:28:21] so tox.ini in the repository probably asks for a py39 environment [09:28:43] yeah, I think we need to switch the base docker image that is used for CI for this repo [09:28:46] is that possible? [09:28:47] and there is no python package for it [09:28:48] https://phabricator.wikimedia.org/T289222 [09:29:00] oh :-( [09:29:05] Kunal wanted to add an extra job solely for python3.9 which would use a docker image based on Bullseye or something [09:29:18] which would cause old pythons to vanish [09:29:36] so we would need to build recent python for whatever OS is currently the base for releng/tox [09:29:49] or switch that image to Bullseye and build the old python 3 version we still want to support [09:30:09] ok, nevermind! I'll drop my patch for now, no problem [09:30:41] thanks hashar ! [09:31:15] Ideally I should start the effort to determinate which python3 versions we still want to support on all our repositories [09:31:28] then get those versions built for bullseye [09:31:39] and migrate the releng/tox image to that OS with the python packages [09:34:37] arturo: the twist is we never went to establish a channel or working group to determinate which versions we wanna support [09:34:45] which applies to nodejs/java/php etc versions [09:34:58] probably the easies is to support all the versions :-P [09:35:03] easiest* [09:35:12] yeah [09:48:10] hashar: do we have instructions on how to run the same CI checks but locally in our laptop? [09:50:16] problem: the CI asks for py3.7 and I only have py3.9 on my laptop, installing a py 3.7 venv in my laptop is producing some weird errors, so perhaps the way to go is to run the CI checks as a docker image on my laptop [09:53:05] hmm [09:53:18] that depends! [09:53:34] releng/tox was the very first image created and it has everything to run a load [09:53:39] but is heavily tied to CI [09:54:00] the entry point does fetch/checkout the patch and requires some CI specific environment variable to indicate the repo and patch to fetch [09:54:10] it will then invoke tox from `/src` [09:54:19] ... mmm ok [09:54:37] for other images, the entry point is the test command for example "composer install && composer test" which expect the source to be bind mounted at `/src` [09:54:55] so what I do (and others) is that I clone the repo/ checkout the patch on my local machine [09:55:19] run the image with a bind mount of that director to /src ( `--volume "$(pwd):/src"`) [09:55:29] and use bash `--entrypoint=bash` [09:55:38] I can then run the entrypoint `/run.sh` in most cases [09:55:48] or directly invoke the test command (`tox` in this case) [09:55:55] volans uses pyenv [09:56:01] but he is smarter than me [09:56:27] also lot of people don't even bother and just send a patch to CI (execution as a service) [09:56:51] ... ok [10:12:53] arturo: we can pair it if you want? [10:12:58] that is often easier [10:13:00] :D [10:13:32] will use the execution as a service for now [10:13:35] thanks! [10:15:11] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mfossati - https://phabricator.wikimedia.org/T302204 (10mfossati) [11:17:47] James_F: the latest stylint-config-wikimedia won't work https://github.com/miraheze/CreateWiki/runs/5273134048?check_suite_focus=true [11:20:42] (03PS1) 10David Caro: Add cloud/toolforge/toolforge-cli ci config [integration/config] - 10https://gerrit.wikimedia.org/r/764329 (https://phabricator.wikimedia.org/T299026) [11:27:18] RhinosF1: hm, is that because postcss-less is nested below stylelint-config-wikimedia? https://github.com/miraheze/CreateWiki/blob/83d5f0b5f11908a2e81fa695d9b09b62e2a1cbad/package-lock.json#L6435-L6441 [11:27:42] because I thought I could simplify the stylelint config elsewhere by removing the direct postcss-less dependency again https://gerrit.wikimedia.org/r/c/wikidata/query/gui/+/764325/1/package.json [11:27:58] but maybe that only works because in β€œmy” package.json postcss-less is still at the root level [11:28:17] maybe it should be a peerDependency of stylelint-config-wikimedia, and still be mentioned in individual projects’ package.json? [11:29:05] Lucas_WMDE: all I know is the automatic updates are failing [11:29:11] * RhinosF1 is merging all the things [11:36:34] 10Release-Engineering-Team (Done by Feb 23πŸ”₯), 10MW-on-K8s, 10serviceops: Make scap deploy to kubernetes together with the legacy systems - https://phabricator.wikimedia.org/T299648 (10MatthewVernon) [11:39:46] 10Continuous-Integration-Infrastructure, 10Deployments, 10Security: Give deployers access to build-mw-container-image Jenkins job - https://phabricator.wikimedia.org/T302213 (10Lucas_Werkmeister_WMDE) [11:40:18] 10Continuous-Integration-Infrastructure, 10Deployments, 10Security: Give deployers access to build-mw-container-image Jenkins job - https://phabricator.wikimedia.org/T302213 (10Lucas_Werkmeister_WMDE) @hashar reports that the build may actually happen automatically, triggered by a commit in `/srv/patches/`;... [11:43:10] 10Continuous-Integration-Infrastructure, 10Deployments, 10Security: Give deployers access to build-mw-container-image Jenkins job - https://phabricator.wikimedia.org/T302213 (10Joe) 05Openβ†’03Declined This won't be necessary once T297673 is completed. Tentatively closing as declined as it shouldn't be an... [13:54:52] 10Continuous-Integration-Config, 10Fresh, 10Performance-Team: Remove Node.js support from CI - https://phabricator.wikimedia.org/T302221 (10Krinkle) [13:55:15] 10Continuous-Integration-Config, 10Fresh, 10Performance-Team: Remove Node.js support from CI - https://phabricator.wikimedia.org/T302221 (10Krinkle) [13:55:43] (03PS5) 10Krinkle: Docker: Drop node10-test-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720153 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [13:55:59] (03PS4) 10Krinkle: Docker: Drop node10-test, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720154 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [13:56:13] (03PS4) 10Krinkle: Docker: Drop node10, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720156 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [13:56:27] (03CR) 10jerkins-bot: [V: 04-1] Docker: Drop node10, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720156 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [13:56:32] 10Continuous-Integration-Config, 10Fresh, 10Performance-Team, 10Patch-For-Review: Remove Node.js 10 support from CI - https://phabricator.wikimedia.org/T302221 (10Krinkle) [13:57:43] 10Continuous-Integration-Config, 10Fresh, 10Performance-Team, 10Patch-For-Review: Remove Node.js 10 support from CI - https://phabricator.wikimedia.org/T302221 (10Krinkle) p:05Triageβ†’03Medium a:03Krinkle [14:06:18] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10HTTPS: The certificate for upload.wikimedia.beta.wmflabs.org expired on February 16, 2022. - https://phabricator.wikimedia.org/T301995 (10MatthewVernon) 05Openβ†’03Resolved a:03MatthewVernon [I think this task can be closed, since the issue was resolve... [14:06:26] 10Beta-Cluster-Infrastructure, 10Quality-and-Test-Engineering-Team (QTE), 10SRE, 10Traffic, and 2 others: [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy - https://phabricator.wikimedia.org/T293585 (10MatthewVernon) [14:19:46] 10Phabricator (Upstream), 10Upstream: Create User-ItamarWMDE project - https://phabricator.wikimedia.org/T302225 (10ItamarWMDE) [14:36:22] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23πŸ”₯): Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10valerio.bozzolan) Hi @brennen! is this request OK for you? If I understand correctly T282842. Thank you! [14:36:35] (03CR) 10Hashar: [C: 03+2] Add cloud/toolforge/toolforge-cli ci config [integration/config] - 10https://gerrit.wikimedia.org/r/764329 (https://phabricator.wikimedia.org/T299026) (owner: 10David Caro) [14:37:33] hashar, T302226 [14:37:34] T302226: Merges in mediawiki/core failing with "KeyError: 'server'" - https://phabricator.wikimedia.org/T302226 [14:38:00] hi, i just came to report that ^ [14:38:16] looks like a bug in the CI scripts, i was wondering if there was a recent change or something [14:38:34] could that be the quibble upgrade? [14:38:42] https://gerrit.wikimedia.org/r/c/integration/config/+/763559 [14:39:00] (03Merged) 10jenkins-bot: Add cloud/toolforge/toolforge-cli ci config [integration/config] - 10https://gerrit.wikimedia.org/r/764329 (https://phabricator.wikimedia.org/T299026) (owner: 10David Caro) [14:46:26] (03CR) 10David Caro: "Thanks! \o/" [integration/config] - 10https://gerrit.wikimedia.org/r/764329 (https://phabricator.wikimedia.org/T299026) (owner: 10David Caro) [14:48:14] (03CR) 10Hashar: "Revisiting my pending reviews. I don't see a need to run profiling when running testsuite on CI and I would like to avoid adding yet anot" [integration/config] - 10https://gerrit.wikimedia.org/r/748312 (https://phabricator.wikimedia.org/T225730) (owner: 10Ladsgroup) [14:52:15] 10Continuous-Integration-Config, 10MediaWiki-General, 10ci-test-error (WMF-deployed Build Failure): Merges in mediawiki/core failing with "KeyError: 'server'" - https://phabricator.wikimedia.org/T302226 (10matmarex) [14:52:31] zabe: MatmaRex: that would be the quibble upgrade this morning :/ [14:53:32] 10Continuous-Integration-Config, 10MediaWiki-General, 10ci-test-error (WMF-deployed Build Failure): Merges in mediawiki/core failing with "KeyError: 'server'" - https://phabricator.wikimedia.org/T302226 (10hashar) [14:54:20] (03CR) 10Hashar: "deployed! you can `recheck` on the project." [integration/config] - 10https://gerrit.wikimedia.org/r/764329 (https://phabricator.wikimedia.org/T299026) (owner: 10David Caro) [14:57:58] (03PS1) 10Hashar: Revert "jjb: update Quibble jobs from 1.3.0 to 1.4.0" [integration/config] - 10https://gerrit.wikimedia.org/r/763979 (https://phabricator.wikimedia.org/T302226) [14:58:37] !log Reverting Quibble jobs from 1.4.0 to 1.3.0 # T302226 [14:58:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:58:40] T302226: Merges in mediawiki/core failing with "KeyError: 'server'" - https://phabricator.wikimedia.org/T302226 [14:58:51] (03CR) 10Hashar: [C: 03+2] Revert "jjb: update Quibble jobs from 1.3.0 to 1.4.0" [integration/config] - 10https://gerrit.wikimedia.org/r/763979 (https://phabricator.wikimedia.org/T302226) (owner: 10Hashar) [15:00:37] Project mwcore-phpunit-coverage-master build #1966: 04FAILURE in 35 sec: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1966/ [15:00:47] (03Merged) 10jenkins-bot: Revert "jjb: update Quibble jobs from 1.3.0 to 1.4.0" [integration/config] - 10https://gerrit.wikimedia.org/r/763979 (https://phabricator.wikimedia.org/T302226) (owner: 10Hashar) [15:05:06] 10Project-Admins: Create User-ItamarWMDE project - https://phabricator.wikimedia.org/T302225 (10Zabe) [15:08:36] (03CR) 10Hashar: [C: 03+2] "Lets go!" [integration/config] - 10https://gerrit.wikimedia.org/r/756047 (owner: 10Hashar) [15:10:33] (03Merged) 10jenkins-bot: Enforce editorconfig settings [integration/config] - 10https://gerrit.wikimedia.org/r/756047 (owner: 10Hashar) [15:17:38] (03CR) 10Hashar: "recheck" [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/754543 (owner: 10Hashar) [15:21:07] Yippee, build fixed! [15:21:07] Project mwcore-phpunit-coverage-master build #1967: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1967/ [15:28:03] (03PS1) 10Hashar: Raise timeout for gearman-java jobs [integration/config] - 10https://gerrit.wikimedia.org/r/764380 (https://phabricator.wikimedia.org/T273086) [15:28:32] (03CR) 10Hashar: "recheck timeout due to archiva.wikimedia.org" [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/754543 (owner: 10Hashar) [15:28:46] (03CR) 10Hashar: [C: 03+2] Raise timeout for gearman-java jobs [integration/config] - 10https://gerrit.wikimedia.org/r/764380 (https://phabricator.wikimedia.org/T273086) (owner: 10Hashar) [15:30:34] (03Merged) 10jenkins-bot: Raise timeout for gearman-java jobs [integration/config] - 10https://gerrit.wikimedia.org/r/764380 (https://phabricator.wikimedia.org/T273086) (owner: 10Hashar) [15:37:28] James_F: why was https://gerrit.wikimedia.org/r/c/mediawiki/extensions/BlogPage/+/759785 merged [15:37:38] It violates extensions compat policy [15:45:31] (03CR) 10Hashar: "recheck grmblblbl. Bumped timeout to 35 minutes" [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/754543 (owner: 10Hashar) [16:04:00] (03CR) 10Hashar: [C: 03+2] Add EditorConfig [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/754543 (owner: 10Hashar) [16:15:02] (03PS6) 10Hashar: Docker: Drop node10-test-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720153 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [16:17:04] (03PS5) 10Hashar: Docker: Drop node10-test, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720154 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [16:17:06] (03PS5) 10Hashar: Docker: Drop node10, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720156 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [16:17:57] (03CR) 10Hashar: [C: 03+2] "Fresh now more refers to our Node 10 images which was the last blocker ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/720153 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [16:18:01] (03CR) 10Hashar: [C: 03+2] "Fresh now more refers to our Node 10 images which was the last blocker ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/720154 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [16:18:05] (03CR) 10Hashar: "Fresh now more refers to our Node 10 images which was the last blocker ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/720156 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [16:18:12] less images! [16:21:45] (03Merged) 10jenkins-bot: Add EditorConfig [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/754543 (owner: 10Hashar) [17:13:45] (03PS1) 10Kosta Harlan: jjb: Add quibble-fullrun-sqlite job [integration/config] - 10https://gerrit.wikimedia.org/r/764420 (https://phabricator.wikimedia.org/T302226) [17:14:28] (03CR) 10Kosta Harlan: jjb: Add quibble-fullrun-sqlite job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/764420 (https://phabricator.wikimedia.org/T302226) (owner: 10Kosta Harlan) [17:32:38] (03CR) 10Hashar: [C: 03+2] jjb: Add quibble-fullrun-sqlite job [integration/config] - 10https://gerrit.wikimedia.org/r/764420 (https://phabricator.wikimedia.org/T302226) (owner: 10Kosta Harlan) [17:34:22] (03Merged) 10jenkins-bot: jjb: Add quibble-fullrun-sqlite job [integration/config] - 10https://gerrit.wikimedia.org/r/764420 (https://phabricator.wikimedia.org/T302226) (owner: 10Kosta Harlan) [17:35:11] (03CR) 10Hashar: "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/764420 (https://phabricator.wikimedia.org/T302226) (owner: 10Kosta Harlan) [17:39:09] (03CR) 10Hashar: [C: 03+2] "Dropped James blocker which was set cause fresh still required node 10. That has been removed." [integration/config] - 10https://gerrit.wikimedia.org/r/720153 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [17:39:30] (03CR) 10Hashar: [C: 03+2] Docker: Drop node10, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720156 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [17:40:30] 10Continuous-Integration-Config, 10Fresh, 10Performance-Team, 10Patch-For-Review: Remove Node.js 10 support from CI - https://phabricator.wikimedia.org/T302221 (10hashar) 05Openβ†’03Resolved [17:40:52] (03Merged) 10jenkins-bot: Docker: Drop node10-test-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720153 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [17:41:21] (03Merged) 10jenkins-bot: Docker: Drop node10-test, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720154 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [17:41:24] (03Merged) 10jenkins-bot: Docker: Drop node10, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720156 (https://phabricator.wikimedia.org/T302221) (owner: 10Jforrester) [17:42:06] James_F: node10 images are gone!!! [18:58:17] 10Release-Engineering-Team (Done by Feb 23πŸ”₯), 10Release, 10Train Deployments: 1.38.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T300199 (10Jdlrobson) The train deployer this week, may get reports of issues relating to the Special:Contributions and the history page relating to the changes...