[00:28:37] CI experts: I could use a hand understanding why this is getting a -1 https://integration.wikimedia.org/ci/job/operations-puppet-tests-buster-docker/40877/console [00:38:20] andrewbogott: 00:27:03 E AssertionError: '/usr/bin/timeout: failed to run command [43 chars]tory' != "/usr/bin/timeout: failed to run command [43 chars]tory" [00:38:20] 00:27:03 E - /usr/bin/timeout: failed to run command ‘nonexistentcommand’: No such file or directory [00:38:31] 00:27:03 FAILED modules/smart/files/tests/test_smart_data_dump.py::TestSmartDataDump::test_suppressed_errors_cmd [00:39:03] but...you're not touching that at all [00:39:20] not sure then, ignore me [00:44:58] since T303092 I'm keeping well away from CI issues for a bit :P [00:45:39] T303092: quibble-vendor-mysql-php72-noselenium-docker fails for a noop PageTriage patch - https://phabricator.wikimedia.org/T303092 [01:08:04] legoktm: yep, a timeout failure in a test I've never heard of, hence my question [01:41:45] 10MediaWiki-Releasing: Make MediaWiki (major) release task tree - https://phabricator.wikimedia.org/T303238 (10Reedy) [01:42:01] 10MediaWiki-Releasing, 10MediaWiki-Documentation: Make MediaWiki (major) release task tree - https://phabricator.wikimedia.org/T303238 (10Reedy) [01:50:37] (03PS1) 10Krinkle: dockerfiles: Upgrade Doxygen to 1.9.1 [integration/config] - 10https://gerrit.wikimedia.org/r/768843 (https://phabricator.wikimedia.org/T295111) [01:50:50] 10MediaWiki-Releasing, 10MediaWiki-Documentation: Make MediaWiki (major) release task tree - https://phabricator.wikimedia.org/T303238 (10Reedy) [01:50:57] (03PS2) 10Krinkle: dockerfiles: Upgrade Doxygen to 1.9.1 [integration/config] - 10https://gerrit.wikimedia.org/r/768843 (https://phabricator.wikimedia.org/T295111) [04:27:16] (03CR) 10Brennen Bearnes: [C: 03+1] "I _think_ this seems reasonable, I note that it builds, and I don't quite know how to test it." [blubber] - 10https://gerrit.wikimedia.org/r/504651 (https://phabricator.wikimedia.org/T301169) (owner: 10Dduvall) [06:46:38] (03CR) 10Kevin Bazira: [C: 03+1] "LGTM!" [integration/config] - 10https://gerrit.wikimedia.org/r/768785 (https://phabricator.wikimedia.org/T294419) (owner: 10Accraze) [08:57:40] (03CR) 10Hashar: [C: 03+2] Remove window_name vars from backport windows [tools/release] - 10https://gerrit.wikimedia.org/r/763780 (owner: 10Lucas Werkmeister (WMDE)) [08:58:33] (03Merged) 10jenkins-bot: Remove window_name vars from backport windows [tools/release] - 10https://gerrit.wikimedia.org/r/763780 (owner: 10Lucas Werkmeister (WMDE)) [09:02:35] (03CR) 10Hashar: [C: 03+2] "Excellent! :)" [integration/config] - 10https://gerrit.wikimedia.org/r/768785 (https://phabricator.wikimedia.org/T294419) (owner: 10Accraze) [09:04:25] (03Merged) 10jenkins-bot: inference-services: draftquality cleanup [integration/config] - 10https://gerrit.wikimedia.org/r/768785 (https://phabricator.wikimedia.org/T294419) (owner: 10Accraze) [09:17:51] (03CR) 10Hashar: [V: 03+2 C: 03+2] Prevent accidental branch push by team members [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767703 (owner: 10Hashar) [09:18:32] (03CR) 10Hashar: [V: 03+2 C: 03+2] Prevent accidental branch push by team members [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767703 (owner: 10Hashar) [09:32:43] (03CR) 10Hashar: "Isabelle and I verified it this morning. You can still push tags but branches push are denied by Gerrit." [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767703 (owner: 10Hashar) [10:10:42] (03PS1) 10Jaime Nuche: deploy-promote: integrate into `scap` [tools/scap] - 10https://gerrit.wikimedia.org/r/769023 (https://phabricator.wikimedia.org/T302488) [10:11:25] (03CR) 10Jaime Nuche: [C: 04-2] "Can't be merged yet" [tools/scap] - 10https://gerrit.wikimedia.org/r/769023 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [10:23:09] (03CR) 10Hashar: [C: 04-1] "Looks like we should port releng/tox to Bullseye first (which in turn requires building various python version for it)." [integration/config] - 10https://gerrit.wikimedia.org/r/767877 (https://phabricator.wikimedia.org/T278203) (owner: 10Jforrester) [10:32:27] (03PS1) 10Jaime Nuche: deploy-promote: migrate script to `tools/scap` repository [tools/release] - 10https://gerrit.wikimedia.org/r/769026 (https://phabricator.wikimedia.org/T30248) [10:37:17] (03CR) 10Jaime Nuche: [C: 04-2] "Can't be merged yet" [tools/release] - 10https://gerrit.wikimedia.org/r/769026 (https://phabricator.wikimedia.org/T30248) (owner: 10Jaime Nuche) [10:48:20] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T300201 (10kostajh) [11:27:24] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T300202 (10kostajh) [11:27:26] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T300201 (10kostajh) [11:35:36] 10Continuous-Integration-Config, 10Quibble, 10Platform Team Initiatives (API Integration Tests): Should core apitests job run extension/skin api tests as well? - https://phabricator.wikimedia.org/T303255 (10kostajh) [12:08:32] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T300202 (10kostajh) [14:01:35] 10Continuous-Integration-Infrastructure, 10Quibble: Parallel npm install times out in Quibble - https://phabricator.wikimedia.org/T303270 (10hashar) [14:02:30] 10Continuous-Integration-Infrastructure, 10Quibble: Parallel npm install times out in Quibble - https://phabricator.wikimedia.org/T303270 (10hashar) [14:12:32] 10Continuous-Integration-Infrastructure, 10Quibble: Parallel npm install times out in Quibble - https://phabricator.wikimedia.org/T303270 (10hashar) I have digged in the Quibble code and tried to reproduce the issue but haven't found anything conclusive. I lost myself in cpython threading module and found htt... [14:26:21] (03PS2) 10Jaime Nuche: deploy-promote: integrate into `scap` [tools/scap] - 10https://gerrit.wikimedia.org/r/769023 (https://phabricator.wikimedia.org/T302488) [14:27:16] Hello all! Can anyone advise me about this CI failure? (I'm pretty sure it's a CI bug and not an actual issue with my patch) [14:27:17] https://integration.wikimedia.org/ci/job/operations-puppet-tests-buster-docker/40877/console [14:27:38] 00:27:03 E AssertionError: '/usr/bin/timeout: failed to run command [43 chars]tory' != "/usr/bin/timeout: failed to run command [43 chars]tory" [15:08:53] 10Continuous-Integration-Infrastructure, 10Quibble: Parallel npm install times out in Quibble - https://phabricator.wikimedia.org/T303270 (10hashar) [15:15:06] andrewbogott: checking ;) [15:16:01] thx hashar! I'm breaking that patch down into smaller pieces just in case that timeout error is just because of a too-big patch [15:16:28] the failing test is in puppet: modules/smart/files/tests/test_smart_data_dump.py [15:18:29] this feels oddly familiar! https://gerrit.wikimedia.org/r/c/operations/puppet/+/736468 [15:19:03] looks like a quote mismatch addressed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/736468 [15:19:07] I have no clue why though [15:19:52] iirc I did that to fix a very similar error [15:20:12] cause the `/usr/bin/timeout` slightly changed its output [15:20:18] then that fix got committed in novemeber [15:20:24] so I don't see why it would surface again [15:20:26] :/ [15:20:36] I really wonder what's changing here [15:21:00] let's see if my revert https://gerrit.wikimedia.org/r/c/operations/puppet/+/768799 passes CI [15:21:44] Does touching tox.ini trigger tests that wouldn't otherwise run? [15:22:18] I can't remember off hand [15:22:25] yes, otherwise jerkins tries to be smart and only run the tests for the files you changed [15:22:28] but Puppet.git has some optization to avoid running everything [15:22:42] so yeah maybe touching tox.ini causes tox to be run [15:22:48] and the test has been failing for a while [15:23:00] but cripples only people that touch tox.ini (or anything that causes tox to be run) [15:23:39] so I would assume it has failed for more than a few people since november 3 [15:24:04] the revert passes CI [15:26:43] * andrewbogott making breakfast but alert to any suggested fix [15:28:34] andrewbogott: https://gerrit.wikimedia.org/r/c/operations/puppet/+/768799 is my suggested fix [15:29:32] E AttributeError: module 'unittest' has no attribute 'mock' [15:29:32] grmblbl [15:35:59] I have patches :) [15:40:06] taavi: andrewbogott: https://gerrit.wikimedia.org/r/c/operations/puppet/+/769060 smart: use a regex to check command error [15:40:26] and another one to ensure the output message is using the C language (== englihs) https://gerrit.wikimedia.org/r/c/operations/puppet/+/769061 smart: set LC_MESSAGES when testing output [15:40:31] cause the test failed for me [15:40:36] or we get cwhite to review those maybe [15:41:38] thanks! [15:41:47] as for why taavi patch at https://gerrit.wikimedia.org/r/c/operations/puppet/+/768799 passed, I don't know [15:42:01] maybe we cause we got the docker image to use a newer version of the timeout command [15:42:05] and that specific test now fails [15:42:11] the regex would catch both cases :] [15:42:14] hey folks. [15:42:18] I have added cwhite as a reviewer [15:43:01] o/ hey dancy! [15:50:05] I filed the task about Quibble hanging somehow which we talked about here (last friday iirc) https://phabricator.wikimedia.org/T303270 [15:50:12] it happens from time to time [15:55:04] (03CR) 10Jforrester: "Is it OK for us to regress from WMF-provided PHP? Should we mention this in the changelog?" [integration/config] - 10https://gerrit.wikimedia.org/r/768843 (https://phabricator.wikimedia.org/T295111) (owner: 10Krinkle) [16:06:12] (03PS3) 10Krinkle: dockerfiles: Upgrade Doxygen to 1.9.1 [integration/config] - 10https://gerrit.wikimedia.org/r/768843 (https://phabricator.wikimedia.org/T295111) [16:08:32] (03CR) 10Krinkle: dockerfiles: Upgrade Doxygen to 1.9.1 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/768843 (https://phabricator.wikimedia.org/T295111) (owner: 10Krinkle) [16:08:41] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T300201 (10Jdlrobson) [16:11:53] (03PS3) 10Jaime Nuche: deploy-promote: integrate into `scap` [tools/scap] - 10https://gerrit.wikimedia.org/r/769023 (https://phabricator.wikimedia.org/T302488) [16:15:03] (03CR) 10Jforrester: [C: 03+1] dockerfiles: Upgrade Doxygen to 1.9.1 [integration/config] - 10https://gerrit.wikimedia.org/r/768843 (https://phabricator.wikimedia.org/T295111) (owner: 10Krinkle) [16:28:15] (03PS1) 10Jaime Nuche: train: use the new version of `deploy-promote` integrated in `scap` [tools/train-dev] - 10https://gerrit.wikimedia.org/r/769073 (https://phabricator.wikimedia.org/T302488) [16:36:56] (03CR) 10Jaime Nuche: [C: 04-2] "Cannot merge yet" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/769073 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [16:48:12] hashar: now my patch passes! Thanks [16:53:14] andrewbogott: magic team work!!! :-) [17:03:47] (03CR) 10Ahmon Dancy: deploy-promote: integrate into `scap` (0320 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/769023 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [17:04:56] (03CR) 10Ahmon Dancy: [C: 03+1] deploy-promote: migrate script to `tools/scap` repository [tools/release] - 10https://gerrit.wikimedia.org/r/769026 (https://phabricator.wikimedia.org/T30248) (owner: 10Jaime Nuche) [17:20:45] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10AlexisJazz) >>! In T302699#7756446, @Vgutierrez wrote: > in this case a 502 is emitted by ats-backend cause it is... [17:31:46] kostajh: is there a task already about apache vs sqlite builds? It seems like we may want to turn that off again. https://phabricator.wikimedia.org/T303225 [17:31:54] keeping it for mysql-based selenium/qunit tests seems useful though [17:32:26] also - maybe we don't need to run selenium/qunit in sqlite as well. I'd expect phpunit to sufficiently cover that for things that care about backend things. [17:32:56] maybe this is a leftover optimissation from a time where sqlite was the "fast" job variant. [17:33:09] (this has been obsoleted by mysql for many years now) [17:35:57] (03CR) 10Ahmon Dancy: Provide scap prep auto history browsing and replay (038 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/763864 (https://phabricator.wikimedia.org/T301417) (owner: 10Dduvall) [17:41:38] (03CR) 10Ahmon Dancy: [C: 03+1] Update all dependencies and stop using dep (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/768179 (https://phabricator.wikimedia.org/T301169) (owner: 10Dduvall) [17:51:35] (03CR) 10Ahmon Dancy: Support blubber as a buildkit gateway and Dockerfile frontend (034 comments) [blubber] - 10https://gerrit.wikimedia.org/r/504651 (https://phabricator.wikimedia.org/T301169) (owner: 10Dduvall) [17:52:09] (03CR) 10Ahmon Dancy: [C: 03+1] train: use the new version of `deploy-promote` integrated in `scap` [tools/train-dev] - 10https://gerrit.wikimedia.org/r/769073 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [18:23:19] (03CR) 10Dduvall: [C: 04-1] "Thanks for the reviews! I'm unable to get this to work via `docker build` or `buildctl`. I'm currently debugging. I'll try to implement an" [blubber] - 10https://gerrit.wikimedia.org/r/504651 (https://phabricator.wikimedia.org/T301169) (owner: 10Dduvall) [18:28:21] (03CR) 10Ahmon Dancy: Support blubber as a buildkit gateway and Dockerfile frontend (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/504651 (https://phabricator.wikimedia.org/T301169) (owner: 10Dduvall) [18:38:13] two test failures on a pipelinelib project (Toolhub) due to 503 response from https://blubberoid.wikimedia.org [18:40:24] i think just fallout from known issues; guessing a number of jobs will have to be re-run when things are definitely clear [19:21:46] (03PS1) 10Ahmon Dancy: Redirect image build output to a file [tools/scap] - 10https://gerrit.wikimedia.org/r/769097 (https://phabricator.wikimedia.org/T297673) [19:52:32] 10Release-Engineering-Team (Next), 10Code-Stewardship-Reviews, 10Documentation: Document checklist steps to undeploy / sunset a codebase on WMF servers (not: archiving) - https://phabricator.wikimedia.org/T294329 (10LNguyen) [20:28:22] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23 🧟): Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10brennen) > Understood. Having said this GitLab group is still useful and thanks for that; then, if the staff needs to create private T... [20:31:16] !log requiring 2fa for all users under /repos [20:31:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:39:58] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23 🧟): Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10brennen) 05In progress→03Resolved I've added all requested users to the group, and @CVimercati-WMIT should be able to manage membe... [20:43:36] 10GitLab (Project Migration), 10Release-Engineering-Team (Doing), 10Product-Analytics: Create new GitLab people & project groups for Product Analytics - https://phabricator.wikimedia.org/T302727 (10brennen) 05Open→03In progress a:03brennen [20:55:21] 10GitLab (Project Migration), 10Release-Engineering-Team (Doing), 10Product-Analytics: Create new GitLab people & project groups for Product Analytics - https://phabricator.wikimedia.org/T302727 (10brennen) 05In progress→03Resolved Actions taken: - Created https://gitlab.wikimedia.org/repos/product-an... [21:15:05] (03PS1) 10RLazarus: helm-linter: Update envoyproxy to 1.18.3 [integration/config] - 10https://gerrit.wikimedia.org/r/769119 (https://phabricator.wikimedia.org/T300324) [22:27:53] (03CR) 10Ahmon Dancy: deploy-promote: integrate into `scap` (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/769023 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [23:47:46] (03PS10) 10Dduvall: Support blubber as a buildkit gateway and Dockerfile frontend [blubber] - 10https://gerrit.wikimedia.org/r/504651 (https://phabricator.wikimedia.org/T301169) [23:47:50] (03CR) 10Dduvall: [C: 04-1] Support blubber as a buildkit gateway and Dockerfile frontend (032 comments) [blubber] - 10https://gerrit.wikimedia.org/r/504651 (https://phabricator.wikimedia.org/T301169) (owner: 10Dduvall) [23:52:04] (03CR) 10jerkins-bot: [V: 04-1] Support blubber as a buildkit gateway and Dockerfile frontend [blubber] - 10https://gerrit.wikimedia.org/r/504651 (https://phabricator.wikimedia.org/T301169) (owner: 10Dduvall) [23:52:51] (03CR) 10Dduvall: [C: 03+1] "I've fixed some issues with the revived change, namely that dependency graphs (new relative to the original patchsets) needed to be initia" [blubber] - 10https://gerrit.wikimedia.org/r/504651 (https://phabricator.wikimedia.org/T301169) (owner: 10Dduvall) [23:55:12] (03PS11) 10Dduvall: Support blubber as a buildkit gateway and Dockerfile frontend [blubber] - 10https://gerrit.wikimedia.org/r/504651 (https://phabricator.wikimedia.org/T301169)