[00:00:32] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:32] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:47] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:10:47] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [06:04:03] 10GitLab, 10Release-Engineering-Team, 10serviceops: LDAP group sync fails on gitlab1004: EOFError: EOF when reading a line - https://phabricator.wikimedia.org/T347132 (10Jelto) [06:04:28] 10GitLab, 10Release-Engineering-Team, 10collaboration-services: LDAP group sync fails on gitlab1004: EOFError: EOF when reading a line - https://phabricator.wikimedia.org/T347132 (10Jelto) [07:36:53] I am restarting Gerrit for https://gerrit.wikimedia.org/r/c/operations/puppet/+/953967 [07:52:46] 10GitLab, 10Release-Engineering-Team, 10collaboration-services: LDAP group sync fails on gitlab1004: EOFError: EOF when reading a line - https://phabricator.wikimedia.org/T347132 (10eoghan) @dancy has a patch for this: https://gerrit.wikimedia.org/r/c/operations/puppet/+/959881 [07:56:20] Yippee, build fixed! [07:56:20] Project beta-scap-sync-world build #121738: 09FIXED in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/121738/ [08:16:32] (03CR) 10Hashar: [C: 03+2] dockerfiles: update Quibble to 1.5.6 [integration/config] - 10https://gerrit.wikimedia.org/r/959804 (https://phabricator.wikimedia.org/T342088) (owner: 10Hashar) [08:17:47] (03Merged) 10jenkins-bot: dockerfiles: update Quibble to 1.5.6 [integration/config] - 10https://gerrit.wikimedia.org/r/959804 (https://phabricator.wikimedia.org/T342088) (owner: 10Hashar) [08:23:38] 10GitLab, 10Release-Engineering-Team, 10collaboration-services: LDAP group sync fails on gitlab1004: EOFError: EOF when reading a line - https://phabricator.wikimedia.org/T347132 (10Jelto) 05Open→03Resolved p:05Triage→03Medium a:03eoghan Thanks for spotting and applying the change. This is resolved... [08:32:38] (03CR) 10Hashar: [C: 03+2] "Successfully published image docker-registry.discovery.wmnet/releng/quibble-buster-php80:1.5.6" [integration/config] - 10https://gerrit.wikimedia.org/r/959804 (https://phabricator.wikimedia.org/T342088) (owner: 10Hashar) [08:33:25] (03PS1) 10Hashar: jjb: update integration/quibble jobs to Quibble 1.5.6 [integration/config] - 10https://gerrit.wikimedia.org/r/959951 [08:33:39] (03CR) 10Hashar: [C: 03+2] jjb: update integration/quibble jobs to Quibble 1.5.6 [integration/config] - 10https://gerrit.wikimedia.org/r/959951 (owner: 10Hashar) [08:34:52] (03Merged) 10jenkins-bot: jjb: update integration/quibble jobs to Quibble 1.5.6 [integration/config] - 10https://gerrit.wikimedia.org/r/959951 (owner: 10Hashar) [09:16:20] is it a known issue that assertion failures for Message objects result in absolutely humongous CI outputs? [09:16:59] the consoleFull of https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium-docker/127002/console is apparently 64 MB, some 325k lines [09:17:14] because the var dumped Message object contains way too many nested objects [09:22:10] I guess it’s related to https://phabricator.wikimedia.org/T277618 (but PHPUnit isn’t directly using var_dump() so the fix for that task doesn’t affect it?) [09:26:26] 10Phabricator maintenance bot, 10Commons: Review Global Herald H109 - Add #Commons project to #MediaWiki-File-management and #MediaWiki-extensions-GWToolset - https://phabricator.wikimedia.org/T250026 (10Aklapper) 05Resolved→03Open [09:27:13] Lucas_WMDE: I have never head of that before [09:27:44] I am guessing that is the custom PHPUnit test failure reporter being at fault [09:27:56] it prints the logs generated during the test [09:28:08] + whatever code doing some full dump over wfDebug() [09:29:12] 1) Wikibase\Lexeme\Tests\Unit\mediawiki\LexemeHistoryActionTest::testFallsBackToParentIfNoLexemeId [09:29:12] Failed asserting that Message Object &000000003075b419000000006090625b ( [09:29:32] or that is PHPUnit doing a recursive full dump of the object without limitation [09:30:15] there is also the whole Common/lualib/mw.lua file content :) [09:47:51] yeah, it’s in some HashBagOStuff it seems [10:13:26] hm, if I try it locally, the result looks much more reasonable [10:13:37] (with a test that just does: $this->assertEquals( 'foo', wfMessage( 'parentheses', 'foo' ) ); ) [10:13:49] maybe there was something about this particular test that bloated the Message instance [10:14:57] also I just realized that the CI job ran for exactly an hour [10:15:20] that’s why the output cuts off without closing all the nested objects – it just timed out while it was still busy dumping the Message o_O [10:17:01] ok, if I make it wfMessage( 'foo' )->inLanguage( $this->getServiceContainer()->getContentLanguage() ), then it takes four seconds to run and dumps 84k lines ^^ [10:17:22] (including my whole environment, which is funny because it contains some terminal escape sequences for `less` which change the color for the dump partway through ^^) [10:17:35] should I make a phab task? [10:20:26] 10GitLab (Infrastructure), 10Release-Engineering-Team, 10collaboration-services: GitLab email confirmation mail ends up in spam folder - https://phabricator.wikimedia.org/T346607 (10LSobanski) I tried resending confirmation instructions via https://gitlab.wikimedia.org/users/confirmation/new but didn't recei... [10:30:06] meh, it looks like this might not be fixable :/ [10:30:30] IIUC PHPUnit uses the sebastian/exporter library, and I don’t see any support for excluding certain properties in https://github.com/sebastianbergmann/exporter/blob/5.1.0/src/Exporter.php [10:30:40] it dumps all the properties whether we like it or not [11:22:06] hashar: i just noticed on a gerrit change that i have the wrong author https://gerrit.wikimedia.org/r/c/operations/puppet/+/960022 [11:22:30] the change was authorded by me on the laptop and there was only one patrch set so im not sure what the issues could be, any idea? [11:23:25] that cr was submited with this one https://gerrit.wikimedia.org/r/c/operations/software/puppet-compiler/+/959827/ (which looks correct) [11:24:04] ignore that last comment they where not submited to gether [11:26:21] i tested a random cr (https://gerrit.wikimedia.org/r/c/operations/puppet/+/960024) and didn't get the same behaviour [11:50:43] jbond: hmm [11:52:35] patchset 1 has eoghan as `Author` https://gerrit.wikimedia.org/r/c/operations/puppet/+/960022/1//COMMIT_MSG [11:53:30] yes i saw that but ot quite sure how :/ [11:53:49] when it got submitted, Gerrit automatically rebased it behind your hood to make it a fast forward commit [11:55:45] which is reflected in the generated patchset 2 by the `Commit` author being changed from your local lower case `jbond` to `Jbond` which I am guessing is your Gerrit "full name" [11:57:54] hashar: i get that im confused why eoghan was the author on PS1 [11:58:16] but i guess that could be some strange local issue (its not repeated) [11:58:56] at least from Gerrit I see the change being authored by you and Eoghan was put on CC (cause he was the Author in the commit message) [11:59:04] maybe a cherrypick/rebase? [11:59:10] o_O [11:59:28] im not sure, ill ignore for now and try to get more info if it happens again [12:00:02] also patchset 1 had for parent https://gerrit.wikimedia.org/r/c/operations/puppet/+/960022 [12:00:14] err [12:00:21] yes i noticed that as well which is owned by eoghan [12:00:44] the parent was a change by Balthazar [12:03:27] That one I did a puppet-merge on, since it was landing at the same time as mine. [12:04:14] sorry for all the pings eoghan i dont think this is anything you have done much more likley something with me [12:04:25] It's fine (: [12:04:26] maybe it is a weird race condition inside Gerrit [12:05:05] yes could be, have not seen it before [12:05:05] something like while Change B is received it get assigned data from Change A being submitted [12:06:24] 10Phabricator, 10Patch-For-Review: Development environment with deployable Phab/Phorge - https://phabricator.wikimedia.org/T343989 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/scap3-dev/-/merge_requests/30 phorge: add service [12:07:31] then the timing does not match [12:08:43] jbond: I am willing to ignore it, assuming somehow you ended up sending a commit with the wrong Author: somehow [12:09:00] with still having a possibility that maybe GErrit got confused on the server side, but that will really surprise me [12:09:02] hashar: yes lets ignore it for now unless we see it again [12:09:07] +1 [12:09:13] thanks [12:19:27] 10Phabricator, 10Patch-For-Review: Development environment with deployable Phab/Phorge - https://phabricator.wikimedia.org/T343989 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/scap3-dev/-/merge_requests/30 phorge: add service [12:23:57] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.41-release: Expand the set of bundled extensions and skins in MediaWiki 1.41 - https://phabricator.wikimedia.org/T333405 (10Jdforrester-WMF) Tagging against 1.41, though in practice it doesn't seem there's been a... [12:44:59] (03PS1) 10Xqt: update dockerfiles/tox-pywikibot/example-run.sh [integration/config] - 10https://gerrit.wikimedia.org/r/960047 [12:50:04] 10Phabricator: Development environment with deployable Phab/Phorge - https://phabricator.wikimedia.org/T343989 (10jnuche) 05Open→03Resolved a:03jnuche First version of the environment is ready to use. Any missing production setup (such as the notification server or the Phab daemons) can be added in the fut... [12:55:21] (03CR) 10Hashar: [C: 03+2] update dockerfiles/tox-pywikibot/example-run.sh [integration/config] - 10https://gerrit.wikimedia.org/r/960047 (owner: 10Xqt) [12:56:42] (03Merged) 10jenkins-bot: update dockerfiles/tox-pywikibot/example-run.sh [integration/config] - 10https://gerrit.wikimedia.org/r/960047 (owner: 10Xqt) [12:59:58] (03PS1) 10Jbond: operations-puppet-catalog-compiler: add support for Change-Private [integration/config] - 10https://gerrit.wikimedia.org/r/960051 (https://phabricator.wikimedia.org/T265633) [13:24:55] (03CR) 10Hashar: [C: 03+2] "Job updated!" [integration/config] - 10https://gerrit.wikimedia.org/r/960051 (https://phabricator.wikimedia.org/T265633) (owner: 10Jbond) [13:26:52] (03Merged) 10jenkins-bot: operations-puppet-catalog-compiler: add support for Change-Private [integration/config] - 10https://gerrit.wikimedia.org/r/960051 (https://phabricator.wikimedia.org/T265633) (owner: 10Jbond) [13:37:01] 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - operations/cookbooks - https://phabricator.wikimedia.org/T341474 (10LSobanski) Running tox in a CI job is supported, see https://gitlab.wikimedia.org/repos/sre/schema-changes/-/ci/editor?branch_name=main for an example. [13:37:10] 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - operations/cookbooks - https://phabricator.wikimedia.org/T341474 (10LSobanski) [14:12:43] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10Pywikibot, 10Upstream: tox v4 with skipsdist=true does not recognize use_develop=true - https://phabricator.wikimedia.org/T346238 (10hashar) [14:13:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10Pywikibot, 10Upstream: tox v4 with skipsdist=true does not recognize use_develop=true - https://phabricator.wikimedia.org/T346238 (10hashar) [14:16:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10Pywikibot, 10Upstream: tox v4 with skipsdist=true does not recognize use_develop=true - https://phabricator.wikimedia.org/T346238 (10hashar) [14:23:14] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10Pywikibot, 10Patch-For-Review, 10Upstream: tox v4 with skipsdist=true does not recognize use_develop=true - https://phabricator.wikimedia.org/T346238 (10hashar) [14:25:09] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10Pywikibot, 10Patch-For-Review, 10Upstream: tox v4 with skipsdist=true does not recognize use_develop=true - https://phabricator.wikimedia.org/T346238 (10hashar) [14:26:20] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10Pywikibot, 10Patch-For-Review, 10Upstream: tox v4 with skipsdist=true does not recognize use_develop=true - https://phabricator.wikimedia.org/T346238 (10hashar) As far as I can tell, the three affected repositories are p... [14:56:08] 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - September 2023 - https://phabricator.wikimedia.org/T345590 (10eoghan) [15:32:58] (03PS1) 10Hashar: Require tox v4 and remove skipsdist/use_develop [integration/quibble] - 10https://gerrit.wikimedia.org/r/960082 (https://phabricator.wikimedia.org/T346238) [15:33:28] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10Pywikibot, 10Patch-For-Review, 10Upstream: tox v4 with skipsdist=true does not recognize use_develop=true - https://phabricator.wikimedia.org/T346238 (10hashar) [15:34:33] (03CR) 10CI reject: [V: 04-1] Require tox v4 and remove skipsdist/use_develop [integration/quibble] - 10https://gerrit.wikimedia.org/r/960082 (https://phabricator.wikimedia.org/T346238) (owner: 10Hashar) [15:38:06] (03PS1) 10Hashar: Require tox v4 and remove skipsdist/use_develop [integration/tox-jenkins-override] - 10https://gerrit.wikimedia.org/r/960086 (https://phabricator.wikimedia.org/T346238) [15:38:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10Pywikibot, 10Patch-For-Review, 10Upstream: tox v4 with skipsdist=true does not recognize use_develop=true - https://phabricator.wikimedia.org/T346238 (10hashar) [15:42:21] (03CR) 10Jbond: Add Puppet 7 catalogue compiler job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/937618 (https://phabricator.wikimedia.org/T236373) (owner: 10Hashar) [15:42:52] (03PS1) 10Hashar: jjb: switch integration/quibble tox jobs to tox v4 [integration/config] - 10https://gerrit.wikimedia.org/r/960091 (https://phabricator.wikimedia.org/T345695) [15:43:43] (03PS2) 10Hashar: Require tox v4 and remove skipsdist/use_develop [integration/quibble] - 10https://gerrit.wikimedia.org/r/960082 (https://phabricator.wikimedia.org/T346238) [15:44:20] (03CR) 10Hashar: "I have updated the jobs and testing with https://gerrit.wikimedia.org/r/c/integration/quibble/+/960082/" [integration/config] - 10https://gerrit.wikimedia.org/r/960091 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [15:46:11] (03CR) 10Hashar: [C: 03+2] jjb: switch integration/quibble tox jobs to tox v4 [integration/config] - 10https://gerrit.wikimedia.org/r/960091 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [15:46:34] (03CR) 10Hashar: "I have switched the CI jobs to use tox v4.8.0 :)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/960082 (https://phabricator.wikimedia.org/T346238) (owner: 10Hashar) [15:47:26] (03Merged) 10jenkins-bot: jjb: switch integration/quibble tox jobs to tox v4 [integration/config] - 10https://gerrit.wikimedia.org/r/960091 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [15:51:29] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mabualruz - https://phabricator.wikimedia.org/T345186 (10colewhite) [17:55:34] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T345888 (10thcipriani) 05Open→03Resolved [18:25:44] 10Release-Engineering-Team: Automatically handle manual release notes for MediaWiki - https://phabricator.wikimedia.org/T347207 (10thcipriani) [19:08:07] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.41.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T347080 (10thcipriani) p:05Triage→03Medium a:03jeena [19:39:44] 10Release-Engineering-Team: Automatically handle manual release notes for MediaWiki - https://phabricator.wikimedia.org/T347207 (10Quiddity) [19:41:31] !log Reconfigure https://packagist.org/packages/wikimedia/metrics-platform soure from GitLab back to github.com to fix absence of "dist" source when rebuilding mediawiki-vendor. Reef https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/960127 [19:41:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:45:43] 10GitLab (Pipeline Services Migration🐤), 10Metrics Platform Backlog, 10Data Products (Sprint 01), 10Patch-For-Review: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10Krinkle) I've reconfigured the `wikimedia/metrics-platform` package back from GitLab to github.com bec... [19:59:58] brennen: could use CR on these two patches - https://gerrit.wikimedia.org/r/q/project:performance%252FWikimediaDebug [20:00:22] not sure if there's a Gerrit dashboard to include this repo in a query on or something, but FYI :) [20:07:26] derp. should add that repo to our "Let's keep this empy" section https://gerrit.wikimedia.org/r/p/wikimedia/+/dashboard/teams:releng [20:10:45] (03CR) 10Brennen Bearnes: [C: 03+2] docs: Fix link in 2.8.0 changelog [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/948207 (owner: 10Krinkle) [20:11:11] (03Merged) 10jenkins-bot: docs: Fix link in 2.8.0 changelog [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/948207 (owner: 10Krinkle) [20:12:13] (03CR) 10Brennen Bearnes: [C: 03+2] build: Add `npm run changelog` command to help with releases [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/949108 (owner: 10Krinkle) [20:12:37] (03Merged) 10jenkins-bot: build: Add `npm run changelog` command to help with releases [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/949108 (owner: 10Krinkle) [20:49:39] Thx :) [21:00:39] 10GitLab (Pipeline Services Migration🐤), 10Metrics Platform Backlog, 10Data Products (Sprint 01), 10Patch-For-Review: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10thcipriani) >>! In T344733#9186878, @Reedy wrote: >>>! In T344733#9186854, @phuedx wrote: >> @thcipria...