[00:26:07] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834#12206203 (10Daimona) [02:20:10] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for gaiarajan - https://phabricator.wikimedia.org/T434626 (10gaiarajan) 03NEW [02:54:25] 10Continuous-Integration-Config, 07ci-test-error: CI on gerrit can't run git - citoid-pipeline-test - https://phabricator.wikimedia.org/T434515#12206360 (10Peachey88) [02:58:20] 06Project-Admins: Create project tag for MediaWiki-extensions-WandaVoice - https://phabricator.wikimedia.org/T434418#12206363 (10Peachey88) @sanjaisid Are you planning to turn off github issues once the phabricator is created? [03:12:16] 06Project-Admins: Create project tag for MediaWiki-extensions-WandaVoice - https://phabricator.wikimedia.org/T434418#12206365 (10sanjaisid) @Peachey88 Yes, you are right [05:16:10] 10Continuous-Integration-Config, 07ci-test-error: CI on gerrit can't run git - citoid-pipeline-test - https://phabricator.wikimedia.org/T434515#12206408 (10Mvolz) Seems resolved? [05:16:15] 10Continuous-Integration-Config, 07ci-test-error: CI on gerrit can't run git - citoid-pipeline-test - https://phabricator.wikimedia.org/T434515#12206409 (10Mvolz) 05Open→03Resolved [06:26:04] (03CR) 10Phedenskog: [C:03+1] jjb: skip npm install for CodeHealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1324323 (https://phabricator.wikimedia.org/T427822) (owner: 10Hashar) [07:11:18] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12206479 (10ABran-WMF) [07:18:28] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: inference-services-pipeline-pre-commit-check failing with JDK error after Java upgrade - https://phabricator.wikimedia.org/T434572#12206488 (10hashar) [07:18:29] 10Continuous-Integration-Config, 07ci-test-error: CI on gerrit can't run git - citoid-pipeline-test - https://phabricator.wikimedia.org/T434515#12206491 (10hashar) →14Duplicate dup:03T434572 [07:33:30] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12206548 (10ABran-WMF) gitlab-replica-b is behind CDN and behaving as intended. [08:17:27] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12206717 (10LSobanski) 05Open→03Stalled >>! In T431003#12160867, @Dzahn wrote: > Re-enabled puppet on zuul1001 where I had applied [... [08:17:31] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services, 06tools-platform-team: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12206720 (10LSobanski) [08:17:42] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12206723 (10LSobanski) p:05Triage→03Medium [08:40:35] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Doing 😎), 06Collaboration-Services: contint1003 is missing docker-buildx / auth to registry fails, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12206795 (10LSobanski) 05In progress... [08:45:40] (03open) 10phedenskog: Only count completed runs in critical-path tables [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/11 [08:53:13] 10Phabricator: Add "thank you" to award token - https://phabricator.wikimedia.org/T434584#12206822 (10Aklapper) Well, I don't interpret 🙏 as a "Thank You" token. It's "Folded Hands" which "can indicate sorrow or regret; can also indicate pleading, praying, bowing, or thanking". I'd prefer not to add such an ambi... [08:53:17] 10Phabricator: Add "thank you" to award token - https://phabricator.wikimedia.org/T434584#12206824 (10Aklapper) p:05Triage→03Low [08:53:44] 10Gerrit, 06Collaboration-Services, 06Infrastructure-Foundations, 13Patch-For-Review, 07Puppet: Change puppet-merge git origin to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org - https://phabricator.wikimedia.org/T420184#12206826 (10LSobanski) a:03ABran-WMF [09:02:00] 10Gerrit, 06Release-Engineering-Team: Automatically abandon old changes after some time - https://phabricator.wikimedia.org/T393376#12206889 (10LSobanski) [09:45:04] 10Phabricator: Add "thank you" to award token - https://phabricator.wikimedia.org/T434584#12206965 (10Aklapper) > acknowledge comments without writing a comment FYI "Allow awarding Tokens to individual Comments" is upstream https://we.phorge.it/T15048 [11:08:31] 10Phabricator (Upstream), 07Upstream: Phabricator TOTP token should be evaluated against the currently valid one instead of expiring - https://phabricator.wikimedia.org/T434520#12207235 (10Aklapper) [11:19:39] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Infra: Make sure our cluster VMs are running on different hypervisor hosts (+piggybacked machine upgrades) - https://phabricator.wikimedia.org/T426874#12207272 (10jnuche) Rollout plan: * Scale down services: * `kubec... [11:56:55] (03CR) 10Hashar: "I have updated the two jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/1324323 (https://phabricator.wikimedia.org/T427822) (owner: 10Hashar) [11:59:41] !log integration: on Castor: `sudo rm -fR /srv/castor/*/*/*codehealth*/npm` # T427822 [11:59:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:59:44] T427822: Investigate multiple npm cache folders for mwext-codehealth-master-non-voting - https://phabricator.wikimedia.org/T427822 [12:06:35] (03CR) 10Hashar: [C:03+2] "Done" [integration/config] - 10https://gerrit.wikimedia.org/r/1324323 (https://phabricator.wikimedia.org/T427822) (owner: 10Hashar) [12:08:37] (03Merged) 10jenkins-bot: jjb: skip npm install for CodeHealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1324323 (https://phabricator.wikimedia.org/T427822) (owner: 10Hashar) [12:09:16] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform, 13Patch-For-Review: Investigate multiple npm cache folders for mwext-codehealth-master-non-voting - https://phabricator.wikimedia.org/T427822#12207380 (10hashar) a:05hashar→03Peter Assigning to @Peter who found the root cause. I have... [12:15:44] (03PS1) 10Blake: helm-linter: Update envoy-future to 1.39.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1324699 (https://phabricator.wikimedia.org/T421418) [12:30:59] !log integration: on Castor: `sudo rm -fR /srv/castor/*/*/mwext-phpunit-coverage*/npm` # T427922 [12:31:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:31:02] T427922: Speedup the mwext-phpunit-coverage-publish castor job - https://phabricator.wikimedia.org/T427922 [12:32:23] (03CR) 10Hashar: [C:03+2] "I have nuked the caches." [integration/config] - 10https://gerrit.wikimedia.org/r/1321626 (https://phabricator.wikimedia.org/T427922) (owner: 10Mhurd) [12:34:11] (03Merged) 10jenkins-bot: jjb: Skip npm install for mwext PHPUnit coverage jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1321626 (https://phabricator.wikimedia.org/T427922) (owner: 10Mhurd) [12:54:30] (03CR) 10JMeybohm: [C:03+1] helm-linter: Update envoy-future to 1.39.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1324699 (https://phabricator.wikimedia.org/T421418) (owner: 10Blake) [13:20:02] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12207766 (10hashar) I went to query all 25 instances for the rep... [13:21:23] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Infra: Make sure our cluster VMs are running on different hypervisor hosts (+piggybacked machine upgrades) - https://phabricator.wikimedia.org/T426874#12207771 (10jnuche) 05Open→03Resolved Changes are in production [13:24:11] (03PS1) 10Jsn.sherman: Zuul: [mediawiki/extensions/GrowthExperiments] Add PersonalDashboard Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1324721 (https://phabricator.wikimedia.org/T432039) [13:25:47] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/GrowthExperiments] Add PersonalDashboard Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1324721 (https://phabricator.wikimedia.org/T432039) (owner: 10Jsn.sherman) [13:41:41] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12207887 (10hashar) Looking again at https://integration.wikimed... [13:47:57] (03CR) 10Urbanecm: "question: should this also be a regular non-phan dependency? without that, the tests that exercise PersonalDashboard wouldn't run." [integration/config] - 10https://gerrit.wikimedia.org/r/1324721 (https://phabricator.wikimedia.org/T432039) (owner: 10Jsn.sherman) [14:05:07] (03PS1) 10Blake: jjb: update job to releng/helm-linter:0.8.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1324730 (https://phabricator.wikimedia.org/T421418) [14:07:20] (03CR) 10CI reject: [V:04-1] jjb: update job to releng/helm-linter:0.8.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1324730 (https://phabricator.wikimedia.org/T421418) (owner: 10Blake) [14:10:32] (03PS2) 10Jsn.sherman: Zuul: [mediawiki/extensions/GrowthExperiments] Add PersonalDashboard dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1324721 (https://phabricator.wikimedia.org/T432039) [14:11:33] (03CR) 10Jsn.sherman: "🤦 yep! thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/1324721 (https://phabricator.wikimedia.org/T432039) (owner: 10Jsn.sherman) [14:42:01] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12208164 (10hashar) The list of CPU we for later reference: | C... [14:46:24] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06Collaboration-Services, 10Data-Services, 06tools-infrastructure-team: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12208182 (10dcaro) [14:51:53] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12208204 (10taavi) Does not seem like an infrastructure issue to me, thus untagging. [15:02:25] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎), 10Striker, 06tools-platform-team: Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12208281 (10Aklapper) [15:02:55] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎), 10Striker, 06tools-platform-team: Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12208284 (10Aklapper) [Hopefully] last patch merged; to be v... [15:07:06] 10Continuous-Integration-Config, 10Diffusion, 10Phabricator: integration-agent-docker machines excessively pull some Wikibase related Git repos in Diffusion - https://phabricator.wikimedia.org/T349921#12208301 (10Aklapper) ` mysql:phstats@m3-slave.eqiad.wmnet [phabricator_maniphest]> SELECT COUNT(r.name) AS... [16:27:20] (03open) 10brennen: Remove train log triage meeting from schedule [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/256 [16:34:24] (03merge) 10jforrester: deployment-calendar: Make each backport window addressable per day, and drop Lucas from the Wednesday one [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/255 [16:37:03] (03open) 10jforrester: build: Upgrade mediawiki/mediawiki-codesniffer to 52.0.0 [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/257 [16:38:12] (03merge) 10jforrester: build: Upgrade mediawiki/mediawiki-codesniffer to 52.0.0 [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/257 [16:38:46] (03update) 10jforrester: Remove train log triage meeting from schedule [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/256 (owner: 10brennen) [16:39:54] (03merge) 10jforrester: Remove train log triage meeting from schedule [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/256 (owner: 10brennen) [17:09:25] Thanks James_F! [17:09:35] Good job hosting the meeting too. [17:09:52] Always. [17:09:56] Aww, thanks. [17:52:51] +1 [18:55:33] 10Phabricator (Upstream), 07Upstream: Phabricator TOTP token should be evaluated against the currently valid one instead of expiring - https://phabricator.wikimedia.org/T434520#12209296 (10Aklapper) @Michael: Hi, I don't fully understand the steps above, I'm afraid: * I am logged out in Phab * I am logged in... [19:02:05] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834#12209333 (10Ladsgroup) [19:49:02] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834#12209532 (10brennen) [19:52:58] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform: Investigate multiple npm cache folders for mwext-codehealth-master-non-voting - https://phabricator.wikimedia.org/T427822#12209548 (10hashar) 05Open→03Resolved The `npm` cache directory has not reappeared which solves this issue. ` d... [20:34:35] 06Release-Engineering-Team: Where is my change running tool - https://phabricator.wikimedia.org/T434726 (10dancy) 03NEW [20:43:52] dancy: That just sounds like per-group release branches with extra steps. :-0 [20:52:39] 10Phabricator: Remove diffusion.ssh-user setting - https://phabricator.wikimedia.org/T434728 (10Aklapper) 03NEW p:05Triage→03Low [20:52:54] (03open) 10aklapper: Remove unneeded "diffusion.ssh-user" setting [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/116 (https://phabricator.wikimedia.org/T434728) [20:53:21] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Remove diffusion.ssh-user setting - https://phabricator.wikimedia.org/T434728#12209858 (10Aklapper) a:03Aklapper [22:04:20] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for gaiarajan - https://phabricator.wikimedia.org/T434626#12210074 (10gaiarajan) Hoping to work on this ticket: https://phabricator.wikimedia.org/T376761. Thank you so much, @brennen! [22:47:52] brennen: re: https://phabricator.wikimedia.org/T434712 we could revert the patch that likely introduced the error (see aaron's comments on the task), I guess at this hour wmde folks are probably sound asleep... thoughts? [22:52:29] apergos: it's nearing 5pm local and i usually err on the side of not doing train ops this late in the day [22:52:52] so on hold until tomorrow? [22:52:52] that said, i think if there's not a response from relevant folks by my morning that sounds like the best way forward [22:52:56] yeah [22:53:04] ok passing it on to my team and thanks [22:53:15] i'll update the task accordingly for visibility, and thanks for checking in [22:54:54] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834#12210186 (10brennen) It's coming up on 17:00 local. Considering train parked overnight, pending a resolution on T434712. If no res...