[04:26:55] (03PS1) 10Gergő Tisza: Add Winston Sung (Fandom) to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/742278 [04:34:17] (03CR) 10Winston Sung: "It should be Fandom ZH Team instead of Fandom (see difference: https://github.com/Fandom-zh vs. https://github.com/Wikia ). Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/742278 (owner: 10Gergő Tisza) [05:31:22] 10Project-Admins: Create project tag for Schematree_recommender - https://phabricator.wikimedia.org/T296599 (10Aklapper) @Michaelcochez: Thanks! Where exactly would the codebase be located (URL), and service hosted (if the latter makes sense and if this is separate)? I assume other Wikidata folks are aware and a... [07:32:24] 10Project-Admins, 10Infrastructure-Foundations, 10PM, 10Puppet: Clarify Puppet tag - https://phabricator.wikimedia.org/T295221 (10Aklapper) @joanna_borun: Could you please answer? Thanks in advance! :) [07:51:16] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T293952 (10kostajh) [07:52:04] 10Release-Engineering-Team, 10Growth-Team, 10GrowthExperiments, 10Browser-Tests, and 4 others: ERROR @wdio/sync: Error: socket hang up - https://phabricator.wikimedia.org/T296508 (10kostajh) 05Open→03Resolved a:03kostajh >>! In T296508#7531440, @Pchelolo wrote: > Yup, reverting the patch mentioned by... [09:00:54] 10Release-Engineering-Team, 10SRE, 10cloud-services-team (Kanban): Consider removing labnet-users group - https://phabricator.wikimedia.org/T296574 (10MoritzMuehlenhoff) For context/clarification: This group currently permits the unprivileged login to cloudnet* servers. Tagging the current members of the lab... [09:32:50] (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 2" [integration/config] - 10https://gerrit.wikimedia.org/r/742238 (owner: 10Majavah) [09:34:36] (03Merged) 10jenkins-bot: Configure CI for cloud/toolforge/delete-crashing-pods [integration/config] - 10https://gerrit.wikimedia.org/r/742238 (owner: 10Majavah) [10:10:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10SRE, 10cloud-services-team (Kanban): Consider removing labnet-users group - https://phabricator.wikimedia.org/T296574 (10hashar) The purpose was for Releng to monitor the status of OpenStack when we used #nodepool (a system that dynamic... [10:23:39] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T293952 (10Sgs) [13:20:29] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) 05Open→03In progress p:05Triage→03High [13:46:04] (03CR) 10Anne Tomasevich: doc: Add Codex, wikimedia-ui-base, less.php (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/740937 (owner: 10Krinkle) [14:00:40] (03CR) 10Jforrester: [C: 04-1] "Please do not use inappropriate terms, and do no make two commits in one; re-sorting this list should be done in a stand-alone patch, if e" [integration/config] - 10https://gerrit.wikimedia.org/r/741985 (owner: 104nn1l2) [14:31:21] (03PS4) 10Jforrester: Zuul: Add 4nn1l2 to the CI trusted users list [integration/config] - 10https://gerrit.wikimedia.org/r/741985 (owner: 104nn1l2) [14:36:53] (03CR) 10Jforrester: [C: 03+2] Zuul: Add 4nn1l2 to the CI trusted users list [integration/config] - 10https://gerrit.wikimedia.org/r/741985 (owner: 104nn1l2) [14:39:27] (03Merged) 10jenkins-bot: Zuul: Add 4nn1l2 to the CI trusted users list [integration/config] - 10https://gerrit.wikimedia.org/r/741985 (owner: 104nn1l2) [14:39:43] !log Zuul: Add 4nn1l2 to the CI trusted users list [14:39:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:40:15] (03PS1) 10Jforrester: Zuul: Alpha-sort allowlist sections [integration/config] - 10https://gerrit.wikimedia.org/r/742474 [14:40:17] (03PS1) 10Jforrester: Zuul: Merge GCI allowlist into main as GCI is over forever [integration/config] - 10https://gerrit.wikimedia.org/r/742475 [14:40:51] (03CR) 10Jforrester: [C: 03+2] Zuul: Alpha-sort allowlist sections [integration/config] - 10https://gerrit.wikimedia.org/r/742474 (owner: 10Jforrester) [14:40:56] (03CR) 10Jforrester: [C: 03+2] Zuul: Merge GCI allowlist into main as GCI is over forever [integration/config] - 10https://gerrit.wikimedia.org/r/742475 (owner: 10Jforrester) [14:43:51] (03Merged) 10jenkins-bot: Zuul: Alpha-sort allowlist sections [integration/config] - 10https://gerrit.wikimedia.org/r/742474 (owner: 10Jforrester) [14:43:54] (03Merged) 10jenkins-bot: Zuul: Merge GCI allowlist into main as GCI is over forever [integration/config] - 10https://gerrit.wikimedia.org/r/742475 (owner: 10Jforrester) [15:15:30] 10Beta-Cluster-Infrastructure, 10Maps, 10Product-Infrastructure-Team-Backlog: Puppet config is broken for the maps instance on deployment-prep - https://phabricator.wikimedia.org/T291624 (10Majavah) Does anyone regularly use this instance for something? In other words, what would happen if we got rid of the... [15:54:40] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10MediaWiki-extensions-FlaggedRevs: addWiki.php failed in Beta Cluster, trying to run Flagged Revs code where none was enabled - https://phabricator.wikimedia.org/T296644 (10Jdforrester-WMF) [16:21:17] Project beta-update-databases-eqiad build #54936: 04FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54936/ [16:42:53] (03PS1) 10Accraze: inference-services: add outlink pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/742489 (https://phabricator.wikimedia.org/T290930) [17:21:14] Project beta-update-databases-eqiad build #54937: 04STILL FAILING in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54937/ [17:53:41] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10serviceops-radar, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10daniel) >>! In T296331#7525529, @Pchelolo wrote: > We are not strictly blocked by this yet, but we wi... [18:02:56] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T293952 (10ssastry) **{ DRAFT. To be filled out with a bit more info ]** ##### Risky Patch! 🚂🔥 * **Change**: Rollout of patches to implement {T261181} * **Summary**: Pa... [18:08:48] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10serviceops-radar, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10Legoktm) So it sounds like it is a blocking issue :) I'll do the packaging now. [18:09:52] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T293952 (10ssastry) [18:21:32] Project beta-update-databases-eqiad build #54938: 04STILL FAILING in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54938/ [18:59:50] !log gitlab-runners: added j.elto as an admin [19:00:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:43:34] (03CR) 10Jeena Huneidi: [C: 03+2] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/742489 (https://phabricator.wikimedia.org/T290930) (owner: 10Accraze) [19:45:48] (03Merged) 10jenkins-bot: inference-services: add outlink pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/742489 (https://phabricator.wikimedia.org/T290930) (owner: 10Accraze) [19:47:11] (03PS2) 10Jforrester: dockerfiles: Install php-yaml everywhere [integration/config] - 10https://gerrit.wikimedia.org/r/740928 (https://phabricator.wikimedia.org/T296331) (owner: 10Dduvall) [19:47:12] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/742489 [19:47:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:51:36] 10Release-Engineering-Team (Doing), 10MediaWiki-Core-Tests, 10MediaWiki-ResourceLoader, 10Patch-For-Review, 10Performance-Team (Radar): Move bundlesize test from npm script to MediaWikiIntegrationTest - https://phabricator.wikimedia.org/T255149 (10Krinkle) [19:58:00] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10serviceops-radar, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10dduvall) >>! In T296331#7534915, @Legoktm wrote: > So it sounds like it is a blocking issue :) I'll d... [20:21:18] Project beta-update-databases-eqiad build #54939: 04STILL FAILING in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54939/ [20:24:39] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/742525 should fix this ^ [20:35:01] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T293952 (10ssastry) [20:59:58] !log Switching Jenkins CI config for Beta Cluster host from deployment-deploy01 to deployment-deploy03 for T278689 [21:00:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:00:01] T278689: Upgrade deployment-prep deploy* hosts to Buster - https://phabricator.wikimedia.org/T278689 [21:01:18] (03PS3) 10Jforrester: dockerfiles: Install php-yaml everywhere [integration/config] - 10https://gerrit.wikimedia.org/r/740928 (https://phabricator.wikimedia.org/T296331) (owner: 10Dduvall) [21:01:20] (03PS1) 10Jforrester: [WIP] dockerfiles: Provide php81 [integration/config] - 10https://gerrit.wikimedia.org/r/742537 (https://phabricator.wikimedia.org/T296489) [21:04:22] (03PS1) 10Jforrester: jjb: Update Beta Cluster docs for new deploy server [integration/config] - 10https://gerrit.wikimedia.org/r/742539 (https://phabricator.wikimedia.org/T278689) [21:05:00] Project beta-scap-sync-world build #28963: 04FAILURE in 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/28963/ [21:05:58] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/742539 (https://phabricator.wikimedia.org/T278689) (owner: 10Jforrester) [21:08:09] (03Merged) 10jenkins-bot: jjb: Update Beta Cluster docs for new deploy server [integration/config] - 10https://gerrit.wikimedia.org/r/742539 (https://phabricator.wikimedia.org/T278689) (owner: 10Jforrester) [21:09:33] (03PS1) 10Majavah: Update deployment-prep deploy host [tools/scap] - 10https://gerrit.wikimedia.org/r/742540 (https://phabricator.wikimedia.org/T278689) [21:09:54] Project beta-scap-sync-world build #28964: 04STILL FAILING in 2 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/28964/ [21:10:10] fixing [21:12:39] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team: Create a Beta Cluster version of Wikifunctions.org - https://phabricator.wikimedia.org/T284162 (10Jdforrester-WMF) [21:12:41] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator: Work out how we're going to have "production-like" versions of the wikifunctions evaluator and orchestrator services in Beta Cluster - https://phabricator.wikimedia.org/T289315 (10Jdforrester-WMF) 05Op... [21:13:35] majavah: Anything I can do to help? [21:14:05] can you +2 https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/742540/ so I get to fix the special scap apt repo? [21:14:22] Sure, but that's a doc-only change. [21:14:30] (03PS2) 10Jforrester: docs: Update deployment-prep deploy host [tools/scap] - 10https://gerrit.wikimedia.org/r/742540 (https://phabricator.wikimedia.org/T278689) (owner: 10Majavah) [21:14:34] (03CR) 10Jforrester: [C: 03+2] docs: Update deployment-prep deploy host [tools/scap] - 10https://gerrit.wikimedia.org/r/742540 (https://phabricator.wikimedia.org/T278689) (owner: 10Majavah) [21:14:52] Project beta-scap-sync-world build #28965: 04STILL FAILING in 2 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/28965/ [21:14:57] keyholder is failing with "sign_and_send_pubkey: signing failed: agent refused operation" [21:15:05] Hmm. Is it armed? [21:15:16] yes, it'd have a different error otherwise (and I just armed it) [21:15:20] Right. [21:15:48] I restarted the proxy, that has helped previously [21:16:44] (03Merged) 10jenkins-bot: docs: Update deployment-prep deploy host [tools/scap] - 10https://gerrit.wikimedia.org/r/742540 (https://phabricator.wikimedia.org/T278689) (owner: 10Majavah) [21:16:50] https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/lastSuccessfulBuild/console passed. [21:18:49] https://integration.wikimedia.org/ci/job/beta-publish-deb/656/console :/ [21:19:12] "stretch-deployment-prep". [21:19:22] Should that be called something different now it's not on stretch? [21:19:26] Yippee, build fixed! [21:19:27] Project beta-scap-sync-world build #28966: 09FIXED in 3 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/28966/ [21:19:38] At least there's that. [21:22:11] Project beta-update-databases-eqiad build #54940: 04STILL FAILING in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54940/ [21:23:16] !log taavi@deployment-deploy03:~$ sudo aptly publish --architectures="all,amd64" --skip-signing --distribution "buster-deployment-prep" repo buster-deployment-prep && sudo aptly publish --skip-signing update buster-deployment-prep [21:23:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:23:44] deb publishing should be fixed now [21:24:17] Is that something that should be puppetized? [21:26:52] probably, role::aptly::server only includes a repo for stretch and I probably created the buster one manually [21:27:59] althought that's currently a style guide violation in its own :/ [22:21:59] Project beta-update-databases-eqiad build #54941: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54941/ [22:59:39] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T293952 (10thcipriani) 05Open→03Declined [23:02:06] thcipriani, ^ is that intentional? [23:03:57] subbu: yeah, we don't have the capacity for this week, unfortunately. I know it's short notice, but I'll be sending a bigger explanation shortly. [23:04:15] I see. okay. [23:23:48] Yippee, build fixed! [23:23:48] Project beta-update-databases-eqiad build #54942: 09FIXED in 3 min 48 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54942/