[00:37:18] (03PS3) 10Jeena Huneidi: docker: don't use docker.io package [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765350 [03:44:16] (03CR) 10Thcipriani: [C: 03+2] sync-wikiversions: Remove redundant tasks.compile_wikiversions call [tools/scap] - 10https://gerrit.wikimedia.org/r/765345 (owner: 10Ahmon Dancy) [03:44:59] (03Merged) 10jenkins-bot: sync-wikiversions: Remove redundant tasks.compile_wikiversions call [tools/scap] - 10https://gerrit.wikimedia.org/r/765345 (owner: 10Ahmon Dancy) [05:14:10] 10Release-Engineering-Team (Next), 10tech-decision-forum, 10Code-Stewardship-Reviews, 10Documentation: Document checklist steps to undeploy / sunset a codebase on WMF servers (not: archiving) - https://phabricator.wikimedia.org/T294329 (10Aklapper) Boldly adding #tech-decision-forum in lack of better ideas... [06:57:30] 10Release-Engineering-Team (Done by Feb 23πŸ”₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T300199 (10AndyRussG) Hi! Not sure if this UBN bug is a train blocker, but the fix does need to be deployed to Meta (group 1) right away: {... [07:01:11] 10Release-Engineering-Team (Done by Feb 23πŸ”₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T300199 (10RhinosF1) If it only affects one wiki, there's no point blocking the train but we should see asap about getting it deployed. [07:05:51] Amir1, apergos: hi, do we want to wait until the window for ^ or can I steal you early? [07:10:36] I'd prefer to wait the 50 minutes, as I'd like to not be the sole deployer for things [07:11:06] apergos: okay, I will add to calendar [07:11:20] I've got a test for it ready and spoke to Andy [07:11:28] great [07:12:21] given this is so far the only patch, it will also be the one used as the example for the two trainees who are due to show up [07:15:20] no problem [07:53:30] (03CR) 10Hashar: [C: 03+2] Add new "Tuleap*" skin and extensions to CI [integration/config] - 10https://gerrit.wikimedia.org/r/765192 (owner: 10Robert Vogel) [07:55:33] (03Merged) 10jenkins-bot: Add new "Tuleap*" skin and extensions to CI [integration/config] - 10https://gerrit.wikimedia.org/r/765192 (owner: 10Robert Vogel) [08:05:20] (03CR) 10Hashar: "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/765192 (owner: 10Robert Vogel) [08:26:48] hey hashar, see operations, kind of urgent [08:29:49] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-eqiad, 10serviceops: Q3:(Need By: TBD) rack/setup/install gitlab100[2|3] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10LSobanski) a:05LSobanskiβ†’03RobH Fine by me. [08:35:36] 10Continuous-Integration-Infrastructure, 10Release Pipeline: Pipeline lib leaves workspace being on the Jenkins agents - https://phabricator.wikimedia.org/T302477 (10hashar) [08:37:22] (03CR) 10Hashar: [C: 03+2] dockerfiles: fix shellcheck in mediawiki-phan-testrun [integration/config] - 10https://gerrit.wikimedia.org/r/764809 (owner: 10Hashar) [08:39:07] (03Merged) 10jenkins-bot: dockerfiles: fix shellcheck in mediawiki-phan-testrun [integration/config] - 10https://gerrit.wikimedia.org/r/764809 (owner: 10Hashar) [08:39:24] (03PS1) 10Hashar: jjb: update mw-tools-phan-mwcore-testrun image [integration/config] - 10https://gerrit.wikimedia.org/r/765472 [08:48:12] (03PS1) 10Jaime Nuche: sync-canaries: remove command [tools/scap] - 10https://gerrit.wikimedia.org/r/765473 (https://phabricator.wikimedia.org/T301717) [08:48:20] (03CR) 10jerkins-bot: [V: 04-1] sync-canaries: remove command [tools/scap] - 10https://gerrit.wikimedia.org/r/765473 (https://phabricator.wikimedia.org/T301717) (owner: 10Jaime Nuche) [08:58:04] !log Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1 [08:58:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:58:19] (03CR) 10Hashar: [C: 03+2] jjb: update mw-tools-phan-mwcore-testrun image [integration/config] - 10https://gerrit.wikimedia.org/r/765472 (owner: 10Hashar) [09:00:21] (03Merged) 10jenkins-bot: jjb: update mw-tools-phan-mwcore-testrun image [integration/config] - 10https://gerrit.wikimedia.org/r/765472 (owner: 10Hashar) [09:15:29] (03PS2) 10Jaime Nuche: sync-canaries: remove command [tools/scap] - 10https://gerrit.wikimedia.org/r/765473 (https://phabricator.wikimedia.org/T301717) [09:16:53] (03PS3) 10Jaime Nuche: sync-canaries: remove command [tools/scap] - 10https://gerrit.wikimedia.org/r/765473 (https://phabricator.wikimedia.org/T301717) [09:28:19] (03CR) 10Hashar: [C: 03+2] dockerfiles: drop verbatim \n in echo commands [integration/config] - 10https://gerrit.wikimedia.org/r/764811 (owner: 10Hashar) [09:30:10] (03Merged) 10jenkins-bot: dockerfiles: drop verbatim \n in echo commands [integration/config] - 10https://gerrit.wikimedia.org/r/764811 (owner: 10Hashar) [10:38:53] (03PS1) 10Hashar: jjb: update image for phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/765481 [10:39:07] (03CR) 10Hashar: [C: 03+2] jjb: update image for phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/765481 (owner: 10Hashar) [10:40:52] (03Merged) 10jenkins-bot: jjb: update image for phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/765481 (owner: 10Hashar) [10:42:15] (03PS1) 10Hashar: jjb update image for node jobs [integration/config] - 10https://gerrit.wikimedia.org/r/765482 [10:43:09] (03PS2) 10Hashar: jjb update image for node jobs [integration/config] - 10https://gerrit.wikimedia.org/r/765482 [10:43:32] (03CR) 10Hashar: [C: 03+2] jjb update image for node jobs [integration/config] - 10https://gerrit.wikimedia.org/r/765482 (owner: 10Hashar) [10:44:34] (03CR) 10Hashar: [C: 03+2] dockerfiles: pass shellcheck on operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/764814 (owner: 10Hashar) [10:45:18] (03Merged) 10jenkins-bot: jjb update image for node jobs [integration/config] - 10https://gerrit.wikimedia.org/r/765482 (owner: 10Hashar) [10:46:17] (03Merged) 10jenkins-bot: dockerfiles: pass shellcheck on operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/764814 (owner: 10Hashar) [10:55:24] (03PS2) 10Hashar: dockerfiles: php-compile fix ambiguous env assignment [integration/config] - 10https://gerrit.wikimedia.org/r/764820 [10:55:26] (03PS2) 10Hashar: dockerfiles: simplify sonar-project.properties generation [integration/config] - 10https://gerrit.wikimedia.org/r/764818 [10:55:28] (03PS2) 10Hashar: dockerfiles: shellcheck: separated glob from grep options [integration/config] - 10https://gerrit.wikimedia.org/r/764819 [10:55:30] (03PS2) 10Hashar: Fix shellcheck for castor snippets [integration/config] - 10https://gerrit.wikimedia.org/r/764817 [10:55:32] (03PS2) 10Hashar: dockerfiles: double quotes in bash scripts [integration/config] - 10https://gerrit.wikimedia.org/r/764815 [10:55:34] (03PS7) 10Hashar: Run shellcheck against shell files [integration/config] - 10https://gerrit.wikimedia.org/r/756088 [11:04:52] !log Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6 [11:04:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:06:37] (03PS1) 10Hashar: jjb: update operations/puppet job to latest image [integration/config] - 10https://gerrit.wikimedia.org/r/765487 [11:07:19] (03CR) 10Hashar: [C: 03+2] jjb: update operations/puppet job to latest image [integration/config] - 10https://gerrit.wikimedia.org/r/765487 (owner: 10Hashar) [11:09:04] (03Merged) 10jenkins-bot: jjb: update operations/puppet job to latest image [integration/config] - 10https://gerrit.wikimedia.org/r/765487 (owner: 10Hashar) [11:16:44] (03CR) 10Hashar: [C: 03+2] dockerfiles: php-compile fix ambiguous env assignment [integration/config] - 10https://gerrit.wikimedia.org/r/764820 (owner: 10Hashar) [11:18:36] (03Merged) 10jenkins-bot: dockerfiles: php-compile fix ambiguous env assignment [integration/config] - 10https://gerrit.wikimedia.org/r/764820 (owner: 10Hashar) [11:24:06] (03PS1) 10Hashar: jjb: update image for php compile jobs [integration/config] - 10https://gerrit.wikimedia.org/r/765492 [11:25:47] 10Release-Engineering-Team (Done by Feb 23πŸ”₯), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10Joe) >>! In T297673#7733295, @dancy wrote: > @joe I tried running this today but it failed: > ` > dancy... [11:30:57] (03CR) 10Hashar: "* Built image docker-registry.discovery.wmnet/releng/php81-compile:0.0.2" [integration/config] - 10https://gerrit.wikimedia.org/r/764820 (owner: 10Hashar) [11:31:12] (03CR) 10Hashar: [C: 03+2] jjb: update image for php compile jobs [integration/config] - 10https://gerrit.wikimedia.org/r/765492 (owner: 10Hashar) [11:32:07] 10Phabricator: Spaces request for Technical-Program-Management - https://phabricator.wikimedia.org/T277107 (10Aklapper) Again, it has been confusing in public (!) `T302441` that the `Technical-Program-Management` project tag is only shown as `Unknown Object (Project)` under "Tags" for nearly everybody. I don't s... [11:33:07] (03Merged) 10jenkins-bot: jjb: update image for php compile jobs [integration/config] - 10https://gerrit.wikimedia.org/r/765492 (owner: 10Hashar) [11:34:28] (03CR) 10Hashar: [C: 03+2] dockerfiles: simplify sonar-project.properties generation [integration/config] - 10https://gerrit.wikimedia.org/r/764818 (owner: 10Hashar) [11:35:14] (03PS3) 10Hashar: dockerfiles: simplify sonar-project.properties generation [integration/config] - 10https://gerrit.wikimedia.org/r/764818 [11:35:27] (03CR) 10Hashar: [C: 03+2] dockerfiles: simplify sonar-project.properties generation [integration/config] - 10https://gerrit.wikimedia.org/r/764818 (owner: 10Hashar) [11:37:53] (03Merged) 10jenkins-bot: dockerfiles: simplify sonar-project.properties generation [integration/config] - 10https://gerrit.wikimedia.org/r/764818 (owner: 10Hashar) [11:40:38] (03PS1) 10Hashar: jjb: update Sonar scanner jobs to latest image [integration/config] - 10https://gerrit.wikimedia.org/r/765496 [11:41:25] :log Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2 [11:41:30] !log Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2 [11:41:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:41:42] taavi: deployed that change and it has resolved the issue, thank you again <3 [11:42:01] (03CR) 10Hashar: [C: 03+2] jjb: update Sonar scanner jobs to latest image [integration/config] - 10https://gerrit.wikimedia.org/r/765496 (owner: 10Hashar) [11:42:35] (03PS3) 10Hashar: dockerfiles: shellcheck: separated glob from grep options [integration/config] - 10https://gerrit.wikimedia.org/r/764819 [11:43:11] (03CR) 10Hashar: [C: 03+2] dockerfiles: shellcheck: separated glob from grep options [integration/config] - 10https://gerrit.wikimedia.org/r/764819 (owner: 10Hashar) [11:44:04] (03PS3) 10Hashar: Fix shellcheck for castor snippets [integration/config] - 10https://gerrit.wikimedia.org/r/764817 [11:44:06] (03Merged) 10jenkins-bot: jjb: update Sonar scanner jobs to latest image [integration/config] - 10https://gerrit.wikimedia.org/r/765496 (owner: 10Hashar) [11:44:41] 10Release-Engineering-Team (Done by Feb 23πŸ”₯), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10Joe) aaand should be fixed now. [11:44:59] (03Merged) 10jenkins-bot: dockerfiles: shellcheck: separated glob from grep options [integration/config] - 10https://gerrit.wikimedia.org/r/764819 (owner: 10Hashar) [11:54:15] !log Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1 [11:54:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:55:19] (03PS1) 10Hashar: jjb: update image of shellcheck job [integration/config] - 10https://gerrit.wikimedia.org/r/765500 [11:55:45] (03CR) 10Hashar: [C: 03+2] jjb: update image of shellcheck job [integration/config] - 10https://gerrit.wikimedia.org/r/765500 (owner: 10Hashar) [11:57:34] (03Merged) 10jenkins-bot: jjb: update image of shellcheck job [integration/config] - 10https://gerrit.wikimedia.org/r/765500 (owner: 10Hashar) [11:59:10] 10Release-Engineering-Team, 10Scap: Preserve supplied command-line config in local `scap` subprocess calls - https://phabricator.wikimedia.org/T302487 (10jnuche) [12:04:36] 10Release-Engineering-Team, 10Scap: Preserve supplied command-line config in local Scap subprocess calls - https://phabricator.wikimedia.org/T302487 (10jnuche) [12:21:36] 10Release-Engineering-Team, 10Scap: Integrate deploy-promote and stage-train scripts into Scap - https://phabricator.wikimedia.org/T302488 (10jnuche) [12:22:10] 10Release-Engineering-Team (Next), 10Scap: Integrate deploy-promote and stage-train scripts into Scap - https://phabricator.wikimedia.org/T302488 (10jnuche) [12:27:13] 10Release-Engineering-Team (Next), 10Scap: Integrate deploy-promote and stage-train scripts into Scap - https://phabricator.wikimedia.org/T302488 (10jnuche) a:03jnuche [12:31:45] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23πŸ”₯): Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10valerio.bozzolan) They ask me if there is anything Wikimedia IT can do to make this possible. Thank you so much! [12:38:58] (03CR) 10Hashar: [C: 03+2] Fix shellcheck for castor snippets [integration/config] - 10https://gerrit.wikimedia.org/r/764817 (owner: 10Hashar) [12:40:42] (03Merged) 10jenkins-bot: Fix shellcheck for castor snippets [integration/config] - 10https://gerrit.wikimedia.org/r/764817 (owner: 10Hashar) [13:10:15] !log Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817 [13:10:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:13:39] !log Built image docker-registry.discovery.wmnet/releng/castor:0.2.5 [13:13:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:14:21] (03PS3) 10Hashar: dockerfiles: double quotes in bash scripts [integration/config] - 10https://gerrit.wikimedia.org/r/764815 [13:15:00] (03PS8) 10Hashar: Run shellcheck against shell files [integration/config] - 10https://gerrit.wikimedia.org/r/756088 [13:16:55] (03CR) 10Hashar: dockerfiles: double quotes in bash scripts (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/764815 (owner: 10Hashar) [13:17:49] (03PS4) 10Hashar: dockerfiles: double quotes in bash scripts [integration/config] - 10https://gerrit.wikimedia.org/r/764815 [13:18:06] (03CR) 10Hashar: [C: 03+2] dockerfiles: double quotes in bash scripts [integration/config] - 10https://gerrit.wikimedia.org/r/764815 (owner: 10Hashar) [13:19:57] (03Merged) 10jenkins-bot: dockerfiles: double quotes in bash scripts [integration/config] - 10https://gerrit.wikimedia.org/r/764815 (owner: 10Hashar) [13:20:13] (03PS9) 10Hashar: Run shellcheck against shell files [integration/config] - 10https://gerrit.wikimedia.org/r/756088 [13:44:29] (03CR) 10Hashar: [C: 03+2] "Finally it is passing shellcheck!!!!" [integration/config] - 10https://gerrit.wikimedia.org/r/756088 (owner: 10Hashar) [13:44:57] !log integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088 [13:45:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:46:19] (03Merged) 10jenkins-bot: Run shellcheck against shell files [integration/config] - 10https://gerrit.wikimedia.org/r/756088 (owner: 10Hashar) [13:56:01] 10Release-Engineering-Team, 10Scap: Preserve supplied command-line config in local Scap subprocess calls - https://phabricator.wikimedia.org/T302487 (10jnuche) 05Openβ†’03In progress a:03jnuche [14:19:31] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Discovery, 10Discovery-Search (Current work): Deploy new bullseye elastic cluster nodes on deployment-prep - https://phabricator.wikimedia.org/T299797 (10Jdforrester-WMF) [14:19:33] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Jdforrester-WMF) [14:19:52] 10Beta-Cluster-Infrastructure, 10Elasticsearch, 10Discovery-Search (Current work): Upgrade deployment-prep Elastic cluster to Debian Buster or newer - https://phabricator.wikimedia.org/T298252 (10Jdforrester-WMF) [14:19:54] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Discovery, 10Discovery-Search (Current work): Deploy new bullseye elastic cluster nodes on deployment-prep - https://phabricator.wikimedia.org/T299797 (10Jdforrester-WMF) [14:20:12] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Jdforrester-WMF) [14:20:14] 10Beta-Cluster-Infrastructure, 10Elasticsearch, 10Discovery-Search (Current work): Upgrade deployment-prep Elastic cluster to Debian Buster or newer - https://phabricator.wikimedia.org/T298252 (10Jdforrester-WMF) [15:26:43] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mfossati - https://phabricator.wikimedia.org/T302204 (10ArielGlenn) Followup: Training happened, got the full experience: CI broken, first patch failed and revert necessary. Lots of learning opportunities :-) [15:26:47] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for Hannah Okwelum - https://phabricator.wikimedia.org/T302275 (10ArielGlenn) Followup: Training happened, got the full experience: CI broken, first patch failed and revert necessary. Lots of learning opportunities :-) [15:30:10] (03CR) 10Ahmon Dancy: sync-canaries: remove command (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/765473 (https://phabricator.wikimedia.org/T301717) (owner: 10Jaime Nuche) [15:31:43] 10Release-Engineering-Team (Done by Feb 23πŸ”₯), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) >>! In T297673#7734748, @Joe wrote: > aaand should be fixed now. Confirmed. Thanks @Joe! [15:35:28] 10Release-Engineering-Team (Done by Feb 23πŸ”₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T300199 (10AndyRussG) >>! In T300199#7734277, @RhinosF1 wrote: > If it only affects one wiki, there's no point blocking the train but we sh... [15:37:30] (03CR) 10Ahmon Dancy: "Testing..." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765350 (owner: 10Jeena Huneidi) [15:45:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) 05Resolvedβ†’03Open And it's back today. Cripes. [15:52:29] (03PS4) 10Jaime Nuche: sync-canary: remove command [tools/scap] - 10https://gerrit.wikimedia.org/r/765473 (https://phabricator.wikimedia.org/T301717) [15:52:53] (03CR) 10Jaime Nuche: sync-canary: remove command (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/765473 (https://phabricator.wikimedia.org/T301717) (owner: 10Jaime Nuche) [15:53:23] (03CR) 10Ahmon Dancy: [C: 03+2] sync-canary: remove command [tools/scap] - 10https://gerrit.wikimedia.org/r/765473 (https://phabricator.wikimedia.org/T301717) (owner: 10Jaime Nuche) [15:53:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) Between the time I reopened this report and started debugging, the lintian packages landed on integration-agent-pkgbuilder-... [15:54:21] (03Merged) 10jenkins-bot: sync-canary: remove command [tools/scap] - 10https://gerrit.wikimedia.org/r/765473 (https://phabricator.wikimedia.org/T301717) (owner: 10Jaime Nuche) [15:55:31] 10Release-Engineering-Team (Doing), 10Scap, 10Patch-For-Review: Refactor scap sync-canary - https://phabricator.wikimedia.org/T301717 (10jnuche) 05Openβ†’03Resolved [15:55:42] (03CR) 10Ahmon Dancy: [C: 03+2] "Works." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765350 (owner: 10Jeena Huneidi) [15:56:12] (03Merged) 10jenkins-bot: docker: don't use docker.io package [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765350 (owner: 10Jeena Huneidi) [15:58:14] 10Release-Engineering-Team (Next), 10Scap: Integrate deploy-promote and stage-train scripts into Scap - https://phabricator.wikimedia.org/T302488 (10thcipriani) > Not used by anyone outside of RelEng (not 100% on this point, I did some asking around and that seems to be the case) I am 99.99% sure you're corre... [16:02:11] !log Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2 [16:02:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:50:01] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-eqiad, 10serviceops: Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10RobH) [16:50:36] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-eqiad, 10serviceops: Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10RobH) a:05RobHβ†’03Jclark-ctr >>! In T301177#7734357, @LSobanski wrote: > Fine by me. Thanks, updated th... [18:59:00] 10Phabricator, 10Developer-Advocacy (Jan-Mar 2022), 10Patch-For-Review: Followup fixes to weekly 2FA Phab cronjob - https://phabricator.wikimedia.org/T302385 (10Dzahn) @Aklapper https://www.youtube.com/watch?v=DGOgSPVBmIE [19:07:57] 10Phabricator: Spaces request for Technical-Program-Management - https://phabricator.wikimedia.org/T277107 (10MBinder_WMF) Hi, @Aklapper ! Thanks for flagging this again. I agree that the UI in these circumstances is unfortunate and confusing, and that public-by-default is generally valuable, values-friendly, a... [19:10:22] 10Release-Engineering-Team (Doing), 10Scap: Preserve supplied command-line config in local Scap subprocess calls - https://phabricator.wikimedia.org/T302487 (10thcipriani) [19:23:56] 10Phabricator: Spaces request for Technical-Program-Management - https://phabricator.wikimedia.org/T277107 (10MBinder_WMF) I also re-read your [[ https://phabricator.wikimedia.org/T277107#7734722 | comment ]] and realize that you already showed which task you screenshotted in {T302441}, so apologies for missing... [19:26:26] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mfossati - https://phabricator.wikimedia.org/T302204 (10thcipriani) 05Openβ†’03Resolved a:03ArielGlenn ## {icon check-circle-o} Training completed! Hey @mfossati thanks for attending deployment training I have a list... [19:26:37] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for Hannah Okwelum - https://phabricator.wikimedia.org/T302275 (10thcipriani) 05Openβ†’03Resolved a:03ArielGlenn ## {icon check-circle-o} Training completed! Hey @Hokwelum thanks for attending deployment training I have... [19:29:06] (03PS1) 10Stegmujo: Adding JS to Trusted users [integration/config] - 10https://gerrit.wikimedia.org/r/765590 (https://phabricator.wikimedia.org/T302523) [19:31:11] 10Phabricator: When a project is not public, it appears to users outside "Visible To" criteria as "Unknown Object", which is confusing - https://phabricator.wikimedia.org/T302525 (10MBinder_WMF) [19:32:13] 10Phabricator: Spaces request for Technical-Program-Management - https://phabricator.wikimedia.org/T277107 (10MBinder_WMF) I filed a task for improving the UX in the current setup (regardless of what we decide): {T302525} [19:35:02] 10Phabricator, 10Developer-Advocacy (Jan-Mar 2022): Followup fixes to weekly 2FA Phab cronjob - https://phabricator.wikimedia.org/T302385 (10Aklapper) 05Openβ†’03Resolved Thanks Dzahn! Looks great! <3 [20:34:55] 10Phabricator, 10Release-Engineering-Team: Consider migrating from Phabricator to Phorge once that new upstream is ready for such a migration - https://phabricator.wikimedia.org/T302528 (10Jdforrester-WMF) [20:45:20] 10Phabricator, 10Release-Engineering-Team, 10serviceops-radar: Consider migrating from Phabricator to Phorge once that new upstream is ready for such a migration - https://phabricator.wikimedia.org/T302528 (10Dzahn) [21:23:47] 10Phabricator (Upstream), 10Release-Engineering-Team, 10serviceops-radar: Consider migrating from Phabricator to Phorge once that new upstream is ready for such a migration - https://phabricator.wikimedia.org/T302528 (10Izno) [22:08:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) `name=grep lintian /var/log/dpkg.log 2022-02-24 19:48:26 status installed lintian:all 2.104.0~bpo10+1 2022-02-24 19:48:26 r... [22:10:52] dancy: as far as I can tell `lintian` is installed on integration-agent-pkgbuilder-1001.integration.eqiad1.wikimedia.cloud but not on 1002 - that might explain why it's sometimes present and sometimes is not [22:12:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) ` # puppet agent --test Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugi... [22:12:33] taavi: Have a look at my most recent message on the ticket. [22:13:08] looks like a conflict between `/Stage[main]/Package_builder/Package[lintian]/ensure` and `/Stage[main]/Package_builder/Package[pkg-js-tools]/ensure` [22:16:27] pkg-js-tools is causing `jenkins-debian-glue-buildenv lintian` to be removed. [22:17:06] somehow your instance has both buster, jessie and stretch present in /etc/apt/sources.list(.d) [22:17:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) Looks like the pkg-js-tools package causes `jenkins-debian-glue-buildenv` and `lintian` to be removed. [22:18:10] taavi: rm /etc/apt/sources.list.d/stretch* ? [22:18:51] I removed the entire .d folder and let puppet recreate it [22:19:43] on which host? [22:20:00] 1001 [22:20:34] Looks like the stretch stuff came back. But, the lintian and pkg-js-tools packages seem to be coming from the buster repo [22:20:40] (which seems right) [22:21:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) ` root@integration-agent-pkgbuilder-1001# apt show lintian Package: lintian Version: 2.104.0~bpo10+1 Priority: optional Sec... [22:22:54] removing the lintian backports pin lets me install both at the same time [22:23:53] Breaks: funny-manpages (<< 1.3-5.1), NOOO [22:24:36] The pin showed up here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/506598/2/modules/package_builder/manifests/init.pp [22:25:17] mutante: out of curiosity, can you check if the same (pkg-js-tools and lintian getting installed every other puppet run) is happening on deneb or build2001 too? [22:25:24] ok [22:25:41] I bet it is, and related to build2002 replacing denebg [22:26:04] the pin you linked to uses ${::lsbdistcodename}. maybe that is one place after all where we DO want to hardcode a distro name [22:26:12] as opposed to most other places [22:26:18] it has been updated since then [22:26:22] ah, ACK [22:26:23] sorry for that confusion. [22:26:42] Current code is : [22:26:42] ``` [22:26:42] apt::pin { 'lintian': [22:26:43] pin => "release a=${debian::codename()}-backports", [22:26:43] package => 'lintian', [22:26:43] priority => 1001, [22:26:43] before => Package['lintian'], [22:26:43] this is surely caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/765250 [22:26:44] } [22:26:44] ``` [22:26:48] deneb: Notice: /Stage[main]/Package_builder/Package[pkg-js-tools]/ensure: created (corrective) [22:26:58] ("C:package_builder: install tools to build node packages") [22:27:00] taavi: Looks like a likely candidate [22:27:37] what about build2001 (bullseye)? [22:28:35] this is lintian on buster-backports btw: Breaks: funny-manpages (<< 1.3-5.1), pkg-js-tools (<< 0.9.4), pkg-perl-tools (<< 0.53) [22:29:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) xref https://gerrit.wikimedia.org/r/c/operations/puppet/+/765250 [22:29:33] taavi: nothing on build2001 [22:31:23] * taavi wonders how to fix that [22:31:49] two options I can see are: 1) only include pkg-js-tools on bullseye and newer or 2) pull pkg-js-tools from buster-backports too [22:31:50] upgrade to bullseye seems to be one option apparently [22:32:10] I linked to your ticket on that gerrit change. [22:32:20] I would just wait for a response tomorrow I guess [22:33:01] John is out tomorrow I believe [22:34:58] wait, John is reverting [22:35:19] he just made https://gerrit.wikimedia.org/r/c/operations/puppet/+/765629 [22:38:40] taavi: dancy: https://gerrit.wikimedia.org/r/c/operations/puppet/+/765648/ [22:39:31] LGTM [22:39:40] when did people start using John for me, need to add that to my highlights :) [22:41:08] then they will use J.ohn :) [22:42:29] lol [22:43:57] ok merged [22:44:22] it's fixed on deneb [22:45:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) 05Openβ†’03Resolved Fixed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/765648/ Thanks to @Majavah @Dzahn and... [22:49:00] (03PS1) 10Jeena Huneidi: Exit if change is not mergeable [tools/scap] - 10https://gerrit.wikimedia.org/r/765649 (https://phabricator.wikimedia.org/T295494) [22:50:42] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap backport change_url command - https://phabricator.wikimedia.org/T287042 (10jeena) [22:50:44] 10Release-Engineering-Team (Done by Wed 24 Nov 🧟), 10Scap, 10Patch-For-Review: Scap backport change_url: Implement β€œwait for my commits to be merged” - https://phabricator.wikimedia.org/T295494 (10jeena) 05Openβ†’03In progress [22:51:39] (03CR) 10Ahmon Dancy: Exit if change is not mergeable (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/765649 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [22:53:06] 10Release-Engineering-Team (Done by Feb 23πŸ”₯), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) [22:54:03] (03PS2) 10Jeena Huneidi: Exit if change is not mergeable [tools/scap] - 10https://gerrit.wikimedia.org/r/765649 (https://phabricator.wikimedia.org/T295494) [22:54:17] (03CR) 10Jeena Huneidi: Exit if change is not mergeable (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/765649 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [22:55:39] (03PS3) 10Jeena Huneidi: Exit if change is not mergeable [tools/scap] - 10https://gerrit.wikimedia.org/r/765649 (https://phabricator.wikimedia.org/T295494) [23:00:37] 10Release-Engineering-Team, 10Scap, 10MW-on-K8s, 10serviceops: Deploy MediaWiki images for kubernetes from the deployment servers - https://phabricator.wikimedia.org/T302539 (10dancy) [23:02:12] 10Release-Engineering-Team (Done by Feb 23πŸ”₯), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) [23:53:45] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-extensions-Nuke: Special:Nuke on betacommons failed (Secure Connection Failed/502 Next Hop Connection Failed) - https://phabricator.wikimedia.org/T302544 (10AlexisJazz) [23:58:00] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-extensions-Nuke: Special:Nuke on betacommons failed (Secure Connection Failed/502 Next Hop Connection Failed) - https://phabricator.wikimedia.org/T302544 (10AlexisJazz)