[00:17:24] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for scherukuwada - https://phabricator.wikimedia.org/T293087 (10thcipriani) Hi @SCherukuwada have you been able to attend a training? How did it go? [00:48:42] (03CR) 10Jforrester: [C: 03+2] build: Updating dependencies [integration/docroot] - 10https://gerrit.wikimedia.org/r/737511 (owner: 10Libraryupgrader) [01:47:00] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [02:07:37] 10Continuous-Integration-Config, 10Quibble, 10ci-test-error: `npm -s run test:size` is blocking WikimediaEvents merges - https://phabricator.wikimedia.org/T295341 (10Jdlrobson) [02:08:36] 10Continuous-Integration-Config, 10Quibble, 10MediaWiki-extensions-WikimediaEvents, 10ci-test-error: `npm -s run test:size` is blocking WikimediaEvents merges - https://phabricator.wikimedia.org/T295341 (10Jdlrobson) [02:20:21] 10Continuous-Integration-Config, 10Quibble, 10MediaWiki-extensions-WikimediaEvents, 10ci-test-error: `npm -s run test:size` is blocking WikimediaEvents merges - https://phabricator.wikimedia.org/T295341 (10Jdforrester-WMF) This seems also to be blocking all merges in extensions that rely on EventLogging:... [02:23:28] 10Continuous-Integration-Config, 10Quibble, 10MediaWiki-extensions-WikimediaEvents, 10ci-test-error: `npm -s run test:size` is blocking WikimediaEvents merges - https://phabricator.wikimedia.org/T295341 (10Reedy) [02:34:33] bug bug T1 [02:34:34] T1: Get puppet runs into logstash - https://phabricator.wikimedia.org/T1 [02:38:02] !logerrors [02:38:02] https://phabricator.wikimedia.org/tag/wikimedia-log-errors/board/ [03:30:48] 10Continuous-Integration-Config, 10Quibble, 10MediaWiki-extensions-WikimediaEvents, 10ci-test-error: `npm -s run test:size` is blocking WikimediaEvents merges - https://phabricator.wikimedia.org/T295341 (10Krinkle) > The Vector `npm run test:size` job appears to be run separately now by Quibble […] What d... [03:49:06] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [06:24:55] 10Continuous-Integration-Config, 10Quibble, 10MediaWiki-extensions-WikimediaEvents, 10ci-test-error: `npm -s run test:size` is blocking WikimediaEvents merges - https://phabricator.wikimedia.org/T295341 (10abi_) Add Translate to the list of extensions affected by this: https://gerrit.wikimedia.org/r/c/medi... [06:55:29] 10Continuous-Integration-Config, 10Quibble, 10ci-test-error: `npm -s run test:size` is blocking WikimediaEvents merges - https://phabricator.wikimedia.org/T295341 (10Nikerabbit) p:05Triage→03Unbreak! > Add Translate to the list of extensions affected by this: https://gerrit.wikimedia.org/r/c/mediawiki/ex... [06:56:11] 10Continuous-Integration-Config, 10Quibble, 10ci-test-error: `npm -s run test:size` is blocking merges to many extensions - https://phabricator.wikimedia.org/T295341 (10Nikerabbit) [08:07:27] 10Continuous-Integration-Config, 10Quibble, 10ci-test-error: `npm -s run test:size` is blocking merges to many extensions - https://phabricator.wikimedia.org/T295341 (10hashar) Will check. The `test:size` entrypoint is used in MinervaNeue and Vector skin. The target URL comes from `MW_SCRIPT_PATH` environmen... [08:15:25] 10Continuous-Integration-Config, 10Quibble, 10ci-test-error: `npm -s run test:size` is blocking merges to many extensions - https://phabricator.wikimedia.org/T295341 (10hashar) From the Translate extension **Good** Around 13:30 UTC https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/736229 ht... [08:32:37] 10Continuous-Integration-Infrastructure, 10Quibble, 10castor, 10ci-test-error: `npm -s run test:size` is blocking merges to many extensions - https://phabricator.wikimedia.org/T295341 (10hashar) 05Open→03Resolved a:03hashar That is probably the same issue as T294426 or T293937: the npm cache ends up... [08:41:52] 10Continuous-Integration-Infrastructure, 10castor: npm cache saved by castor get corrupted for unknown reason - https://phabricator.wikimedia.org/T295351 (10hashar) [09:10:08] 10Continuous-Integration-Infrastructure, 10castor: npm cache saved by castor get corrupted for unknown reason - https://phabricator.wikimedia.org/T295351 (10hashar) Crawling through the build of [[ https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-selenium-docker/ | quibble-vendor-mysql-php72... [09:28:12] 10Continuous-Integration-Infrastructure, 10castor: npm cache saved by castor get corrupted for unknown reason - https://phabricator.wikimedia.org/T295351 (10hashar) Looking at a file from the corrupted cache there is the timestamp 20:31:13 ( /srv/corrupted-quibble-vendor-mysql-php72-selenium-docker/npm/_cacach... [11:21:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Patch-For-Review: Add a CI Job for running bundlesize tests in Vector - https://phabricator.wikimedia.org/T255149 (10kostajh) I didn't know about this task, but came to the same conclusion as others here about doing these bundlesize check... [11:39:38] (03CR) 10Kosta Harlan: [C: 03+2] doc: move LICENSE to its own document page [integration/quibble] - 10https://gerrit.wikimedia.org/r/735380 (owner: 10Hashar) [11:40:55] (03PS2) 10Kosta Harlan: doc: hide the table of content [integration/quibble] - 10https://gerrit.wikimedia.org/r/735379 (owner: 10Hashar) [11:41:07] (03CR) 10Kosta Harlan: [C: 03+2] doc: hide the table of content [integration/quibble] - 10https://gerrit.wikimedia.org/r/735379 (owner: 10Hashar) [12:00:05] (03Merged) 10jenkins-bot: doc: hide the table of content [integration/quibble] - 10https://gerrit.wikimedia.org/r/735379 (owner: 10Hashar) [12:10:21] (03PS3) 10Hashar: doc: move LICENSE to its own document page [integration/quibble] - 10https://gerrit.wikimedia.org/r/735380 [12:11:23] (03CR) 10Hashar: [C: 03+2] "I have rebased due to parent patch having the commit message amended." [integration/quibble] - 10https://gerrit.wikimedia.org/r/735380 (owner: 10Hashar) [12:30:09] (03Merged) 10jenkins-bot: doc: move LICENSE to its own document page [integration/quibble] - 10https://gerrit.wikimedia.org/r/735380 (owner: 10Hashar) [12:30:51] (03Abandoned) 10Hashar: Job to run npm run test:size inside Quibble env [integration/config] - 10https://gerrit.wikimedia.org/r/639166 (https://phabricator.wikimedia.org/T255149) (owner: 10Hashar) [12:31:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Patch-For-Review: Add a CI Job for running bundlesize tests in Vector - https://phabricator.wikimedia.org/T255149 (10hashar) Ah that is great. Using PHPUnit was my suggestion on the original change that went using JavaScript / API reques... [12:35:43] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Patch-For-Review: Add a CI Job for running bundlesize tests in Vector - https://phabricator.wikimedia.org/T255149 (10kostajh) >>! In T255149#7492136, @hashar wrote: > Ah that is great. Using PHPUnit was my suggestion on the original chan... [12:57:02] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:20:06] 10Phabricator: silence batch edit job - https://phabricator.wikimedia.org/T295335 (10Aklapper) @LGoto: Should be done?: ` aklapper@phab1001:~$ sudo /srv/phab/phabricator/bin/bulk make-silent --id 2903 Configured job "2903" to run silently. ` General disclaimer and more info: https://www.mediawiki.org/wiki/Phabri... [13:21:28] (03Abandoned) 10Hashar: tests: ensure extensions and skins have unique name [integration/config] - 10https://gerrit.wikimedia.org/r/737359 (owner: 10Hashar) [13:37:04] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-team (Design Systems Team FY2021-22 Kanban Board), 10Patch-For-Review: Deploy current main branch's docs site to doc.wikimedia.org - https://phabricator.wikimedia.org/T293704 (10AnneT) Could we consider an alternate solution where we use Netlify to... [13:43:08] (03PS1) 10Hashar: zuul: polish set_doc_variables() [integration/config] - 10https://gerrit.wikimedia.org/r/737677 [13:50:40] (03PS1) 10Hashar: dockerfiles: init test repo with explicit branch name [integration/config] - 10https://gerrit.wikimedia.org/r/737679 [13:52:33] (03PS6) 10Hashar: zuul: move doc_destination to a YAML file [integration/config] - 10https://gerrit.wikimedia.org/r/737059 [13:52:38] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: init test repo with explicit branch name [integration/config] - 10https://gerrit.wikimedia.org/r/737679 (owner: 10Hashar) [13:58:02] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [14:00:01] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T293950 (10Lucas_Werkmeister_WMDE) [14:25:52] (03CR) 10Nikki Nikkhoui: "@James - do you know if these changes take effect immediately or not? this repo is still running the "rehearse" stage on gate-and-submit s" [integration/config] - 10https://gerrit.wikimedia.org/r/737486 (https://phabricator.wikimedia.org/T288134) (owner: 10Nikki Nikkhoui) [14:31:46] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/737486 [14:31:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:32:47] (03CR) 10Reedy: Zuul: [mediawiki/services/example-node-api] Skip rehearse step for now (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/737486 (https://phabricator.wikimedia.org/T288134) (owner: 10Nikki Nikkhoui) [14:37:15] (03CR) 10Nikki Nikkhoui: Zuul: [mediawiki/services/example-node-api] Skip rehearse step for now (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/737486 (https://phabricator.wikimedia.org/T288134) (owner: 10Nikki Nikkhoui) [14:41:16] (03CR) 10Reedy: Zuul: [mediawiki/services/example-node-api] Skip rehearse step for now (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/737486 (https://phabricator.wikimedia.org/T288134) (owner: 10Nikki Nikkhoui) [15:08:37] (03PS1) 10Hashar: jjb: use job template for mediawiki/core phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/737721 [15:19:15] (03PS1) 10Hashar: jjb: had the template name to the job description [integration/config] - 10https://gerrit.wikimedia.org/r/737723 [15:20:23] (03PS2) 10Hashar: jjb: had the template name to the job description [integration/config] - 10https://gerrit.wikimedia.org/r/737723 [16:07:08] (03CR) 10Ahmon Dancy: [C: 03+2] backport: Approve selected/given changes [tools/scap] - 10https://gerrit.wikimedia.org/r/736589 (https://phabricator.wikimedia.org/T294454) (owner: 10Dduvall) [16:07:51] (03Merged) 10jenkins-bot: backport: Approve selected/given changes [tools/scap] - 10https://gerrit.wikimedia.org/r/736589 (https://phabricator.wikimedia.org/T294454) (owner: 10Dduvall) [16:12:24] 10Continuous-Integration-Infrastructure, 10Quibble, 10castor, 10ci-test-error: `npm -s run test:size` is blocking merges to many extensions - https://phabricator.wikimedia.org/T295341 (10Jdlrobson) Thank you @hashar for making my day a little less stressful by unblocking these <3 [16:24:51] 10Phabricator: silence batch edit job - https://phabricator.wikimedia.org/T295335 (10LGoto) 05Open→03Resolved a:03LGoto Looks good, thanks! [16:48:40] hi thcipriani, scap q [16:48:59] iirc, it is not possible to make a single scap source deploy to different target destination directories, right? [16:54:01] hashar: is there zero overlap in gate jobs for same repo/branch? [16:54:16] (Re: castor corruption) [17:01:19] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [17:01:51] ^ needs DRAC firmware upgrade which would probably fix that [17:25:12] (03PS1) 10Ahmon Dancy: auto-stage: Honor MEDIAWIKI_CONFIG_COMMIT and MEDIAWIKI_COMMIT [tools/release] - 10https://gerrit.wikimedia.org/r/737751 [17:28:42] (03CR) 10Ahmon Dancy: [C: 04-1] "Need to rethink the mediawiki/core commit pointer. It's branch-specific." [tools/release] - 10https://gerrit.wikimedia.org/r/737751 (owner: 10Ahmon Dancy) [17:43:56] Any idea when the official mw1.37 might happen? [17:51:00] * legoktm quickly reassigns the remaining blockers to addshore [17:56:08] addshore: late this month [17:58:16] addshore: Yeah, rc.2 out tomorrow, then within the next couple of weeks 1.37 should be fine [17:58:21] pls help test rc.2 :P [18:02:08] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [18:08:24] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Release 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295176 (10Reedy) [18:08:30] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Write release announcement for 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295178 (10Reedy) 05Open→03In progress [18:08:40] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Release 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295176 (10Reedy) 05Open→03In progress [18:08:46] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Release MW 1.37.0 - https://phabricator.wikimedia.org/T289585 (10Reedy) [18:10:48] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Write release announcement for 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295178 (10Reedy) [18:11:24] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Write release announcement for 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295178 (10Reedy) [18:22:04] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release, 10Patch-For-Review: Release 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295176 (10Reedy) [18:22:36] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Write release announcement for 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295178 (10Reedy) [18:29:05] (03PS2) 10Ahmon Dancy: auto-stage: Honor MEDIAWIKI_CONFIG_COMMIT and MEDIAWIKI_COMMITS [tools/release] - 10https://gerrit.wikimedia.org/r/737751 [18:38:25] (03CR) 10Ahmon Dancy: [C: 03+2] auto-stage: Honor MEDIAWIKI_CONFIG_COMMIT and MEDIAWIKI_COMMITS [tools/release] - 10https://gerrit.wikimedia.org/r/737751 (owner: 10Ahmon Dancy) [18:39:55] (03Merged) 10jenkins-bot: auto-stage: Honor MEDIAWIKI_CONFIG_COMMIT and MEDIAWIKI_COMMITS [tools/release] - 10https://gerrit.wikimedia.org/r/737751 (owner: 10Ahmon Dancy) [18:45:28] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Write release announcement for 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295178 (10Reedy) a:03Reedy [18:45:45] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release, 10Patch-For-Review: Release 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295176 (10Reedy) a:03Reedy [18:46:21] legoktm: hehhee [18:46:25] *looks at the board* [18:46:55] wait, it has no blockers? :P [18:50:12] Cause we've resolved them ;) [18:53:03] (03PS1) 10Ahmon Dancy: Makefile: Simplify passing environment variables to docker [tools/release] - 10https://gerrit.wikimedia.org/r/737766 [18:54:16] (03PS2) 10Ahmon Dancy: Makefile: Simplify passing environment variables to docker [tools/release] - 10https://gerrit.wikimedia.org/r/737766 [18:56:54] (03CR) 10Ahmon Dancy: [C: 03+2] Makefile: Simplify passing environment variables to docker [tools/release] - 10https://gerrit.wikimedia.org/r/737766 (owner: 10Ahmon Dancy) [18:58:37] (03Merged) 10jenkins-bot: Makefile: Simplify passing environment variables to docker [tools/release] - 10https://gerrit.wikimedia.org/r/737766 (owner: 10Ahmon Dancy) [19:00:21] (03PS1) 10Ahmon Dancy: build-mw-image-loop.py: Use GIT_BASE instead of git_base [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737768 [19:00:32] (03CR) 10Ahmon Dancy: [C: 03+2] build-mw-image-loop.py: Use GIT_BASE instead of git_base [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737768 (owner: 10Ahmon Dancy) [19:01:09] (03Merged) 10jenkins-bot: build-mw-image-loop.py: Use GIT_BASE instead of git_base [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737768 (owner: 10Ahmon Dancy) [19:12:43] ottomata: sorry lots of meetings, that's probably true. A simple work around might be to check it out twice from the deployment server, but I'm guessing you need the two directories to move in lock-step? [19:13:07] actually...i discovered it does work! and I think I made it work years ago! [19:13:27] no, don't need two dirs to move lock stuff, basically same repo used for multiple deployments [19:13:42] mostly the same scap config, but different hosts...i thinkk i can figure this out [19:14:49] oh :D [19:15:27] well, glad to hear it works, sorry for the late reply but it sounds like I may have not been much help :P [19:16:08] thanks for the gitlab scap_source patch by-the-by, very exciting :) [19:16:50] :) i hope that works! [19:17:02] and/or...uhhh maybe some other cool way to deploy with gitlab [19:17:08] scap still is THE way, right? [19:17:37] for now :) [19:17:46] unless it's something that runs on k8s [19:18:37] yeah [19:18:48] we'd love to use more k8s, but it doesn't work with kerberos unfortunetly [19:30:53] 10Release-Engineering-Team (Done by Thu 04 Nov 🧟), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap backport change_url: approve changes - https://phabricator.wikimedia.org/T294454 (10dancy) 05In progress→03Resolved [19:30:55] 10Release-Engineering-Team (Done by Thu 04 Nov 🧟), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap backport change_url command - https://phabricator.wikimedia.org/T287042 (10dancy) [19:36:30] !log gitlab: creating top-level /repos group (T292094) [19:36:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:36:33] T292094: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 [19:43:52] 10Release-Engineering-Team (Doing), 10Security-Team, 10GitLab (CI & Job Runners), 10SecTeam-Processed, 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10mmodell) >>! In T292094#7480801, @gmodena wrote: > Do you think this makes sense? Would... [19:49:03] 10Phabricator, 10Data-Engineering: Herald_rule_for_Data-Engineering - https://phabricator.wikimedia.org/T295397 (10odimitrijevic) [19:49:33] 10Phabricator, 10Data-Engineering: Herald rule for Data-Engineering - https://phabricator.wikimedia.org/T295397 (10Reedy) [19:50:46] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Release MW 1.37.0 - https://phabricator.wikimedia.org/T289585 (10Reedy) [19:54:13] 10Phabricator, 10Data-Engineering: Herald rule for Data-Engineering - https://phabricator.wikimedia.org/T295397 (10odimitrijevic) @Milimetric are there any other rules we want to carry over? There are many other analytics related tag rules that were defined by https://phabricator.wikimedia.org/T153763, howeve... [19:58:47] 10Phabricator, 10Release-Engineering-Team (Done by Thu 04 Nov 🧟): give visibility for "in progress" tasks on a work board - https://phabricator.wikimedia.org/T291593 (10mmodell) @AntiCompositeNumber - how about this: {F34739547} [20:03:45] 10Phabricator: Disable Slowvote application - https://phabricator.wikimedia.org/T285304 (10mmodell) It seems like the consensus is to disable it but if there is a use-case for it then I'm all for supporting it. [20:04:41] 10Release-Engineering-Team (Doing), 10Security-Team, 10GitLab (CI & Job Runners), 10SecTeam-Processed, 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10brennen) A quick update here: In the short term (i.e. today and this week), we're going... [20:07:06] 10Phabricator, 10Data-Engineering: Herald rule for Data-Engineering - https://phabricator.wikimedia.org/T295397 (10Milimetric) @odimitrijevic I think we can easily make rules as we need them. Maybe you should decide on the #analytics-clusters rule (it works like #analytics-radar now). The problem I see is th... [20:10:47] 10Continuous-Integration-Infrastructure: Migrate quibble images from node10 to something modern - https://phabricator.wikimedia.org/T294931 (10Jdforrester-WMF) This is blocking https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/737715 as even though we don't actually run server-side JS it's failing o... [20:16:55] (03PS1) 10Jforrester: docker: [quibble] Replace buster-native node10 with node14 [integration/config] - 10https://gerrit.wikimedia.org/r/737777 (https://phabricator.wikimedia.org/T294931) [20:27:34] (03CR) 10Jforrester: [C: 03+2] docker: [quibble] Replace buster-native node10 with node14 [integration/config] - 10https://gerrit.wikimedia.org/r/737777 (https://phabricator.wikimedia.org/T294931) (owner: 10Jforrester) [20:29:58] (03CR) 10Jforrester: [C: 03+2] "Thank you! Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/737721 (owner: 10Hashar) [20:30:43] (03Merged) 10jenkins-bot: docker: [quibble] Replace buster-native node10 with node14 [integration/config] - 10https://gerrit.wikimedia.org/r/737777 (https://phabricator.wikimedia.org/T294931) (owner: 10Jforrester) [20:32:01] (03Merged) 10jenkins-bot: jjb: use job template for mediawiki/core phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/737721 (owner: 10Hashar) [20:32:07] !log Publishing quibble images with node14 not node10 for T294931 [20:32:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:32:10] T294931: Migrate quibble images from node10 to something modern - https://phabricator.wikimedia.org/T294931 [20:38:06] (03PS1) 10Ahmon Dancy: Hide ssh-keyscan/ssh-keygen noise [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737779 [20:38:20] (03PS1) 10Ahmon Dancy: Install docker in deploy container [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737780 [20:38:42] 10Phabricator: Disable Slowvote application - https://phabricator.wikimedia.org/T285304 (10Aklapper) Please see my initial task description here: I wrote that we originally declined Slowvote in T89906, and that I have not the slightest clue why random things nevertheless got enabled at some point (does someone k... [20:40:03] (03PS1) 10Jforrester: Follow-up b74854e59: Add missing file so things actually build [integration/config] - 10https://gerrit.wikimedia.org/r/737781 [20:40:05] (03PS1) 10Jforrester: jjb: Migrate quibble jobs to images with node14 not node10 [integration/config] - 10https://gerrit.wikimedia.org/r/737782 (https://phabricator.wikimedia.org/T294931) [20:40:30] (03CR) 10Jforrester: [C: 03+2] "*coughs*" [integration/config] - 10https://gerrit.wikimedia.org/r/737781 (owner: 10Jforrester) [20:42:28] (03Merged) 10jenkins-bot: Follow-up b74854e59: Add missing file so things actually build [integration/config] - 10https://gerrit.wikimedia.org/r/737781 (owner: 10Jforrester) [20:45:36] building is overrated [21:28:27] !log gitlab: migrating /releng/* to /repos/releng/ = cc: addshore, jeena as this includes cli, dev-images [21:28:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:28:40] ooooo [21:28:45] (that was supposed to be a - and not a =. oh well.) [21:29:13] will there be redirects? D: [21:29:21] * addshore goes to check if that matters [21:29:31] there should be. [21:29:36] nice :) [21:29:52] i'm not totally sure if they'll persist if i zap the top-level /releng [21:31:41] ...made it private, which should be good enough for the moment. [21:32:15] * addshore confirms mwcli updating still works :) nice It uses the project is of 12 anyway so i guess everything else get auto figured out [21:32:21] *16 [21:38:10] (03CR) 10Hashar: [C: 03+1] "There are two things I am thinking about:" [integration/config] - 10https://gerrit.wikimedia.org/r/737782 (https://phabricator.wikimedia.org/T294931) (owner: 10Jforrester) [21:45:50] (03CR) 10Ahmon Dancy: "Timo, this is what I'm trying to do that led to https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/737468" [tools/scap] - 10https://gerrit.wikimedia.org/r/737495 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy) [21:50:16] what is the thing in CI that talks to Gerrit and does it have logs? [21:51:46] TimStarling: possibly "zuul" [contint1001:/var/log/zuul] or "jenkins" [contint1001:/var/log/jenkins] $ [21:52:07] that would be contint1001.wikimedia.org [21:52:16] eh, wait, contint2001 is currenty production [21:52:28] that but in codfw [21:52:38] not logs, but a status UI: https://integration.wikimedia.org/zuul/ [21:54:54] I'm just investigating a change that was given V+2 but not submitted -- but its branch history is weird and confusing, it probably just needs rebasing [21:55:53] assuming it's the same repo as usual? some repos can have different config where it doesn't auto submit [21:56:14] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/735761/2 [21:56:20] *nod* [21:57:12] maybe putting "recheck" on it makes it try again [21:57:20] agreed, the red (Merged) for its parent commit looks suspicious [21:57:28] its parent got CR+2 before its grandparent, so I rebased the parent out of the topic branch and merged it separately [21:57:41] I can rebase it on top of master now to fix that [21:57:54] 👍🏾 [21:58:14] It would have been nice if you had received some notification though. [21:59:29] meh, it's all going to be replaced by gitlab soon enough right? [21:59:46] That is the plan. [22:02:09] for certain values of soon :) [22:02:31] things do look like they are progressing though which is nice [22:02:34] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Improve efficiency of scap l10n operations - https://phabricator.wikimedia.org/T295304 (10Krinkle) Note that the cdb->json->rsync->json->cdb dance is expected to go away "soon" per {T99740}. [22:02:35] nod. [22:03:38] Big projects take a long time just to work though all the parts and I think we all sometimes forget that and want results now. [22:04:31] (03CR) 10Krinkle: Add cdb_rebuild_using_rebuildLocalisationCache config option (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/737495 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy) [22:08:52] (03CR) 10Ahmon Dancy: Add cdb_rebuild_using_rebuildLocalisationCache config option (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/737495 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy) [22:29:07] !log gitlab: finished moving releng, security, data-engineering projects to /repos (T292094) [22:29:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:29:10] T292094: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 [22:44:05] (03CR) 10Jforrester: jjb: Migrate quibble jobs to images with node14 not node10 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/737782 (https://phabricator.wikimedia.org/T294931) (owner: 10Jforrester) [22:48:12] !log gitlab-runners: unregistered instance-wide shared runners, re-registered to /repos (T292094) [22:48:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:48:15] T292094: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 [23:06:24] (03CR) 10Jforrester: [C: 03+2] "OK, deployed and all looks OK so far." [integration/config] - 10https://gerrit.wikimedia.org/r/737782 (https://phabricator.wikimedia.org/T294931) (owner: 10Jforrester) [23:08:18] (03Merged) 10jenkins-bot: jjb: Migrate quibble jobs to images with node14 not node10 [integration/config] - 10https://gerrit.wikimedia.org/r/737782 (https://phabricator.wikimedia.org/T294931) (owner: 10Jforrester) [23:19:19] I am repeatedly smashing my head against the desk trying to work out a fix for a problem running the Toolhub dev environment on a Linux host. Maybe talking out loud here will either help me figure out what to try next or invite help from someone. [23:20:42] The basic problem is getting a docker-compose stack running containers built with blubber to let Docker on Linux mount a git clone read-write into the container at /srv/app. [23:21:22] On an OSX host, Docker somehow magically manages the uid/gid difference and things "just work" [23:21:32] hehe [23:21:53] on a Linux host, the uid/gid mismatch between the container and the host makes it all go boom [23:22:25] so then you do the trick of setting the effective UID/GID in the docker-compose file for the container [23:22:41] get us a step closer, and r/w should work [23:23:39] but... now there is a new problem with uid/gid and perms. The container runs `npm install` things both at container build and at container start. [23:24:15] The run that happens during build creates $HOME/.npm cache bits owned by the UID that blubber sets [23:24:44] when the container is later started as a different UID that cache is not readable and npm goes boom [23:25:09] so the next trick to try is mounting a volume over the $HOME/.npm location [23:25:44] should be a nice hack right? get an empty (or even persistent) $HOME/.npm cache of things for the runtime [23:26:13] but... Docker stabs me in the heart again by mounting the volume with root:root owenrship [23:26:54] so the effective UID can't write there and also npm has some "oops I see root owned things in the directory, bye" logic [23:27:30] This is where I'm now out of ideas. [23:28:29] "buy Raymond a mackbook" is looking like an elegant, but difficult to scale, solution :/ [23:30:19] The runtime npm error is https://phabricator.wikimedia.org/P17714 [23:30:55] which comes with a helpful sudo fix, but of course the EUID user has no sudo rights so that's right out [23:40:36] Sounds like what you need most is to run code inside the container using the same uid as outside the container. [23:42:15] I'm trying a new idea for the $HOME/.npm issue and I think it may work. The idea is to create a /tmp/runtime-home as the EUID, export HOME=/tmp/runtime-home, and then ?? profit ?? [23:42:32] 10Continuous-Integration-Infrastructure: Migrate quibble images from node10 to something modern - https://phabricator.wikimedia.org/T294931 (10hashar) antoine-approve [23:42:48] I need to undo some other things I have tried here before I can see this work or fail. [23:42:50] Using a path under /tmp is a great workaround for the npm stuff.. [23:49:02] omg... I think this may be working... [23:55:13] w00t w00t w00t. I haz a running container. It wrote new files to the mounted external directory. and the ownership looks ok inside and out. [23:55:38] Nice work [23:55:49] now to package up these changes as a patch and see if they work for Raymond [23:56:08] thank you dancy for being a rubber duck [23:56:21] np [23:56:39] https://en.wikipedia.org/wiki/Rubber_duck_debugging for anyone confused by that statement