[00:00:04] (03Merged) 10jenkins-bot: jjb: Drop ooui-special-node12-plus-php80-composer-docker, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/770631 (owner: 10Jforrester) [00:00:06] (03Merged) 10jenkins-bot: jjb: Provide wikipeg-special-node14-plus-php80-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770632 (owner: 10Jforrester) [00:00:34] (03PS1) 10Jforrester: jjb: [oojs-ui-docker-publish] Switch from node12 to node14 image [integration/config] - 10https://gerrit.wikimedia.org/r/770635 [00:00:50] (03CR) 10Jforrester: [C: 03+2] Zuul: [wikipeg] Switch from node12 to node14 special job [integration/config] - 10https://gerrit.wikimedia.org/r/770633 (owner: 10Jforrester) [00:02:33] (03Merged) 10jenkins-bot: Zuul: [wikipeg] Switch from node12 to node14 special job [integration/config] - 10https://gerrit.wikimedia.org/r/770633 (owner: 10Jforrester) [00:08:23] !log Zuul: [wikipeg] Switch from node12 to node14 special job [00:08:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:08:37] (03CR) 10Jforrester: [C: 03+2] jjb: Drop wikipeg-special-node12-plus-php80-composer-docker, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/770634 (owner: 10Jforrester) [00:10:38] (03Merged) 10jenkins-bot: jjb: Drop wikipeg-special-node12-plus-php80-composer-docker, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/770634 (owner: 10Jforrester) [00:48:20] (03CR) 10Jforrester: [C: 03+2] jjb: [oojs-ui-docker-publish] Switch from node12 to node14 image [integration/config] - 10https://gerrit.wikimedia.org/r/770635 (owner: 10Jforrester) [00:50:06] (03Merged) 10jenkins-bot: jjb: [oojs-ui-docker-publish] Switch from node12 to node14 image [integration/config] - 10https://gerrit.wikimedia.org/r/770635 (owner: 10Jforrester) [00:51:21] (03PS3) 10Jforrester: jjb: Switch selenium-daily-* jobs from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770538 [00:53:35] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/770538 (owner: 10Jforrester) [00:53:44] (03PS3) 10Jforrester: jjb: Switch language-screenshots-VisualEditor from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770539 [00:54:19] (03CR) 10Jforrester: [C: 03+2] jjb: Switch language-screenshots-VisualEditor from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770539 (owner: 10Jforrester) [00:54:25] (03PS3) 10Jforrester: jjb: Switch mobileapps-periodic-test from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770540 [00:55:39] (03Merged) 10jenkins-bot: jjb: Switch selenium-daily-* jobs from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770538 (owner: 10Jforrester) [00:56:12] (03Merged) 10jenkins-bot: jjb: Switch language-screenshots-VisualEditor from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770539 (owner: 10Jforrester) [00:56:34] (03CR) 10Jforrester: [C: 03+2] jjb: Switch mobileapps-periodic-test from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770540 (owner: 10Jforrester) [00:56:40] (03PS3) 10Jforrester: jjb: Switch mwext-codehealth* from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770541 [00:58:08] (03PS4) 10Jforrester: jjb: Switch mediawiki-core-javascript-docker-publish from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770541 [00:58:18] (03Merged) 10jenkins-bot: jjb: Switch mobileapps-periodic-test from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770540 (owner: 10Jforrester) [00:58:26] (03CR) 10Jforrester: [C: 03+2] jjb: Switch mediawiki-core-javascript-docker-publish from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770541 (owner: 10Jforrester) [00:58:32] (03PS3) 10Jforrester: jjb: Switch mediawiki-core-javascript-docker-publish from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770542 [01:00:05] (03Merged) 10jenkins-bot: jjb: Switch mediawiki-core-javascript-docker-publish from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770541 (owner: 10Jforrester) [01:02:10] (03PS4) 10Jforrester: jjb: Switch mwext-codehealth- from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770542 [01:02:18] (03CR) 10Jforrester: [C: 03+2] jjb: Switch mwext-codehealth- from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770542 (owner: 10Jforrester) [01:02:34] (03PS3) 10Jforrester: jjb: Switch wikimedia-portals-build from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770543 [01:03:00] (03CR) 10Jforrester: [C: 03+2] jjb: Switch wikimedia-portals-build from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770543 (owner: 10Jforrester) [01:04:09] (03Merged) 10jenkins-bot: jjb: Switch mwext-codehealth- from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770542 (owner: 10Jforrester) [01:04:50] (03Merged) 10jenkins-bot: jjb: Switch wikimedia-portals-build from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770543 (owner: 10Jforrester) [01:05:59] (03PS2) 10Jforrester: jjb: Provide branchdeploy-wikidata-query-builder-node14-npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770581 [01:06:01] (03PS2) 10Jforrester: Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770582 [01:06:03] (03PS2) 10Jforrester: jjb: Drop branchdeploy-wikidata-query-builder-node12-npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770583 [01:06:08] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/770581 (owner: 10Jforrester) [01:07:51] (03Merged) 10jenkins-bot: jjb: Provide branchdeploy-wikidata-query-builder-node14-npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770581 (owner: 10Jforrester) [01:09:08] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10Performance-Team (Radar): Talk to Performance team and ServiceOps about caching for train experiment week - https://phabricator.wikimedia.org/T303758 (10dpifke) [01:12:04] (03CR) 10Jforrester: [C: 03+2] Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770582 (owner: 10Jforrester) [01:13:46] (03Merged) 10jenkins-bot: Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770582 (owner: 10Jforrester) [01:14:46] !log Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14 [01:14:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:16:01] (03CR) 10Jforrester: [C: 03+2] jjb: Drop branchdeploy-wikidata-query-builder-node12-npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770583 (owner: 10Jforrester) [01:17:57] (03Merged) 10jenkins-bot: jjb: Drop branchdeploy-wikidata-query-builder-node12-npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770583 (owner: 10Jforrester) [01:25:06] (03PS1) 10Jforrester: jjb: Provide wikibase-daily-npm-audit-daily-node14-npmaudit-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770641 [01:25:08] (03PS1) 10Jforrester: jjb: Drop wikibase-daily-npm-audit-daily-node12-npmaudit-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770642 [01:25:10] (03PS1) 10Jforrester: jjb: Drop unused node{12,14}-npmaudit jobs [integration/config] - 10https://gerrit.wikimedia.org/r/770643 [01:25:12] (03PS1) 10Jforrester: jjb: Drop unused daily-node12-npmaudit-docker template [integration/config] - 10https://gerrit.wikimedia.org/r/770644 [01:25:14] (03PS1) 10Jforrester: jjb: Drop generic-node12-docs-docker-publish, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/770645 [01:26:38] (03PS1) 10Jforrester: Zuul: Switch services/function* publish job from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770646 [01:26:40] (03PS1) 10Jforrester: jjb: Drop generic-node12-browser-coverage-docker-publish, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/770647 [01:29:01] (03CR) 10Jforrester: [C: 03+2] jjb: Provide wikibase-daily-npm-audit-daily-node14-npmaudit-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770641 (owner: 10Jforrester) [01:29:06] (03CR) 10Jforrester: [C: 03+2] jjb: Drop wikibase-daily-npm-audit-daily-node12-npmaudit-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770642 (owner: 10Jforrester) [01:29:09] (03CR) 10Jforrester: [C: 03+2] jjb: Drop unused node{12,14}-npmaudit jobs [integration/config] - 10https://gerrit.wikimedia.org/r/770643 (owner: 10Jforrester) [01:29:18] (03CR) 10Jforrester: [C: 03+2] jjb: Drop unused daily-node12-npmaudit-docker template [integration/config] - 10https://gerrit.wikimedia.org/r/770644 (owner: 10Jforrester) [01:29:34] (03CR) 10Jforrester: [C: 03+2] jjb: Drop generic-node12-docs-docker-publish, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/770645 (owner: 10Jforrester) [01:30:26] (03CR) 10Jforrester: [C: 03+2] Zuul: Switch services/function* publish job from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770646 (owner: 10Jforrester) [01:31:04] (03Merged) 10jenkins-bot: jjb: Provide wikibase-daily-npm-audit-daily-node14-npmaudit-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770641 (owner: 10Jforrester) [01:31:06] (03Merged) 10jenkins-bot: jjb: Drop wikibase-daily-npm-audit-daily-node12-npmaudit-docker [integration/config] - 10https://gerrit.wikimedia.org/r/770642 (owner: 10Jforrester) [01:31:08] (03Merged) 10jenkins-bot: jjb: Drop unused node{12,14}-npmaudit jobs [integration/config] - 10https://gerrit.wikimedia.org/r/770643 (owner: 10Jforrester) [01:31:10] (03Merged) 10jenkins-bot: jjb: Drop unused daily-node12-npmaudit-docker template [integration/config] - 10https://gerrit.wikimedia.org/r/770644 (owner: 10Jforrester) [01:31:24] (03Merged) 10jenkins-bot: jjb: Drop generic-node12-docs-docker-publish, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/770645 (owner: 10Jforrester) [01:32:30] (03Merged) 10jenkins-bot: Zuul: Switch services/function* publish job from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770646 (owner: 10Jforrester) [01:37:11] !log Zuul: Switch services/function* publish job from node12 to node14 [01:37:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:37:29] (03CR) 10Jforrester: [C: 03+2] jjb: Drop generic-node12-browser-coverage-docker-publish, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/770647 (owner: 10Jforrester) [01:39:14] (03Merged) 10jenkins-bot: jjb: Drop generic-node12-browser-coverage-docker-publish, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/770647 (owner: 10Jforrester) [01:42:20] (03PS3) 10Jforrester: jjb: Switch wikidata-query-*-build from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770544 [01:48:11] (03CR) 10Jforrester: [C: 03+2] jjb: Switch wikidata-query-*-build from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770544 (owner: 10Jforrester) [01:50:26] (03Merged) 10jenkins-bot: jjb: Switch wikidata-query-*-build from node12 to node14 [integration/config] - 10https://gerrit.wikimedia.org/r/770544 (owner: 10Jforrester) [02:01:24] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10User-zeljkofilipin: Upgrade all CI jobs for WMF-deployed projects from Node 12 to Node 14 LTS - https://phabricator.wikimedia.org/T267890 (10Jdforrester-WMF) [02:04:45] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10User-zeljkofilipin: Upgrade all CI jobs for WMF-deployed projects from Node 12 to Node 14 LTS - https://phabricator.wikimedia.org/T267890 (10Jdforrester-WMF) There is a very small set of three services which have a node12 direct CI job (in... [02:15:43] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [02:16:14] 10MediaWiki-Releasing, 10MW-1.38-release, 10Patch-For-Review: Branch REL1_38 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T302909 (10Jdforrester-WMF) 05Openโ†’03Resolved ` jforrester@deploy1002:~/release-tools/make-release$ date Tue 15 Mar 2022 02:08:09 AM UTC ` ` jforres... [02:16:20] 10MediaWiki-Releasing, 10MW-1.38-release: Release MW 1.38.0 - https://phabricator.wikimedia.org/T302893 (10Jdforrester-WMF) [02:20:43] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [02:40:44] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [02:53:33] PROBLEM - Check systemd state on doc1001 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc1002.eqiad.wmnet.service,rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [03:47:15] RECOVERY - Check systemd state on doc1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [03:53:15] (03CR) 10DannyS712: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/770061 (owner: 10DannyS712) [04:29:57] Project beta-update-databases-eqiad build #57280: 04FAILURE in 9 min 57 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/57280/ [05:26:17] (03CR) 10Hashar: [C: 03+2] .editorconfig: use HTTPS for link [integration/config] - 10https://gerrit.wikimedia.org/r/770061 (owner: 10DannyS712) [05:28:01] (03Merged) 10jenkins-bot: .editorconfig: use HTTPS for link [integration/config] - 10https://gerrit.wikimedia.org/r/770061 (owner: 10DannyS712) [05:29:59] Yippee, build fixed! [05:30:00] Project beta-update-databases-eqiad build #57281: 09FIXED in 9 min 59 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/57281/ [08:08:27] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Ensure train-blockers.toolforge.org will find correct train blocker during train experiment week - https://phabricator.wikimedia.org/T303751 (10Majavah) >>! In T303751#7776149, @brennen wrote: > I set this up locally and looked over the code. I **think** that:... [08:51:04] (03CR) 10Jaime Nuche: deploy-promote: integrate into `scap` (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/769023 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [09:04:41] (03CR) 10Jaime Nuche: deploy-promote: integrate into `scap` (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/769023 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [09:22:07] 10Phabricator: Spaces request for Wikimedia IT Private - https://phabricator.wikimedia.org/T303518 (10Aklapper) 05Openโ†’03Resolved a:03Aklapper Private Space S23 (called "WMIT-Private") has been created. Notes for anyone to use this Space to create non-public tasks in that Space: * To create //non-public//... [10:31:45] (03CR) 10Jaime Nuche: deploy-promote: integrate into `scap` (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/769023 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [10:33:51] (03CR) 10Jaime Nuche: [C: 03+2] scap deploy-promote: Make idempotent [tools/scap] - 10https://gerrit.wikimedia.org/r/770029 (owner: 10Ahmon Dancy) [10:34:41] (03Merged) 10jenkins-bot: scap deploy-promote: Make idempotent [tools/scap] - 10https://gerrit.wikimedia.org/r/770029 (owner: 10Ahmon Dancy) [12:58:39] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [14:03:37] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828 (10hashar) [14:04:49] 10Phabricator: Disallow adding Project Tags in the Subscribers field - https://phabricator.wikimedia.org/T303829 (10Aklapper) p:05Triageโ†’03Low [14:04:56] 10Phabricator: Disallow adding Project Tags in the Subscribers field - https://phabricator.wikimedia.org/T303829 (10Aklapper) [14:05:07] 10Release-Engineering-Team (Onboarding ๐Ÿš€): Delete the wmf/1.37.0-wmf.* branches once we've fully deployed to 1.38.0-wmf.1 and finished with them - https://phabricator.wikimedia.org/T291501 (10hashar) I was looking how we delete the wmf branches in T303828. Looks like we got rid of `wmf/1.37.0-wmf.*` for mediawik... [14:05:59] 10Phabricator: Disallow adding Project Tags in the Subscribers field - https://phabricator.wikimedia.org/T303829 (10Aklapper) See also T303773#7777737 which made me create this ticket, finally. [14:17:22] !log Apply label `castor` to node https://integration.wikimedia.org/ci/computer/integration-castor03/ # T216244 [14:17:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:17:27] T216244: Don't hardcode castor url in castor docker container - https://phabricator.wikimedia.org/T216244 [14:19:45] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10castor: Don't hardcode castor url in castor docker container - https://phabricator.wikimedia.org/T216244 (10hashar) a:03hashar [14:21:57] (03PS1) 10Hashar: jjb: tie castor saving job to a label [integration/config] - 10https://gerrit.wikimedia.org/r/770920 (https://phabricator.wikimedia.org/T216244) [14:28:56] (03PS1) 10Hashar: dockerfiles: castor hostname from CASTOR_HOST env [integration/config] - 10https://gerrit.wikimedia.org/r/770921 (https://phabricator.wikimedia.org/T216244) [14:30:48] !log CI Jenkins: globally defined CASTOR_HOST=integration-castor03.integration.eqiad.wmflabs via https://integration.wikimedia.org/ci/configure # T216244 [14:30:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:30:50] T216244: Don't hardcode castor url in castor docker container - https://phabricator.wikimedia.org/T216244 [14:32:44] (03CR) 10Hashar: "I have defined it globally in the CI Jenkins config: https://integration.wikimedia.org/ci/jobConfigHistory/showDiffFiles?name=config×" [integration/config] - 10https://gerrit.wikimedia.org/r/770921 (https://phabricator.wikimedia.org/T216244) (owner: 10Hashar) [14:41:32] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Ensure train-branch-bot can cut three branches - https://phabricator.wikimedia.org/T303834 (10hashar) [14:42:33] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Ensure train-branch-bot can cut three branches - https://phabricator.wikimedia.org/T303834 (10hashar) [14:43:07] 10MediaWiki-Releasing, 10MW-1.38-release: Release MW 1.38.0 - https://phabricator.wikimedia.org/T302893 (10Jdforrester-WMF) [14:43:58] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Ensure train-branch-bot can cut three branches - https://phabricator.wikimedia.org/T303834 (10hashar) I don't see why the branch script could not be invoked several time in a week. We might just have to adjust the cronjob entry to trigger it several times per... [14:44:43] 10Project-Admins: Create Tag for WMIT-CiviCRM - https://phabricator.wikimedia.org/T303835 (10valerio.bozzolan) [14:45:06] 10Project-Admins, 10WMIT-Infrastructure: Create Tag for WMIT-CiviCRM - https://phabricator.wikimedia.org/T303835 (10valerio.bozzolan) [14:45:43] 10Phabricator, 10WMIT-Infrastructure: Spaces request for Wikimedia IT Private - https://phabricator.wikimedia.org/T303518 (10valerio.bozzolan) [14:48:24] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Update deployment calendar/roadmap to reflect upcoming train experiment week - https://phabricator.wikimedia.org/T303365 (10hashar) I have made the bot to read the file as YAML rather than JSON. I have send a patch that reformat the json file to yaml with http... [14:52:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10castor: Castor should aggregate wmf/* cache - https://phabricator.wikimedia.org/T303836 (10hashar) [14:58:23] James_F: hi! The daily-npm-audit jobs are used though not triggered by zuul [14:58:37] https://integration.wikimedia.org/ci/job/wikibase-daily-npm-audit-daily-node12-npmaudit-docker/ is still around [14:58:43] the node14 hasn't been deployed I guess [14:58:45] I will fix it [15:00:13] oh no [15:00:14] it is there :) [15:00:19] https://integration.wikimedia.org/ci/job/wikibase-daily-npm-audit-daily-node14-npmaudit-docker/ [15:00:27] hashar: Yeah. [15:00:33] I have deleted the node12 one [15:00:36] But not generic? [15:00:48] The only user of the template was Wikibase. [15:01:45] that was a nice spam of Gerrit notifications :] [15:02:51] the generic one I have no idea whether it used [15:03:40] deleted https://integration.wikimedia.org/ci/job/generic-node12-npmaudit-docker/ [15:04:45] and the node14 one [15:05:41] addshore commented out the email notification to wikidata-ci-status@wikimedia.de [15:05:48] so I don't think that will ever be fixed [15:07:01] (03CR) 10Hashar: [C: 03+2] jjb: tie castor saving job to a label [integration/config] - 10https://gerrit.wikimedia.org/r/770920 (https://phabricator.wikimedia.org/T216244) (owner: 10Hashar) [15:08:44] (03Merged) 10jenkins-bot: jjb: tie castor saving job to a label [integration/config] - 10https://gerrit.wikimedia.org/r/770920 (https://phabricator.wikimedia.org/T216244) (owner: 10Hashar) [15:46:41] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog ๐Ÿ“ฅ), 10Release Pipeline, 10User-dcaro: Proof of concept for Buildpacks as WMF image build tool in GitLab - https://phabricator.wikimedia.org/T291017 (10dcaro) [15:51:29] (03PS2) 10Jaime Nuche: deploy-promote: migrate script to `tools/scap` repository [tools/release] - 10https://gerrit.wikimedia.org/r/769026 (https://phabricator.wikimedia.org/T30248) [16:14:06] James_F: oi [16:18:26] no I'm stupid [16:21:09] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10Scap: Avoid rsyncing 2GB of L10N each day during train experiment week, if possible - https://phabricator.wikimedia.org/T303749 (10dancy) The experiment is not working out well at the moment because the contents of $wgExtensionMessagesFiles changes between... [16:21:59] (03PS1) 10Kosta Harlan: phan: Add TimedMediaHandler as phan dependency for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/770963 (https://phabricator.wikimedia.org/T302738) [16:40:52] (03CR) 10Ahmon Dancy: [C: 03+2] "Love it. Works great" [tools/scap] - 10https://gerrit.wikimedia.org/r/763864 (https://phabricator.wikimedia.org/T301417) (owner: 10Dduvall) [16:41:50] (03Merged) 10jenkins-bot: Provide scap prep auto history browsing and replay [tools/scap] - 10https://gerrit.wikimedia.org/r/763864 (https://phabricator.wikimedia.org/T301417) (owner: 10Dduvall) [16:46:25] 10Release-Engineering-Team, 10SRE, 10SRE-Access-Requests, 10serviceops: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10Joe) a:03Joe [16:46:42] (03CR) 10Ahmon Dancy: [C: 03+2] phan: Add TimedMediaHandler as phan dependency for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/770963 (https://phabricator.wikimedia.org/T302738) (owner: 10Kosta Harlan) [16:48:38] (03Merged) 10jenkins-bot: phan: Add TimedMediaHandler as phan dependency for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/770963 (https://phabricator.wikimedia.org/T302738) (owner: 10Kosta Harlan) [16:49:11] dancy: thank you! [16:49:40] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/770963 [16:49:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:49:56] You' [16:50:03] You're welcome. It should be fully deployed now [17:01:10] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10Infrastructure-Foundations: Need a service account on deploy servers - https://phabricator.wikimedia.org/T303857 (10dancy) [17:11:17] 10Release-Engineering-Team, 10SRE, 10SRE-Access-Requests, 10serviceops: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10Joe) I would tentatively grant full docker access to the people listed above, but to be revisited at a later time, maybe creating a... [17:15:44] (03PS1) 10Ahmon Dancy: Log rebuildLocalisationCache.php output at INFO level instead DEBUG [tools/scap] - 10https://gerrit.wikimedia.org/r/770977 [17:19:36] (03CR) 10Ahmon Dancy: [C: 03+2] Log rebuildLocalisationCache.php output at INFO level instead DEBUG [tools/scap] - 10https://gerrit.wikimedia.org/r/770977 (owner: 10Ahmon Dancy) [17:20:30] (03Merged) 10jenkins-bot: Log rebuildLocalisationCache.php output at INFO level instead DEBUG [tools/scap] - 10https://gerrit.wikimedia.org/r/770977 (owner: 10Ahmon Dancy) [17:26:59] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10Scap: Avoid rsyncing 2GB of L10N each day during train experiment week, if possible - https://phabricator.wikimedia.org/T303749 (10dancy) 05In progressโ†’03Stalled [17:29:33] 10Release-Engineering-Team (Seen), 10Scap: Automate rebuild l10n cache for Train - https://phabricator.wikimedia.org/T245187 (10dancy) [17:29:52] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Train: Do more time-consuming operations (l10n rebuild, sync-apaches) before a human gets involved - https://phabricator.wikimedia.org/T303764 (10dancy) [17:30:00] 10Release-Engineering-Team (Seen), 10Scap: Automate rebuild l10n cache for Train - https://phabricator.wikimedia.org/T245187 (10dancy) [17:30:30] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10Scap: Automate rebuild l10n cache for Train - https://phabricator.wikimedia.org/T245187 (10dancy) [17:30:54] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10Scap: Automate rebuild l10n cache for Train - https://phabricator.wikimedia.org/T245187 (10dancy) [17:30:56] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10Infrastructure-Foundations: Need a service account on deploy servers - https://phabricator.wikimedia.org/T303857 (10dancy) [17:38:45] RhinosF1: Anything I can help with anyway? :-) [17:40:11] James_F: nope, git kept telling me REL1_38 didn't exist until I tried fetch [17:41:13] you need to install the git mind reader plugin [17:41:27] Reedy: why can't that exist [17:41:53] Well, newest git has a 'did you mean?' for command typos. [17:41:55] Now I got a long list of changed sql files to review [17:41:56] Which I like. [17:42:46] RhinosF1: There's an even longer one coming in 1.39.0 โ€“ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/770972 [17:43:14] James_F: deleted sql files aren't too bad [17:43:21] It's mainly added or changed I care about [17:43:36] so I can work out how I'm going to crash the database this time [17:43:43] Oh, right, yeah. [17:44:04] Well, at least the patches are now auto-generated from the JSON in core, so longer-term this will be predictable. [17:44:20] It would be nice if it spit out you should run these files [17:44:52] I think that's a planned feature? Or at least a dreamed-about one. Ask Amir. [17:45:06] I thought you can already [17:45:20] Reedy: that runs maint scripts for updates though [17:45:42] Sounds like a bug [17:46:01] Reedy: it's intentional [17:46:06] Intentionally annoying [17:46:07] Too [17:46:24] Would be nice if it told me the order to run them in too but I'm really dreaming now [17:58:12] 10Release-Engineering-Team, 10SRE, 10SRE-Access-Requests, 10serviceops, 10Patch-For-Review: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10Majavah) Is there a reason this is limited to the releng team? T297673 says that all scap syncs will be build... [17:59:52] brennen: how would you do l10ncache without a human [18:00:03] It requires at least 1 wiki to have been promoted [18:02:53] RhinosF1: is this in reference to T245187? [18:02:54] T245187: Automate rebuild l10n cache for Train - https://phabricator.wikimedia.org/T245187 [18:03:07] brennen: yes [18:04:22] Make the script promote 1 specified wiki? ;) [18:05:11] i think that question is what the ticket is about. :) [18:06:49] (03PS1) 10Ahmon Dancy: scap clean: Don't do container stuff (fixed) [tools/scap] - 10https://gerrit.wikimedia.org/r/770992 [18:06:51] (03PS1) 10Ahmon Dancy: scap clean: Allow re-running even if branch dir doesn't exist [tools/scap] - 10https://gerrit.wikimedia.org/r/770993 [18:08:45] (03CR) 10Ahmon Dancy: [C: 03+2] scap clean: Don't do container stuff (fixed) [tools/scap] - 10https://gerrit.wikimedia.org/r/770992 (owner: 10Ahmon Dancy) [18:10:10] !log gitlab: finished migrating access for all existing people groups to direct project membership (T274461, T300935) [18:10:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:10:13] T300935: Investigate what's required to allow a user to fork or transfer a project to a group - https://phabricator.wikimedia.org/T300935 [18:10:14] (03Merged) 10jenkins-bot: scap clean: Don't do container stuff (fixed) [tools/scap] - 10https://gerrit.wikimedia.org/r/770992 (owner: 10Ahmon Dancy) [18:10:14] T274461: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 [18:13:22] 10GitLab (Auth & Access), 10Release-Engineering-Team, 10User-brennen: Investigate what's required to allow a user to fork or transfer a project to a group - https://phabricator.wikimedia.org/T300935 (10brennen) 05Openโ†’03In progress This //may// be resolved now that users are direct members of the project... [18:14:28] 10Release-Engineering-Team, 10SRE, 10SRE-Access-Requests, 10serviceops, 10Patch-For-Review: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10dancy) @Majavah The container build stuff happens under `sudo -u mwbuilder`, and `mwbuilder` does have permis... [18:15:30] 10Release-Engineering-Team, 10SRE, 10SRE-Access-Requests, 10serviceops, 10Patch-For-Review: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10dancy) To clarify further, releng needs direct access to dockerd (without having to sudo) while we're debuggi... [18:16:31] 10Release-Engineering-Team (Done by Feb 23 ๐ŸงŸ), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) [18:19:42] 10Continuous-Integration-Config: Some CI images still using composer 1.10.5 rather than 1.10.17 - https://phabricator.wikimedia.org/T268930 (10Reedy) 05Openโ†’03Declined [18:20:02] Project beta-update-databases-eqiad build #57294: 04FAILURE in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/57294/ [18:20:04] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10BlueSpice: BlueSpice related tests fails on gate-and-submit-1.31 in quibble-composer tests - https://phabricator.wikimedia.org/T235807 (10Reedy) 05Openโ†’03Declined [18:20:43] 10Continuous-Integration-Config: Update bundled composer to 2.2.x - https://phabricator.wikimedia.org/T303867 (10Reedy) [18:26:54] !log gitlab: removed most existing /people groups [18:26:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:31:01] (03PS9) 10Jeena Huneidi: gerrit: add Verified label [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) [18:32:15] (03CR) 10Jeena Huneidi: gerrit: add Verified label (032 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [18:39:28] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Ensure train-blockers.toolforge.org will find correct train blocker during train experiment week - https://phabricator.wikimedia.org/T303751 (10brennen) > api.php is used by the deploy-promote script, not sure about create_task.php. Ah, right. It looks like... [18:50:56] 10Release-Engineering-Team (Onboarding ๐Ÿš€): Delete the wmf/1.37.0-wmf.* branches once we've fully deployed to 1.38.0-wmf.1 and finished with them - https://phabricator.wikimedia.org/T291501 (10brennen) I think we worked out a script for this during a yak-shaving week. Looking for that. [19:01:56] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828 (10brennen) There's [[https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/release/+/refs/heads/master/delete-wmf-branches/|delete-wmf-branches]] from T244368, b... [19:04:18] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828 (10brennen) @dancy or @thcipriani does this ring any bells I'm forgetting about? [19:05:18] (03PS1) 10Jforrester: dockerfiles: [composer-scratch] Upgrade composer to 2.2.8 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/771004 [19:05:44] (03PS10) 10Jeena Huneidi: gerrit: add Verified label [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) [19:06:41] (03CR) 10jerkins-bot: [V: 04-1] gerrit: add Verified label [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [19:07:46] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: [composer-scratch] Upgrade composer to 2.2.8 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/771004 (owner: 10Jforrester) [19:08:41] (03PS2) 10Jforrester: dockerfiles: [composer-scratch] Upgrade composer to 2.2.8 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/771004 [19:19:02] (03CR) 10Jeena Huneidi: gerrit: add Verified label (032 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [19:21:44] (03PS11) 10Jeena Huneidi: gerrit: add Verified label [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) [19:30:56] Yippee, build fixed! [19:30:57] Project beta-update-databases-eqiad build #57295: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/57295/ [19:41:54] (03CR) 10Ahmon Dancy: "Lookin good." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [19:45:03] (03CR) 10Jeena Huneidi: gerrit: add Verified label (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [19:56:45] 10Release-Engineering-Team: multiversion/MWScript.php: Allow specifying a specific version of code to run - https://phabricator.wikimedia.org/T303878 (10dancy) [19:56:57] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): multiversion/MWScript.php: Allow specifying a specific version of code to run - https://phabricator.wikimedia.org/T303878 (10dancy) [19:57:02] 10Project-Admins: Create project tag for <#Equity-Landscape> - https://phabricator.wikimedia.org/T303879 (10JAnstee_WMF) [19:58:57] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10Patch-For-Review: multiversion/MWScript.php: Allow specifying a specific version of code to run - https://phabricator.wikimedia.org/T303878 (10dancy) [19:58:59] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10Scap: Automate rebuild l10n cache for Train - https://phabricator.wikimedia.org/T245187 (10dancy) [20:03:54] 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828 (10dancy) https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/683934 is where we modified branch.py to create a tag named after the branch before deleting the branch. [20:51:04] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T300202 (10jeena) [20:57:48] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T300202 (10jeena) [21:03:46] (03CR) 10Dduvall: [C: 04-1] gerrit: add Verified label (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [21:06:41] (03CR) 10Dduvall: [C: 04-1] gerrit: add Verified label (032 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [21:29:42] (03PS2) 10Hashar: dockerfiles: castor hostname from CASTOR_HOST env [integration/config] - 10https://gerrit.wikimedia.org/r/770921 (https://phabricator.wikimedia.org/T216244) [21:30:00] rspec inspired bash unit testing [21:30:07] i should have written it directly in python [21:31:10] I need a bash package manager [21:31:51] (03CR) 10Hashar: [C: 04-1] "I have added some tests for jjb/castor-load-sync.bash to verify CASTOR_HOST is taken in account but they are in pure bash and are not invo" [integration/config] - 10https://gerrit.wikimedia.org/r/770921 (https://phabricator.wikimedia.org/T216244) (owner: 10Hashar) [21:36:00] hashar: you thought that was a joke but ... https://bpkg.sh/ https://www.basher.it/ [21:36:40] AH [21:37:47] maybe one day we will have a universal package manager which is the sum of all the others [21:38:24] will look at those thx! [22:17:52] (03CR) 10Ahmon Dancy: "Please consider moving the common files into a new files/common directory." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [22:31:04] (03PS12) 10Jeena Huneidi: gerrit: add Verified label [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) [22:32:52] (03PS13) 10Jeena Huneidi: gerrit: add Verified label [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) [22:34:01] (03CR) 10Jeena Huneidi: gerrit: add Verified label (036 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [22:40:02] (03CR) 10Dduvall: [C: 03+2] gerrit: add Verified label [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [22:40:19] ๐Ÿฅณ [22:41:08] (03Merged) 10jenkins-bot: gerrit: add Verified label [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [23:02:23] 10Continuous-Integration-Config, 10Codex, 10User-DannyS712: Public codex code coverage report - https://phabricator.wikimedia.org/T303899 (10DannyS712) [23:02:52] (03CR) 10Ahmon Dancy: "Awesome!" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/770451 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi)