[00:05:39] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [02:07:57] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [07:44:18] 10Release-Engineering-Team (Doing), 10MediaWiki-Core-Tests, 10MediaWiki-ResourceLoader, 10MW-1.38-notes (1.38.0-wmf.13; 2021-12-13), 10Performance-Team (Radar): Move bundlesize test from npm script to MediaWikiIntegrationTest - https://phabricator.wikimedia.org/T255149 (10kostajh) >>! In T255149#7566551,... [07:57:44] 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10User-zeljkofilipin: specFileRetries does not seem to apply in extension wdio.conf.js - https://phabricator.wikimedia.org/T296826 (10kostajh) >>! In T296826#7541133, @zeljkofilipin wrote: > @Krinkle, @Catrope do you agree to add @kostajh to the packa... [08:21:37] jayme: jelto: I am rebuilding the helm-linter image [08:21:42] (03CR) 10Hashar: [C: 03+2] helm-linter: remove helm2 from Docker image [integration/config] - 10https://gerrit.wikimedia.org/r/747147 (https://phabricator.wikimedia.org/T251305) (owner: 10Jelto) [08:23:36] (03Merged) 10jenkins-bot: helm-linter: remove helm2 from Docker image [integration/config] - 10https://gerrit.wikimedia.org/r/747147 (https://phabricator.wikimedia.org/T251305) (owner: 10Jelto) [08:29:44] (03CR) 10Hashar: "Successfully published image docker-registry.discovery.wmnet/releng/helm-linter:0.3.0" [integration/config] - 10https://gerrit.wikimedia.org/r/747147 (https://phabricator.wikimedia.org/T251305) (owner: 10Jelto) [08:30:18] (03CR) 10Hashar: [C: 03+2] jjb: update helm-linter job to releng/helm-linter:0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/747148 (https://phabricator.wikimedia.org/T251305) (owner: 10Jelto) [08:31:01] !log Update helm-lint job https://gerrit.wikimedia.org/r/c/integration/config/+/747148 T251305 [08:31:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:31:05] T251305: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 [08:31:07] jayme: jelto: done [08:32:18] (03Merged) 10jenkins-bot: jjb: update helm-linter job to releng/helm-linter:0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/747148 (https://phabricator.wikimedia.org/T251305) (owner: 10Jelto) [08:46:51] hashar: thanks for the quick rebuild. I'll proceed with the change in deployment-charts CI jobs asap :) [09:08:31] jelto: I did a recheck of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/746864 , looks like it is now magically passing when it failed previously :D [11:00:06] 10Continuous-Integration-Config, 10Quibble: Switch all Quibble Selenium jobs to use apache - https://phabricator.wikimedia.org/T285649 (10kostajh) Another thing that will reduce the risk of this change is to roll out {T296826} to extensions/skins first; this will retry a failed test in the event of a failure. [11:47:51] (03PS5) 10Kosta Harlan: BrowserTests: Option to run npm install in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/738061 (https://phabricator.wikimedia.org/T226869) [14:39:29] Hello RelEng, will there be a train deploy during the current window? [14:43:51] 10Continuous-Integration-Config, 10Quibble: Switch all Quibble Selenium jobs to use apache - https://phabricator.wikimedia.org/T285649 (10awight) >>! In T285649#7563424, @kostajh wrote: > Sounds good. Let me know how I can help. @awight will you be around? Yes, I'll be around this week and next. Great to see... [15:33:25] 10GitLab (Auth & Access): Create subgroup for 'wikisp' - https://phabricator.wikimedia.org/T296110 (10brennen) I have created https://gitlab.wikimedia.org/people/volunteer-group-wikimedia-small-projects - please let me know at least one Wikimedia developer account that has been logged into GitLab and should have... [15:59:05] RoanKattouw: I will roll it this evening, I forgot to announce it this time [15:59:27] Ok great! [16:13:57] npm is having some issues, in case anyone is seeing failing CI builds https://status.npmjs.org/incidents/7klw65jd5f5r [16:17:59] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Run standalone-only jobs apart [integration/config] - 10https://gerrit.wikimedia.org/r/747537 [16:22:32] Lucas_WMDE: AWS is struggling. Might be why. [16:23:21] ack, just wanted to leave a note here [16:25:13] thank you Lucas_WMDE [16:26:18] !log npm has some issue going on which affects: www.npmjs.com website, Package installation, Package publishing, Package search, and Security Audit | Upstream status https://status.npmjs.org/incidents/7klw65jd5f5r [16:26:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:29:30] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/WikiLambda] Run standalone-only jobs apart [integration/config] - 10https://gerrit.wikimedia.org/r/747537 (owner: 10Jforrester) [16:30:55] (03PS3) 10Jforrester: Zuul: Add Winston Sung (Fandom ZH community) to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/742278 (owner: 10GergΕ‘ Tisza) [16:31:00] (03PS4) 10Jforrester: Zuul: Add Winston Sung (Fandom ZH community) to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/742278 (owner: 10GergΕ‘ Tisza) [16:31:02] (03CR) 10Jforrester: [C: 03+2] Zuul: Add Winston Sung (Fandom ZH community) to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/742278 (owner: 10GergΕ‘ Tisza) [16:31:26] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Run standalone-only jobs apart [integration/config] - 10https://gerrit.wikimedia.org/r/747537 (owner: 10Jforrester) [16:31:52] !log Zuul: [mediawiki/extensions/WikiLambda] Run standalone-only jobs apart [16:31:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:32:58] (03Merged) 10jenkins-bot: Zuul: Add Winston Sung (Fandom ZH community) to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/742278 (owner: 10GergΕ‘ Tisza) [16:33:20] !log Zuul: Add Winston Sung (Fandom ZH community) to CI allowlist [16:33:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:34:48] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10hashar) [16:40:02] (03PS1) 10Ladsgroup: Start of deploy_security.py [tools/release] - 10https://gerrit.wikimedia.org/r/747544 [16:42:39] (03CR) 10Thcipriani: [C: 03+1] "LGTM, I'll defer to @dancy as keeper of our deployment knowledge, if this is a good temporary spot" [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [16:43:05] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10hashar) The two small blockers T297529 and T297421 are solved. T297517 is actively being tracked. The issue was made more prominent last w... [16:52:13] (03CR) 10SBassett: Start of deploy_security.py (035 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [16:58:27] 10Continuous-Integration-Config, 10Quibble: Switch all Quibble Selenium jobs to use apache - https://phabricator.wikimedia.org/T285649 (10hashar) I had a side track with `docker-pkg` (`seed_images` must be set in the config [[ https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-pkg/+/747060 | Gerr... [16:59:45] (03CR) 10Legoktm: "Awesome, this is great!" [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [17:05:38] (03CR) 10Dduvall: [C: 03+2] macros: Use numeric gid when creating a user [blubber] - 10https://gerrit.wikimedia.org/r/740282 (owner: 10BryanDavis) [17:05:57] (03CR) 10Dduvall: [C: 03+2] "Sorry for the long delay! I lost track of this." [blubber] - 10https://gerrit.wikimedia.org/r/740282 (owner: 10BryanDavis) [17:06:28] (03CR) 10SBassett: Start of deploy_security.py (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [17:09:36] (03CR) 10jerkins-bot: [V: 04-1] macros: Use numeric gid when creating a user [blubber] - 10https://gerrit.wikimedia.org/r/740282 (owner: 10BryanDavis) [17:16:08] 10MediaWiki-Releasing, 10I18n: Split l10n/i18n updates from release patches - https://phabricator.wikimedia.org/T297807 (10Reedy) [17:19:48] (03CR) 10Ahmon Dancy: [C: 04-1] Start of deploy_security.py (0315 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [17:22:05] (03CR) 10Ahmon Dancy: [C: 04-1] Start of deploy_security.py (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [17:28:34] (03CR) 10SBassett: Start of deploy_security.py (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [17:28:37] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release Pipeline, 10serviceops: PipelineLib deploy is broken and needs refactoring to use helm3 - https://phabricator.wikimedia.org/T297809 (10dduvall) [17:29:40] (03CR) 10Dduvall: [C: 03+2] "It appears that PipelineLib's deploy functionality is currently broken which is holding up this merge. See https://phabricator.wikimedia.o" [blubber] - 10https://gerrit.wikimedia.org/r/740282 (owner: 10BryanDavis) [17:29:42] (03CR) 10Ahmon Dancy: [C: 04-1] Start of deploy_security.py (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [17:30:49] (03CR) 10SBassett: Start of deploy_security.py (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [17:32:17] (03CR) 10Ahmon Dancy: [C: 04-1] Start of deploy_security.py (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [17:32:19] (03CR) 10SBassett: Start of deploy_security.py (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [17:32:31] (03CR) 10jerkins-bot: [V: 04-1] macros: Use numeric gid when creating a user [blubber] - 10https://gerrit.wikimedia.org/r/740282 (owner: 10BryanDavis) [17:35:37] (03CR) 10Ladsgroup: Start of deploy_security.py (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [17:48:04] (03CR) 10Ahmon Dancy: [C: 04-1] Start of deploy_security.py (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [18:27:31] (03PS6) 10Jeena Huneidi: scap backport: validate changes [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) [18:29:10] (03CR) 10jerkins-bot: [V: 04-1] scap backport: validate changes [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) (owner: 10Jeena Huneidi) [18:35:17] (03PS1) 10Jeena Huneidi: Update helm commands for helm 3 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/747567 (https://phabricator.wikimedia.org/T297809) [18:37:44] (03CR) 10Jeena Huneidi: "It looks like I need to change the command to "helm3" instead of "helm"" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/747567 (https://phabricator.wikimedia.org/T297809) (owner: 10Jeena Huneidi) [18:40:52] (03PS2) 10Jeena Huneidi: Update helm commands for helm 3 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/747567 (https://phabricator.wikimedia.org/T297809) [18:50:30] (03PS3) 10Jeena Huneidi: Update helm commands for helm 3 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/747567 (https://phabricator.wikimedia.org/T297809) [19:07:54] (03PS4) 10Jeena Huneidi: Update helm commands for helm 3 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/747567 (https://phabricator.wikimedia.org/T297809) [19:12:00] (03PS5) 10Jeena Huneidi: Update helm commands for helm 3 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/747567 (https://phabricator.wikimedia.org/T297809) [19:13:59] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-team (Design Systems Team FY2021-22 Kanban Board), 10Patch-For-Review: Deploy current main branch's docs site to doc.wikimedia.org - https://phabricator.wikimedia.org/T293704 (10Mooeypoo) Not only is netlify self hostable, it directly connects to a... [19:31:16] thcipriani: if you are the keeper of 'restricted' could you please approve https://phabricator.wikimedia.org/T297621? (Or if anyone else here can advise I welcome ideas for getting it unstuck) [19:31:17] thanks! [19:38:06] 10MediaWiki-Releasing, 10Security: Write and send release announcements for MediaWiki 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292230 (10Reedy) [19:38:14] 10MediaWiki-Releasing, 10Security: Write and send release announcements for MediaWiki 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292230 (10Reedy) [19:42:46] 10MediaWiki-Releasing, 10Security: Update onwiki news and Module:Version - https://phabricator.wikimedia.org/T292232 (10Reedy) 05Openβ†’03Resolved p:05Triageβ†’03Medium a:03Reedy [19:42:50] 10MediaWiki-Releasing, 10Security: Update onwiki news and Module:Version - https://phabricator.wikimedia.org/T292232 (10Reedy) [19:43:25] 10MediaWiki-Releasing, 10Security: Write and send pre-release announcements for MediaWiki 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292229 (10Reedy) [19:43:29] 10MediaWiki-Releasing, 10Security: Update Wikidata Q83 - https://phabricator.wikimedia.org/T292234 (10Reedy) [19:47:31] andrewbogott: I might be keeper of restricted: I'll take a look :) [19:47:39] thank you! [19:48:05] Komla is the new developer advocate, occupying joakino's old seat [19:51:39] (03CR) 10Hashar: [C: 03+2] "There are some traces and that apparently comes from 2013 :) Anyway it is 100% behind a feature flag so I am more than happy to +2 it, th" [tools/scap] - 10https://gerrit.wikimedia.org/r/746941 (https://phabricator.wikimedia.org/T297326) (owner: 10Ahmon Dancy) [19:53:12] (03Merged) 10jenkins-bot: Add rsync_cdbs configuration parameter [tools/scap] - 10https://gerrit.wikimedia.org/r/746941 (https://phabricator.wikimedia.org/T297326) (owner: 10Ahmon Dancy) [19:57:53] 10MediaWiki-Releasing, 10Security: Tracking bug for MediaWiki 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292227 (10Reedy) [19:58:09] 10MediaWiki-Releasing, 10Security: Tag 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292231 (10Reedy) p:05Triageβ†’03Medium [19:58:15] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292226 (10Reedy) [19:58:21] 10MediaWiki-Releasing, 10Security: Update HISTORY in master after 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292235 (10Reedy) p:05Triageβ†’03Medium [19:58:31] 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292233 (10Reedy) p:05Triageβ†’03Medium [20:10:44] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap: Stop trying to avoid rsyncing l10n CDB files - https://phabricator.wikimedia.org/T297326 (10Krinkle) [20:18:01] 10MediaWiki-Releasing, 10MediaWiki-General: Composer command to just validate RELEASE-NOTES - https://phabricator.wikimedia.org/T297826 (10Reedy) [20:19:58] 10MediaWiki-Releasing, 10MediaWiki-General: Composer command to just validate RELEASE-NOTES - https://phabricator.wikimedia.org/T297826 (10Reedy) p:05Triageβ†’03Low [20:21:59] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10hashar) [20:24:42] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [20:27:01] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release Pipeline, 10serviceops, 10Patch-For-Review: PipelineLib deploy is broken and needs refactoring to use helm3 - https://phabricator.wikimedia.org/T297809 (10Jdforrester-WMF) I guess [[https://codesearch.wmcloud.org/deployed/?q=deploy%3A&i=nope&files=... [20:27:44] (03PS1) 10Ahmon Dancy: release-scripts/scaps-installed-in-beta: Add -v flag [tools/scap] - 10https://gerrit.wikimedia.org/r/747605 [20:29:31] 10MediaWiki-Releasing, 10Security: Tag 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292231 (10Reedy) 05Openβ†’03In progress a:03Reedy [20:33:46] (03CR) 10Ahmon Dancy: [C: 03+2] release-scripts/scaps-installed-in-beta: Add -v flag [tools/scap] - 10https://gerrit.wikimedia.org/r/747605 (owner: 10Ahmon Dancy) [20:35:01] (03Merged) 10jenkins-bot: release-scripts/scaps-installed-in-beta: Add -v flag [tools/scap] - 10https://gerrit.wikimedia.org/r/747605 (owner: 10Ahmon Dancy) [20:35:04] * dancy twiddles thumbs waiting for zuul [20:36:52] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10hashar) [20:43:32] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [20:48:50] 10MediaWiki-Releasing, 10Security: Tag 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292231 (10Reedy) 05In progressβ†’03Resolved [20:48:55] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292226 (10Reedy) [20:51:39] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [20:53:47] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10hashar) MediaWiki has been rolled back from group 1 wiki since newcomers were unable to login on cawiki: T297827 [20:58:34] 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292233 (10Reedy) 05Openβ†’03Resolved a:03Reedy [20:58:40] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292226 (10Reedy) [21:01:51] (03PS1) 10Reedy: makesecuritytasks.py: Parameterise versions and phab token [tools/release] - 10https://gerrit.wikimedia.org/r/747609 [21:02:39] (03CR) 10jerkins-bot: [V: 04-1] makesecuritytasks.py: Parameterise versions and phab token [tools/release] - 10https://gerrit.wikimedia.org/r/747609 (owner: 10Reedy) [21:03:59] (03PS2) 10Reedy: makesecuritytasks.py: Parameterise versions and phab token [tools/release] - 10https://gerrit.wikimedia.org/r/747609 [21:04:41] (03CR) 10jerkins-bot: [V: 04-1] makesecuritytasks.py: Parameterise versions and phab token [tools/release] - 10https://gerrit.wikimedia.org/r/747609 (owner: 10Reedy) [21:05:55] (03PS3) 10Reedy: makesecuritytasks.py: Parameterise versions and phab token [tools/release] - 10https://gerrit.wikimedia.org/r/747609 [21:10:06] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292226 (10Reedy) [21:10:10] 10MediaWiki-Releasing, 10Patch-For-Review, 10Security: Update HISTORY in master after 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292235 (10Reedy) 05Openβ†’03In progress [21:10:19] (03CR) 10Reedy: [C: 03+2] makesecuritytasks.py: Parameterise versions and phab token [tools/release] - 10https://gerrit.wikimedia.org/r/747609 (owner: 10Reedy) [21:11:01] (03Merged) 10jenkins-bot: makesecuritytasks.py: Parameterise versions and phab token [tools/release] - 10https://gerrit.wikimedia.org/r/747609 (owner: 10Reedy) [21:28:23] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292226 (10Reedy) [21:28:33] 10MediaWiki-Releasing, 10Patch-For-Review, 10Security: Update HISTORY in master after 1.35.5/1.36.3/1.37.1 - https://phabricator.wikimedia.org/T292235 (10Reedy) 05In progressβ†’03Resolved a:03Reedy [21:40:24] (03PS2) 10Ladsgroup: Start of deploy_security.py [tools/release] - 10https://gerrit.wikimedia.org/r/747544 [21:40:32] (03CR) 10Ladsgroup: Start of deploy_security.py (0322 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/747544 (owner: 10Ladsgroup) [21:52:29] (03PS6) 10Jeena Huneidi: Update helm commands for helm 3 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/747567 (https://phabricator.wikimedia.org/T297809) [22:02:00] (03PS1) 10Kosta Harlan: Add phpcs stage [integration/quibble] - 10https://gerrit.wikimedia.org/r/747617 (https://phabricator.wikimedia.org/T297561) [22:02:33] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Quibble, 10Patch-For-Review: Run linters before starting longer running jobs - https://phabricator.wikimedia.org/T297561 (10kostajh) > steps 2-3 could be done entirely in Quibble, but having as a service seems useful for other tool... [22:02:52] (03CR) 10Kosta Harlan: "I could use help figuring out how to fix the unit test complaining about requests not being found." [integration/quibble] - 10https://gerrit.wikimedia.org/r/747617 (https://phabricator.wikimedia.org/T297561) (owner: 10Kosta Harlan) [22:05:23] (03CR) 10jerkins-bot: [V: 04-1] Add phpcs stage [integration/quibble] - 10https://gerrit.wikimedia.org/r/747617 (https://phabricator.wikimedia.org/T297561) (owner: 10Kosta Harlan) [22:07:49] https://usercontent.irccloud-cdn.com/file/wZ129qPi/image.png [22:07:53] 😭 [22:08:14] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release Pipeline, 10serviceops, 10Patch-For-Review: PipelineLib deploy is broken and needs refactoring to use helm3 - https://phabricator.wikimedia.org/T297809 (10jeena) It looks like jenkins needs permissions to create secrets in order to do a helm releas... [22:08:20] :( Amir1 [22:08:52] kicked one directly [22:08:53] https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/33024/console [22:47:00] (03CR) 10Hashar: [C: 03+1] "C. Scott and I talked about this a few times, it good for me but I will let C. Scott give the go since somethings might break here and the" [integration/config] - 10https://gerrit.wikimedia.org/r/742939 (https://phabricator.wikimedia.org/T284611) (owner: 10Kosta Harlan) [22:47:07] (03CR) 10Hashar: [C: 03+1] "C. Scott and I talked about this a few times, it good for me but I will let C. Scott give the go since somethings might break here and the" [integration/config] - 10https://gerrit.wikimedia.org/r/745930 (https://phabricator.wikimedia.org/T94793) (owner: 10C. Scott Ananian) [23:21:39] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [23:28:19] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [23:36:39] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [23:45:02] !log Updating scap in beta to test https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/746941 T297326 [23:45:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:45:04] T297326: Stop trying to avoid rsyncing l10n CDB files - https://phabricator.wikimedia.org/T297326 [23:52:26] 10Beta-Cluster-Infrastructure: deployment-echostore01 periodically going offline - https://phabricator.wikimedia.org/T296013 (10dancy) On this note, there are some enormous files on this host that are blocking operations: ` root@deployment-echostore01:~# ls -lh /var/log/{syslog.1,daemon.log.1} -rw-r----- 1 root... [23:58:12] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10serviceops, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10Legoktm) On all physical hosts now: ` legoktm@cumin1001:~$ sudo cumin A:all-mw 'php -m | grep yaml' 352 ho...