[00:47:49] (03PS1) 10BryanDavis: Convert to Manifest v3 [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) [00:48:08] (03CR) 10CI reject: [V:04-1] Convert to Manifest v3 [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis) [00:57:41] (03PS2) 10BryanDavis: Convert to Manifest v3 [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) [01:11:02] (03PS3) 10BryanDavis: Convert to Manifest v3 [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) [01:12:57] (03CR) 10BryanDavis: Convert to Manifest v3 (033 comments) [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis) [01:16:06] (03CR) 10BryanDavis: "Seems to be working with my macOS Chrome 131.0.6778.205 and Firefox 133.0.3. I would appreciate more local testing results from folks to i" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis) [01:18:01] 06Release-Engineering-Team, 10WikimediaDebug, 13Patch-For-Review: Support WebExtensions Manifest v3 - https://phabricator.wikimedia.org/T312694#10439460 (10bd808) Instructions for testing a local build of the extension can be found in the README: https://gerrit.wikimedia.org/r/plugins/gitiles/performance/Wik... [08:18:04] 10GitLab (Pipeline Services Migration🐀), 06collaboration-services, 13Patch-For-Review: move os-reports.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350794#10439828 (10JMeybohm) >>! In T350794#10438912, @Arnoldokoth wrote: > The only other reason I would have to justify a separate cronjob... [08:45:37] 06Gerrit-Privilege-Requests, 10Infrastructure Security, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10439976 (10WMDE-leszek) > I think we might need some coordination how to handle WMDE staff offboarding in... [08:58:25] 10GitLab: Gitlab MFA reset for Esuvat Bomani - https://phabricator.wikimedia.org/T383123#10439981 (10EBomani-WMF) Absolutely, thanks for flagging that. Added TOTP 2FA auth to the EBomani account on Phabricator since that's the one I use. Additionally, Is there a wa... [10:05:26] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Austin Varshney - https://phabricator.wikimedia.org/T383205 (10austinvarshney) 03NEW [10:12:57] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/CollapsibleVector] Add Vector phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1108830 (owner: 10Fomafix) [10:14:13] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CollapsibleVector] Add Vector phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1108830 (owner: 10Fomafix) [10:25:52] (03PS1) 10Hashar: Remove php-security scanning [integration/config] - 10https://gerrit.wikimedia.org/r/1109036 (https://phabricator.wikimedia.org/T180278) [10:28:00] (03CR) 10Hashar: [C:03+2] "I have deleted all the jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/1109036 (https://phabricator.wikimedia.org/T180278) (owner: 10Hashar) [10:29:24] (03Merged) 10jenkins-bot: Remove php-security scanning [integration/config] - 10https://gerrit.wikimedia.org/r/1109036 (https://phabricator.wikimedia.org/T180278) (owner: 10Hashar) [10:30:48] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Security-Team, 13Patch-For-Review, 07Security: Disable php-composer-security-docker emails to security-admin-feed@lists - https://phabricator.wikimedia.org/T309655#10440112 (10hashar) 05Openβ†’03Resolved a:03hashar Eventually I have found th... [10:31:45] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade): Delete unused Jenkins jobs - https://phabricator.wikimedia.org/T383134#10440122 (10hashar) [10:32:23] 10Continuous-Integration-Infrastructure (Zuul upgrade): Identify the different classes of jobs - https://phabricator.wikimedia.org/T383067#10440125 (10hashar) [10:34:42] 10Continuous-Integration-Config, 10Librarization, 06Security-Team, 07Composer, 07Security: Expand our usage of FriendsOfPHP/security-advisories - https://phabricator.wikimedia.org/T180278#10440139 (10hashar) 05Openβ†’03Declined After discussion on T309655 I have deleted the Jenkins jobs. The notifi... [10:35:24] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade): Delete unused Jenkins jobs - https://phabricator.wikimedia.org/T383134#10440144 (10hashar) [10:36:44] 10Continuous-Integration-Infrastructure (Zuul upgrade): Identify the different classes of jobs - https://phabricator.wikimedia.org/T383067#10440146 (10hashar) [11:14:01] (03open) 10aokoth: projects: add sre/miscweb/os-reports [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/103 (https://phabricator.wikimedia.org/T350794) [11:14:09] (03update) 10aokoth: projects: add sre/miscweb/os-reports [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/103 (https://phabricator.wikimedia.org/T350794) [11:25:06] 06Release-Engineering-Team, 06collaboration-services: Low disk space: doc1003 / doc2002 - https://phabricator.wikimedia.org/T382610#10440281 (10hashar) Thank you @Arnoldokoth ! [13:20:35] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10440725 (10akosiaris) [Docs for our version](https://www.envoyproxy.io/docs/envoy/v1.23.12/configuration/http/http_filters/router_f... [13:31:41] 06Project-Admins: Create project tag for emi - https://phabricator.wikimedia.org/T383219 (10Firmin243) 03NEW [13:32:48] (03PS1) 10Hashar: jjb: remove unused bitu-pipeline-rehearse job [integration/config] - 10https://gerrit.wikimedia.org/r/1109063 [13:36:08] (03CR) 10Hashar: [C:03+2] jjb: remove unused bitu-pipeline-rehearse job [integration/config] - 10https://gerrit.wikimedia.org/r/1109063 (owner: 10Hashar) [13:37:27] (03Merged) 10jenkins-bot: jjb: remove unused bitu-pipeline-rehearse job [integration/config] - 10https://gerrit.wikimedia.org/r/1109063 (owner: 10Hashar) [14:08:46] feels like CI builds are getting more ECONNRESET failures from npmjs.org than usual today :S [14:09:02] (might just be npm’s / microsoft’s fault though) [14:14:49] Lucas_WMDE: I have one patch that absolutely will not pass a selenium build [14:16:31] I also don't really understand 20 minutes (*2) of Selenium tests on every patch 🫠 [14:17:01] 06Project-Admins: Create project tag for emi - https://phabricator.wikimedia.org/T383219#10441003 (10Aklapper) Hi @Firmin243, thanks for taking the time to report this! https://github.com/kaliacad/mostvisitedarticle/issues exists, do you plan to switch that off once there is a project in Wikimedia Phabricator?... [14:17:48] 06Project-Admins: Create project tag for emi - https://phabricator.wikimedia.org/T383219#10441007 (10Aklapper) a:05Firmin243β†’03None [14:23:08] (03PS1) 10Hashar: Archive operations/software/censorship-monitoring [integration/config] - 10https://gerrit.wikimedia.org/r/1109078 (https://phabricator.wikimedia.org/T383229) [14:24:06] (03PS2) 10Hashar: Archive operations/software/censorship-monitoring [integration/config] - 10https://gerrit.wikimedia.org/r/1109078 (https://phabricator.wikimedia.org/T383229) [14:24:36] (03CR) 10Hashar: [C:03+2] Archive operations/software/censorship-monitoring [integration/config] - 10https://gerrit.wikimedia.org/r/1109078 (https://phabricator.wikimedia.org/T383229) (owner: 10Hashar) [14:25:55] (03Merged) 10jenkins-bot: Archive operations/software/censorship-monitoring [integration/config] - 10https://gerrit.wikimedia.org/r/1109078 (https://phabricator.wikimedia.org/T383229) (owner: 10Hashar) [14:27:32] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Update to Phorge upstream 2024.35 release - https://phabricator.wikimedia.org/T370266#10441056 (10Aklapper) >>! In T370266#10426915, @Marostegui wrote: > Are there any significant schema changes on this upgrade? `phabricator_file.file_attachment... [14:29:09] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Update to Phorge upstream 2024.35 release - https://phabricator.wikimedia.org/T370266#10441063 (10Marostegui) >>! In T370266#10441056, @Aklapper wrote: >>>! In T370266#10426915, @Marostegui wrote: >> Are there any significant schema changes on thi... [15:12:54] filed T383237 for those ECONNRESET errors from npm mentioned earlier btw [15:12:55] T383237: Many CI builds failing with ECONNRESET from npmjs.org during install:bridge steps - https://phabricator.wikimedia.org/T383237 [15:17:43] ^ @hashar [15:20:31] 10Release-Engineering-Team (Priority Backlog πŸ“₯): Experiment with WMCS as a k8s provider for gitlab-cloud-runner cluster - https://phabricator.wikimedia.org/T353356#10441316 (10joanna_borun) [15:29:35] 10Diffusion, 10Phabricator, 06cloud-services-team, 10Toolforge, and 2 others: Document diffusion->github mirroring to https://github.com/toolforge/ on wikitech - https://phabricator.wikimedia.org/T361859#10441361 (10joanna_borun) p:05Triageβ†’03Medium [15:36:21] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS: prometheus-openstack-stale-puppet-certs crashing on deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T383153#10441417 (10Andrew) 05Openβ†’03Resolved [15:38:19] Lucas_WMDE: thanks for the task [15:38:20] 15:34:30 npm error network Invalid response body while trying to fetch https://registry.npmjs.org/@typescript-eslint%2fvisitor-keys: aborted [15:38:21] :/ [15:38:25] who knows! [15:39:05] yeah :/ [15:39:09] I have no idea how to investigate it either [15:39:12] (03PS1) 10Robert Vogel: Add dependency to Extension:ContentStabilization [integration/config] - 10https://gerrit.wikimedia.org/r/1109103 [15:42:57] (03PS1) 10Hashar: zuul: remove puppet-module template [integration/config] - 10https://gerrit.wikimedia.org/r/1109107 (https://phabricator.wikimedia.org/T230206) [15:43:45] (03CR) 10Hashar: [C:03+2] Add dependency to Extension:ContentStabilization [integration/config] - 10https://gerrit.wikimedia.org/r/1109103 (owner: 10Robert Vogel) [15:43:51] (03CR) 10Hashar: [C:03+2] zuul: remove puppet-module template [integration/config] - 10https://gerrit.wikimedia.org/r/1109107 (https://phabricator.wikimedia.org/T230206) (owner: 10Hashar) [15:45:09] (03Merged) 10jenkins-bot: Add dependency to Extension:ContentStabilization [integration/config] - 10https://gerrit.wikimedia.org/r/1109103 (owner: 10Robert Vogel) [15:45:13] (03Merged) 10jenkins-bot: zuul: remove puppet-module template [integration/config] - 10https://gerrit.wikimedia.org/r/1109107 (https://phabricator.wikimedia.org/T230206) (owner: 10Hashar) [15:55:16] ValueError: invalid literal for int() with base 10: '${BUILD_TIMEOUT}' [15:55:18] lalala [15:55:51] (03CR) 10Hashar: [C:03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1109103 (owner: 10Robert Vogel) [16:22:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing 😎): Zuul/Jenkins: Investigate tracking what tests run for a given code change - https://phabricator.wikimedia.org/T383243 (10thcipriani) 03NEW [16:28:37] 06Project-Admins: Create project tag for emi - https://phabricator.wikimedia.org/T383219#10441666 (10BamLifa) @Aklapper this is a plan to move to Wikimedia tools: project management and hosting. Thank you so much for your comments. I'm not sure I understand the first unrelated comments. [16:29:53] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing 😎): Zuul/Jenkins: Investigate fan out for stages of quibble test run - https://phabricator.wikimedia.org/T383244 (10thcipriani) 03NEW [16:31:12] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing 😎): Zuul/Jenkins: Investigate fan out for stages of quibble test run - https://phabricator.wikimedia.org/T383244#10441688 (10thcipriani) Tagging in @ArthurTaylor and @kostajh since this idea seems simi... [17:11:31] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07Epic, 10Release Pipeline (Blubber): Deprecate Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289#10441865 (10dduvall) 05In progressβ†’03Resolved [17:55:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing 😎): Zuul/Jenkins: Investigate tracking what tests run for a given code change - https://phabricator.wikimedia.org/T383243#10442055 (10thcipriani) [17:59:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing 😎): Zuul/Jenkins: Investigate tracking what tests run for a given code change - https://phabricator.wikimedia.org/T383243#10442066 (10thcipriani) a:03dduvall [17:59:18] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing 😎): Zuul/Jenkins: Investigate tracking what tests run for a given code change - https://phabricator.wikimedia.org/T383243#10442069 (10thcipriani) p:05Triageβ†’03Medium [18:07:55] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing 😎): Jenkins: In the integration project in WMCS, test out larger workers for CI - https://phabricator.wikimedia.org/T383254 (10thcipriani) 03NEW [18:09:22] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing 😎): Jenkins: In the integration project in WMCS, test out larger workers for CI - https://phabricator.wikimedia.org/T383254#10442142 (10thcipriani) p:05Triageβ†’03Medium a:03brennen [18:29:19] (03CR) 10Mainframe98: [C:03+1] "Tested in Chrome 131 and Firefox 128.5.0 (ESR) on Debian by enabling the extension by setting readonly-mode to true and opening the editor" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis) [18:45:52] (03open) 10taavi: Bump @vue/vue3-jest to a version compatible with our Jest [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/53 [18:46:02] (03merge) 10taavi: Bump @vue/vue3-jest to a version compatible with our Jest [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/53 [18:48:35] (03open) 10taavi: Bump ts-jest to a version compatible with our Jest [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/54 [18:49:09] (03merge) 10taavi: Bump ts-jest to a version compatible with our Jest [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/54 [19:56:19] 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T382362#10442595 (10dduvall) [20:20:38] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [20:25:53] RESOLVED: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [20:30:39] 10Continuous-Integration-Infrastructure (Zuul upgrade): Investigate zuul migrate script used for the Zuul v3 OpenStack migration - https://phabricator.wikimedia.org/T383273 (10hashar) 03NEW [21:25:26] hi! I am getting CI failures like `Invalid response body while trying to fetch https://registry.npmjs.org …` at https://gerrit.wikimedia.org/r/1099840 [21:25:55] in the past this was fixed by clearing the cache of Jenkins or the CI job, I'm not sure exactly… But can someone clear it again for me? [21:26:19] this seems to happen at lot in the CodeMirror repo. I'm not sure why. [21:34:21] musikanimal: it's not just you, I'm also seeing it in gate-and-submit jobs like mwgate-node20 on mediawiki/core patches https://integration.wikimedia.org/ci/job/mwgate-node20/5882/console#:~:text=npm%20error%20code-,ECONNRESET,-21%3A20%3A04 [21:34:42] looks like T383237 from earlier but it got worse maybe [21:34:44] T383237: Many CI builds failing with ECONNRESET from npmjs.org during install:bridge steps - https://phabricator.wikimedia.org/T383237 [21:39:41] (03CR) 10BryanDavis: [C:04-1] Convert to Manifest v3 (031 comment) [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis) [21:44:39] (03PS4) 10BryanDavis: Convert to Manifest v3 [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) [21:49:54] (03CR) 10BryanDavis: Convert to Manifest v3 (031 comment) [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis) [22:01:19] musikanimal: T383237 , that got filed & mentioned by Lucas_WMDE earlier today [22:01:19] T383237: Many CI builds failing with ECONNRESET from npmjs.org during install:bridge steps - https://phabricator.wikimedia.org/T383237 [22:01:25] ah what chris said [22:01:37] hehe yep, I just commented [22:01:54] I should have created a task months ago, I guess. At least for the CodeMirror repo, this is a recurring problem [22:02:30] it is probably an issue with the npmjs registry [22:02:42] maybe we are rate limited [22:02:49] still, could someone just bust the cache for a quick fix? I don't have rights to do so (nor do I know how, even if I did have rights) [22:03:06] yeah good point [22:07:53] !log castor: deleting potentially corrupted npm cache. On integration-castor05: sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/{wmf-quibble-selenium-php74,quibble-vendor-mysql-php74-selenium}/npm # T383237 [22:07:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:07:55] T383237: Many CI builds failing with ECONNRESET from npmjs.org during install:bridge steps - https://phabricator.wikimedia.org/T383237 [22:08:33] ssh integration-castor05.integration.eqiad1.wikimedia.cloud 'sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/{wmf-quibble-selenium-php74,quibble-vendor-mysql-php74-selenium}/npm' [22:08:37] that is the command to run [22:08:44] musikanimal: thanks for the hint :) [22:08:55] I forgot the cache gets corrupted and npm does not quite recovers from that [22:09:06] hehe I'm just repeating what I recall James_F telling me in the past [22:09:53] yeah there is a race condition in how the caches are saved [22:10:08] so that if a job fetch the cache while another is writing to it [22:10:12] that cache is faulty [22:10:25] then hmm it would not be saved [22:10:27] bah [22:10:30] it is too late to think [22:17:25] that cache clear fixed one build for me, but not another :( https://integration.wikimedia.org/ci/job/mwext-node20-rundoc/849/console [22:17:49] I'll just try again later when CI is a little less busy [22:18:11] https://status.npmjs.org/incidents/jlqm624klvs2 [22:18:20] We are currently investigating this issue [22:18:20] Posted 11 minutes ago. Jan 08, 2025 - 22:05 UTC [22:18:23] This incident affects: Package installation. [22:22:26] I am off to sleep [22:22:39] get some sleep! goodnight, and thank you :) [23:28:17] (03CR) 10Krinkle: "A few nits from static code review (untested)." [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis)