[04:05:23] 10GitLab: CodeReviewBot missed 3 of 5 transactions associated with an MR - https://phabricator.wikimedia.org/T359899 (10bd808) 03NEW [04:17:32] 10Beta-Cluster-Infrastructure: error.log is not rotated in beta - https://phabricator.wikimedia.org/T345566#9622402 (10tstarling) 05Openβ†’03Resolved a:03tstarling The issue with logs not being rotated is apparently fixed. If there are some remaining issues with mwlog02, they should probably be discussed on... [06:23:48] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/BounceHandler] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1009285 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [06:23:48] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/CommunityConfiguration] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1009283 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [06:23:50] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/GlobalCssJs] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1009284 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [06:23:56] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/WikibaseCirrusSearch] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1009286 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [06:25:26] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/BounceHandler] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1009285 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [06:25:29] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CommunityConfiguration] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1009283 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [06:25:31] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/GlobalCssJs] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1009284 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [06:25:35] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikibaseCirrusSearch] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1009286 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [06:26:52] !log Reloaded Zuul to enable Sonar Codehealth on BounceHander, CommunityConfiguration, GlobalCssJs, WikibaseCirrusSearch - T321837 [06:26:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:26:56] T321837: Repositories integrated into Codehealth Pipeline (Production) - https://phabricator.wikimedia.org/T321837 [06:34:24] (03CR) 10Hashar: [C: 03+2] Link to Cite docs (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/1007265 (https://phabricator.wikimedia.org/T358641) (owner: 10Awight) [06:35:02] (03Merged) 10jenkins-bot: Link to Cite docs [integration/docroot] - 10https://gerrit.wikimedia.org/r/1007265 (https://phabricator.wikimedia.org/T358641) (owner: 10Awight) [06:41:13] 10Continuous-Integration-Config: Jsdoc for MediaWiki extension is published with an extra directory: master/js/js - https://phabricator.wikimedia.org/T359907 (10hashar) 03NEW [06:42:01] (03CR) 10Hashar: [C: 03+2] "I have deployed the change. For the `master/js/js` publishing destination I have filed T359907 as a placeholder for later." [integration/docroot] - 10https://gerrit.wikimedia.org/r/1007265 (https://phabricator.wikimedia.org/T358641) (owner: 10Awight) [07:45:14] (03CR) 10Awight: Link to Cite docs (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/1007265 (https://phabricator.wikimedia.org/T358641) (owner: 10Awight) [07:48:58] (03CR) 10Hashar: [C: 03+2] Link to Cite docs (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/1007265 (https://phabricator.wikimedia.org/T358641) (owner: 10Awight) [07:50:02] (03CR) 10Hashar: [C: 03+1] "Well done!" [integration/config] - 10https://gerrit.wikimedia.org/r/1010310 (https://phabricator.wikimedia.org/T359838) (owner: 10Jforrester) [07:59:24] (03CR) 10Hashar: "I'd like to avoid adding custom configuration in the Docker image unless that is directly related to the CI infra (such as setting the cac" [integration/config] - 10https://gerrit.wikimedia.org/r/1007861 (https://phabricator.wikimedia.org/T358841) (owner: 10Gehel) [08:39:52] (03CR) 10Hashar: secnpm: initial commit (031 comment) [fresh] - 10https://gerrit.wikimedia.org/r/675346 (owner: 10Krinkle) [08:47:34] (03CR) 10Hashar: "> make systemtest passes for me when all 3 commits in the stack are present." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/971215 (owner: 10Hashar) [09:33:21] 10Continuous-Integration-Infrastructure, 10EntitySchema, 10Wikidata, 10wmde-wikidata-tech, 10Wikidata Dev Team (Wikidata.org Slice): [ES] Investigate OutOfMemory error in browser test, remove workaround - https://phabricator.wikimedia.org/T356402#9622768 (10Lucas_Werkmeister_WMDE) a:03ArthurTaylor [09:34:58] 10Continuous-Integration-Infrastructure, 10EntitySchema, 10Wikidata, 10wmde-wikidata-tech, 10Wikidata Dev Team (Wikidata.org Slice): [ES] Investigate OutOfMemory error in browser test, remove workaround - https://phabricator.wikimedia.org/T356402#9622774 (10Lucas_Werkmeister_WMDE) **Prio Notes:** | Impa... [09:39:43] (03PS1) 10Isabelle Hurbain-Palatin: Add ?useparsoid=0 at the end of legacy URLs [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1010491 [09:41:28] (03CR) 10Isabelle Hurbain-Palatin: "I'd like to have that, but I haven't checked the exact usage of that variable throughout the visual diff test - thought I'd leave that to " [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1010491 (owner: 10Isabelle Hurbain-Palatin) [11:19:40] 10Continuous-Integration-Infrastructure, 10EntitySchema, 10Wikidata, 10wmde-wikidata-tech, and 2 others: [ES] Investigate OutOfMemory error in browser test, remove workaround - https://phabricator.wikimedia.org/T356402#9623035 (10Lucas_Werkmeister_WMDE) > On the one hand, I can see that it's good to run th... [12:59:05] 10Release-Engineering-Team (Now this 🫠), 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Notify MediaWiki security tasks as soon as an uploaded patch fails to apply - https://phabricator.wikimedia.org/T350065#9623317 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/scap/-... [13:04:08] (03CR) 10Subramanya Sastry: [C: 03+2] "The content of these config files are part of /etc/testreduce/* on the parsing-qa-02 server ... so, we'll have to make this change on the " [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1010491 (owner: 10Isabelle Hurbain-Palatin) [13:04:47] (03Merged) 10jenkins-bot: Add ?useparsoid=0 at the end of legacy URLs [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1010491 (owner: 10Isabelle Hurbain-Palatin) [13:25:11] 06Release-Engineering-Team, 06Quality-and-Test-Engineering-Team, 10Temporary accounts, 06Trust and Safety Product Team: Temp accounts deployment and the release train - https://phabricator.wikimedia.org/T355882#9623403 (10kostajh) Adding #quality-and-test-engineering-team as well, cc @Jrbranaa [13:25:16] maintenance-disconnect-full-disks build 578313 integration-agent-docker-1047 (/: 26%, /srv: 5%, /var/lib/docker: 100%): OFFLINE due to disk space [13:35:16] maintenance-disconnect-full-disks build 578315 integration-agent-docker-1047 (/: 26%, /srv: 5%, /var/lib/docker: 100%): still OFFLINE due to disk space [13:41:02] 10Release-Engineering-Team (Now this 🫠), 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Notify MediaWiki security tasks as soon as an uploaded patch fails to apply - https://phabricator.wikimedia.org/T350065#9623472 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/jenkin... [13:41:04] 10Release-Engineering-Team (Now this 🫠), 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Notify MediaWiki security tasks as soon as an uploaded patch fails to apply - https://phabricator.wikimedia.org/T350065#9623473 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/releas... [13:54:10] (03PS1) 10Hashar: Revert "jjb: remove useless yard generated file" [integration/config] - 10https://gerrit.wikimedia.org/r/1010223 (https://phabricator.wikimedia.org/T358507) [13:54:25] (03PS2) 10Hashar: Revert "jjb: remove useless yard generated file" [integration/config] - 10https://gerrit.wikimedia.org/r/1010223 (https://phabricator.wikimedia.org/T358507) [13:55:06] (03PS3) 10Hashar: Revert "jjb: remove useless yard generated file" [integration/config] - 10https://gerrit.wikimedia.org/r/1010223 (https://phabricator.wikimedia.org/T358507) [13:58:53] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.42.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T354440#9623560 (10Lucas_Werkmeister_WMDE) {T359939} seems to be new in this train, though I’m not sure it’s serious enough to be a train blocker. [14:00:15] maintenance-disconnect-full-disks build 578320 integration-agent-docker-1047 (/: 26%, /srv: 5%, /var/lib/docker: 100%): still OFFLINE due to disk space [14:01:11] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.42.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T354440#9623573 (10hashar) [14:02:02] hi jnuche & hashar, would it be possible to sync https://gerrit.wikimedia.org/r/c/integration/config/+/1010500 this week? thanks in advance [14:15:35] (03PS2) 10Reedy: zuul: Enable IPReputation in CI [integration/config] - 10https://gerrit.wikimedia.org/r/1010500 (https://phabricator.wikimedia.org/T359921) (owner: 10Kosta Harlan) [14:15:39] (03CR) 10Reedy: [C: 03+2] zuul: Enable IPReputation in CI [integration/config] - 10https://gerrit.wikimedia.org/r/1010500 (https://phabricator.wikimedia.org/T359921) (owner: 10Kosta Harlan) [14:16:06] thx Reedy! [14:18:04] (03Merged) 10jenkins-bot: zuul: Enable IPReputation in CI [integration/config] - 10https://gerrit.wikimedia.org/r/1010500 (https://phabricator.wikimedia.org/T359921) (owner: 10Kosta Harlan) [14:18:43] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1010500 [14:18:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:20:01] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.42.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T354440#9623627 (10hashar) >>! In T354440#9623560, @Lucas_Werkmeister_WMDE wrote: > {T359939} seems to be new in this train, though I’m not sure it’s serious... [14:20:27] (03CR) 10Jforrester: "-1 In the production section but no production template, and no comment about whether it's headed there." [integration/config] - 10https://gerrit.wikimedia.org/r/1010500 (https://phabricator.wikimedia.org/T359921) (owner: 10Kosta Harlan) [14:21:46] (03CR) 10Kosta Harlan: "Sorry, I missed that this was in a production specific setting. Yes, the short term plan is to swap out CentralAuth's IP reputation code w" [integration/config] - 10https://gerrit.wikimedia.org/r/1010500 (https://phabricator.wikimedia.org/T359921) (owner: 10Kosta Harlan) [14:21:54] (03PS1) 10Zoranzoki21: Zuul: Add Nemoralis to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/1010528 [14:22:17] (03CR) 10Jforrester: "I'll fix." [integration/config] - 10https://gerrit.wikimedia.org/r/1010500 (https://phabricator.wikimedia.org/T359921) (owner: 10Kosta Harlan) [14:24:32] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/IPReputation] Add prod template, comment [integration/config] - 10https://gerrit.wikimedia.org/r/1010529 (https://phabricator.wikimedia.org/T359921) [14:25:16] maintenance-disconnect-full-disks build 578325 integration-agent-docker-1047 (/: 26%, /srv: 5%, /var/lib/docker: 100%): still OFFLINE due to disk space [14:26:43] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/IPReputation] Add prod template, comment [integration/config] - 10https://gerrit.wikimedia.org/r/1010529 (https://phabricator.wikimedia.org/T359921) (owner: 10Jforrester) [14:26:59] (03CR) 10Kosta Harlan: "thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/1010529 (https://phabricator.wikimedia.org/T359921) (owner: 10Jforrester) [14:28:43] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/IPReputation] Add prod template, comment [integration/config] - 10https://gerrit.wikimedia.org/r/1010529 (https://phabricator.wikimedia.org/T359921) (owner: 10Jforrester) [14:31:11] (03PS2) 10Jforrester: Zuul: Add Nemoralis to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1010528 (owner: 10Zoranzoki21) [14:31:38] (03CR) 10Jforrester: [C: 03+2] Zuul: Add Nemoralis to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1010528 (owner: 10Zoranzoki21) [14:31:56] !log Zuul: [mediawiki/extensions/IPReputation] Add prod template, comment [14:31:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:33:28] (03Merged) 10jenkins-bot: Zuul: Add Nemoralis to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1010528 (owner: 10Zoranzoki21) [14:36:59] !log Zuul: Add Nemoralis to CI allow list [14:37:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:50:15] maintenance-disconnect-full-disks build 578330 integration-agent-docker-1047 (/: 26%, /srv: 5%, /var/lib/docker: 100%): still OFFLINE due to disk space [14:52:23] joy [14:52:32] why is /var/lib/docker full? :/ [14:52:39] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review, 15User-brennen: Update wmf/stable to Phorge upstream's 2023.49 stable release - https://phabricator.wikimedia.org/T358610#9623792 (10LSobanski) p:05Triageβ†’03High [14:52:48] Too many images, too few deletions. [14:53:05] Build Cache 82 0 45.57GB 45.57GB [14:53:07] no no [14:53:15] we delete all images on a weekly basis afaik [14:53:28] Do we have multiple versions of Quibble or another very heavy image live in CI at once? [14:53:40] nop [14:53:46] that host only has 390MB of Images [14:53:50] 0 of containers [14:54:03] 0B of volumes [14:54:06] Oh. But 46GiB of build? [14:54:12] Do we clear images but not build artefacts? [14:54:24] I bet it is similar to T338317 [14:54:25] T338317: Python torch fills disk of CI Jenkins instances - https://phabricator.wikimedia.org/T338317 [14:54:32] Joy [14:54:49] yeah [14:59:16] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Machine-Learning-Team: Python torch fills disk of CI Jenkins instances - https://phabricator.wikimedia.org/T338317#9623848 (10hashar) 05Resolvedβ†’03Open And that broke multiple CI Jenkins agents again: ` hashar@integration-agent-docker... [15:00:15] maintenance-disconnect-full-disks build 578332 integration-agent-docker-1047 (/: 26%, /srv: 5%, /var/lib/docker: 2%): RECOVERY disk space OK [15:00:18] !log integration: clearing Docker build cache on all Jenkins agents due to T338317 | sudo cumin --force 'name:docker' 'docker buildx prune --force' [15:00:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:00:22] T338317: Python torch fills disk of CI Jenkins instances - https://phabricator.wikimedia.org/T338317 [15:00:56] that "fixed" it [15:15:42] 10GitLab: CodeReviewBot missed 3 of 5 transactions associated with an MR - https://phabricator.wikimedia.org/T359899#9623945 (10dancy) Taking a look... [15:16:02] 10Diffusion, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Make https://git.wikimedia.org not redirect to Phabricator Diffusion - https://phabricator.wikimedia.org/T323073#9623941 (10LSobanski) p:05Lowβ†’03Medium [15:25:43] 10GitLab, 13Patch-For-Review: CodeReviewBot missed 3 of 5 transactions associated with an MR - https://phabricator.wikimedia.org/T359899#9623957 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/27 Improve Bug: footer regexp [15:31:58] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Machine-Learning-Team: Python torch fills disk of CI Jenkins instances - https://phabricator.wikimedia.org/T338317#9623973 (10dancy) We could configure buildkit gc rules for the Docker daemon: https://docs.docker.com/build/cache/garbage-c... [15:51:10] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review, 15User-brennen: Update wmf/stable to Phorge upstream's 2023.49 stable release - https://phabricator.wikimedia.org/T358610#9623997 (10CodeReviewBot) brennen merged https://gitlab.wikimedia.org/repos/phabricator/deploy... [19:31:02] (03PS4) 10Hashar: Revert "jjb: remove useless yard generated file" [integration/config] - 10https://gerrit.wikimedia.org/r/1010223 (https://phabricator.wikimedia.org/T358507) [19:33:09] (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 3" [integration/config] - 10https://gerrit.wikimedia.org/r/1010223 (https://phabricator.wikimedia.org/T358507) (owner: 10Hashar) [19:41:25] (03Merged) 10jenkins-bot: Revert "jjb: remove useless yard generated file" [integration/config] - 10https://gerrit.wikimedia.org/r/1010223 (https://phabricator.wikimedia.org/T358507) (owner: 10Hashar) [19:43:44] 10Gerrit: git-review prevents git from outputing colors - https://phabricator.wikimedia.org/T359981 (10hashar) 03NEW [19:44:35] 10Gerrit: git-review prevents git from outputing colors - https://phabricator.wikimedia.org/T359981#9624925 (10hashar) [20:13:00] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.42.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T354440#9625008 (10hashar) [20:28:52] FYI folks, I'm building out a new (puppet 7) puppet server and db for deployment-prep. It shouldn't break things for long but may result in some emails or other complaints. [20:29:12] (and I'm paring down to one puppetdb server because 2 seems excessive) [20:29:51] andrewbogott: thank you for the notice and also for the work :) [20:32:46] I'm migrating certs from the old to the new one, so servers created during the transition may wind up broken. I'm hoping/trusting that it's unlikely that the project will get new VMs in the next few hours. [21:01:11] 10Beta-Cluster-Infrastructure: 14error.log is not rotated in beta - 14https://phabricator.wikimedia.org/T345566#9625060 (10Jdforrester-WMF) 14> The issue with logs not being rotated is apparently fixed. Is it fixed? We switched it off for the move to mwlog02 but didn't switch it back on (because it was bro...