[07:46:06] 10WikimediaDebug, 06Content-Platform-Team, 06ServiceOps: k8s-mw-parsoid-eqiad / k8s-mw-parsoid-codfw no longer available for selection in WikimediaDebug extension - https://phabricator.wikimedia.org/T442935#12531905 (10Krinkle) [08:13:48] (03update) 10shivnagi: docs: Add section on image layer and cache efficiency [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/189 (https://phabricator.wikimedia.org/T268780) [08:26:05] (03CR) 10Hashar: [C:03+2] "Thanks for the backporting 😊" [integration/config] - 10https://gerrit.wikimedia.org/r/1284724 (https://phabricator.wikimedia.org/T421798) (owner: 10Jforrester) [08:27:51] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/AutoCreatePage] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1363377 (https://phabricator.wikimedia.org/T440822) (owner: 10Pppery) [08:28:08] (03Merged) 10jenkins-bot: Zuul: […/FlaggedRevs] Drop GoogleNewsSitemap dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1284724 (https://phabricator.wikimedia.org/T421798) (owner: 10Jforrester) [08:29:11] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/DeleteBatch] Add AdminLinks Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1367403 (owner: 10Fomafix) [08:29:59] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/AutoCreatePage] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1363377 (https://phabricator.wikimedia.org/T440822) (owner: 10Pppery) [08:31:03] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/DeleteBatch] Add AdminLinks Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1367403 (owner: 10Fomafix) [08:35:29] 10Gerrit, 06Collaboration-Services, 07Incident Severity 1, 13Patch-For-Review, 07Sustainability (Incident Followup): Add the the ability to warm caches to Gerrit switchover cookbook - https://phabricator.wikimedia.org/T440177#12531991 (10ABran-WMF) 05Open→03Stalled Necessary puppet changes are merged... [08:37:49] (03open) 10shivnagi: config: Iteratively resolve copies dependencies with includes [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/190 (https://phabricator.wikimedia.org/T411377) [09:02:50] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: runner-1036 & runner-1034.gitlab-runners.eqiad1.wikimedia.cloud running out of space - https://phabricator.wikimedia.org/T443833#12532090 (10Addshore) [09:23:58] (03PS1) 10Hashar: zuul: limit pywikibot doc to master/stable [integration/config] - 10https://gerrit.wikimedia.org/r/1367740 (https://phabricator.wikimedia.org/T444044) [09:25:02] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Pywikibot, 10Wikimedia-GitHub, 13Patch-For-Review: Delete the already deleted Pywikibot branch "tests" on Gitub mirror - https://phabricator.wikimedia.org/T444044#12532146 (10hashar) >>! In T444044#12531019, @Xqt wrote: > There is als... [09:29:45] (03open) 10cwilliams: Add repos/sre/pampinus to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/198 (https://phabricator.wikimedia.org/T443897) [09:51:04] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for CollabPatrol - https://phabricator.wikimedia.org/T443867#12532301 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project tag has been created: #MediaWiki-extensions-CollabPatrol (In case you need to edit the... [10:20:44] (03CR) 10Xqt: [C:03+1] zuul: limit pywikibot doc to master/stable [integration/config] - 10https://gerrit.wikimedia.org/r/1367740 (https://phabricator.wikimedia.org/T444044) (owner: 10Hashar) [10:25:12] 06Release-Engineering-Team (Doing 😎): [January 2026] Publish some Phabricator (and/or Gerrit) end-of-year stats for 2025 to wikitech-l@ - https://phabricator.wikimedia.org/T385732#12532576 (10Aklapper) 05Stalled→03Resolved a:03Aklapper Looks like I did this: https://lists.wikimedia.org/hyperkitty/list/... [10:25:30] 10Phabricator, 06Release-Engineering-Team: [January 2027] Publish some Phabricator (and/or Gerrit) end-of-year stats for 2026 to wikitech-l@ - https://phabricator.wikimedia.org/T444116 (10Aklapper) 03NEW [10:26:33] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Pywikibot, 10Wikimedia-GitHub, 13Patch-For-Review: Delete the already deleted Pywikibot branch "tests" on Gitub mirror - https://phabricator.wikimedia.org/T444044#12532604 (10Xqt) > > I have removed the documentation for the `tests`... [10:36:41] (03CR) 10Hashar: [C:03+2] zuul: limit pywikibot doc to master/stable [integration/config] - 10https://gerrit.wikimedia.org/r/1367740 (https://phabricator.wikimedia.org/T444044) (owner: 10Hashar) [10:38:40] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Backtrack usage of `recheck` in Gerrit to find CI failures - https://phabricator.wikimedia.org/T429539#12532680 (10hashar) 05Open→03Declined Maybe I will revisit that later. Meanwhile I am closing the... [10:38:42] (03Merged) 10jenkins-bot: zuul: limit pywikibot doc to master/stable [integration/config] - 10https://gerrit.wikimedia.org/r/1367740 (https://phabricator.wikimedia.org/T444044) (owner: 10Hashar) [10:39:36] (03CR) 10Hashar: [C:03+2] Revert "Zuul: [mediawiki/extensions/PageViewInfo] Add Scribunto phan dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1367322 (https://phabricator.wikimedia.org/T362937) (owner: 10SD0001) [10:40:45] (03Merged) 10jenkins-bot: Revert "Zuul: [mediawiki/extensions/PageViewInfo] Add Scribunto phan dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1367322 (https://phabricator.wikimedia.org/T362937) (owner: 10SD0001) [11:32:33] (03PS2) 10Hashar: Refactor tests framework [fresh] - 10https://gerrit.wikimedia.org/r/1347688 [11:38:35] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Pywikibot, 10Wikimedia-GitHub: Delete the already deleted Pywikibot branch "tests" on Gitub mirror - https://phabricator.wikimedia.org/T444044#12532973 (10Xqt) 05Open→03Resolved a:03Xqt [11:42:41] (03PS7) 10Hashar: test: colorize output of assertions [fresh] - 10https://gerrit.wikimedia.org/r/1347656 [11:46:23] (03Abandoned) 10Hashar: Docker: [java*] Re-build to get latest openjdks with Oct 2026 security updates [integration/config] - 10https://gerrit.wikimedia.org/r/1360519 (owner: 10Jforrester) [11:46:30] (03Abandoned) 10Hashar: jjb: Bump Java-based jobs to latest openjdks with October 2026 security updates [integration/config] - 10https://gerrit.wikimedia.org/r/1360520 (owner: 10Jforrester) [11:47:02] (03Abandoned) 10Hashar: build: Updating postcss-selector-parser to 6.1.5 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1362242 (owner: 10Libraryupgrader) [11:49:44] (03Abandoned) 10Hashar: build: Upgrade eslint-config-wikimedia from 0.23.0 to 0.32.3 and make pass [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1249368 (owner: 10Jforrester) [12:04:27] (03CR) 10CI reject: [V:04-1] test: colorize output of assertions [fresh] - 10https://gerrit.wikimedia.org/r/1347656 (owner: 10Hashar) [12:27:50] (03PS5) 10Arendpieter: castor: Stop blocking builds on the cache save [integration/config] - 10https://gerrit.wikimedia.org/r/1341139 (https://phabricator.wikimedia.org/T352319) [12:31:25] 10WikimediaDebug, 06Content-Platform-Team, 06ServiceOps: k8s-mw-parsoid-eqiad / k8s-mw-parsoid-codfw no longer available for selection in WikimediaDebug extension - https://phabricator.wikimedia.org/T442935#12533173 (10jijiki) We intentionally left those deployments out of debug.json from the get-go, since t... [12:34:27] (03CR) 10CI reject: [V:04-1] Refactor tests framework [fresh] - 10https://gerrit.wikimedia.org/r/1347688 (owner: 10Hashar) [12:41:49] (03CR) 10Arendpieter: "Rebased on master. The new CASTOR_DISABLED early exit now comes after the EXIT trap, so the staged cache still gets removed." [integration/config] - 10https://gerrit.wikimedia.org/r/1341139 (https://phabricator.wikimedia.org/T352319) (owner: 10Arendpieter) [14:30:08] (03CR) 10Hashar: "recheck after I1343af806d4592fc58f37e070581c4b81766df2b" [fresh] - 10https://gerrit.wikimedia.org/r/1347688 (owner: 10Hashar) [14:32:18] (03CR) 10Hashar: "recheck after I1343af806d4592fc58f37e070581c4b81766df2b" [fresh] - 10https://gerrit.wikimedia.org/r/1347656 (owner: 10Hashar) [14:56:30] https://integration.wikimedia.org/ci/job/fresh-test/539/console : SUCCESS in 5m 25s [14:56:32] somehow :) [14:57:39] Krinkle: looks like QEmu fresh tests are way faster than they used to be (5minutes vers 15 minutes). Possibly cause the instance running QEmu got rebuilt to Trixie by dduvall. That gave us a new QEmu version and the instance is probably running on a faster cloudvirt host :) [14:58:02] hashar: nice [14:58:10] I wonder if there's any chance to docker cache maybe? [14:58:36] There was a time when we did this by hand, that we invoked the command once to warm up the inside-VM docker registry cache. [14:58:37] maybe [14:58:47] over time that will become less useful and I htink we stopped doing that [14:58:54] we would need to rebuild the image each time images are updated [14:59:07] well, once a year is enough to get some value [14:59:20] but I guess it isn't needed anbymore if it's fast enough now [14:59:24] the layers get entirely invalidated from time to time though :) [14:59:30] yup [14:59:41] fresh doesn't get as updated as CI itself [14:59:43] and I think it is fine to wait some minutes, given one can always run them locally [15:00:21] and if you feel like it, I have slightly refactored the test suite at https://gerrit.wikimedia.org/r/c/fresh/+/1347688/ [15:00:34] and added some color output https://gerrit.wikimedia.org/r/c/fresh/+/1347688/ [15:00:44] 10MediaWiki-Releasing, 10MediaWiki-Documentation, 05MW-1.45-release: Formally EOL MW 1.45 - https://phabricator.wikimedia.org/T433376#12534242 (10Reedy) [15:00:46] https://gerrit.wikimedia.org/r/c/fresh/+/1347656/ [15:00:53] I'll poke releng about those patches [15:03:06] 10MediaWiki-Releasing, 10MediaWiki-Documentation, 05MW-1.45-release: Formally EOL MW 1.45 - https://phabricator.wikimedia.org/T433376#12534269 (10hashar) [15:12:22] 10WikimediaDebug, 06Content-Platform-Team, 06ServiceOps: k8s-mw-parsoid-eqiad / k8s-mw-parsoid-codfw no longer available for selection in WikimediaDebug extension - https://phabricator.wikimedia.org/T442935#12534357 (10MLechvien-WMF) @cscott could you clarify what has changed in the past two weeks? [15:30:35] (03merge) 10dancy: Add repos/sre/pampinus to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/198 (https://phabricator.wikimedia.org/T443897) (owner: 10cwilliams) [15:39:35] (03PS2) 10Hashar: dockerfiles: remove buildctl image [integration/config] - 10https://gerrit.wikimedia.org/r/1361876 (https://phabricator.wikimedia.org/T443253) [15:40:26] (03CR) 10Hashar: "I have done some archeology, `releng/buildctl` was used in Kokkuri up until v1.1.0:" [integration/config] - 10https://gerrit.wikimedia.org/r/1361876 (https://phabricator.wikimedia.org/T443253) (owner: 10Hashar) [16:17:09] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Spike, 06Test Platform (San Miguel de Allende 31): Create a plan on how to make Castor save jobs async - https://phabricator.wikimedia.org/T425411#12534716 (10AMarkossyan-WMF) 05Open→03Declined [16:22:08] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (San Miguel de Allende 31): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12534736 (10Mhurd) 05Open→03Resolved a:03Mhurd @dcaro Yeah I think the 4x-io... [16:22:54] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (San Miguel de Allende 31): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12534739 (10AMarkossyan-WMF) a:03Mhurd [16:27:52] 10Continuous-Integration-Infrastructure, 10Castor, 05MW-1.48-notes (1.48.0-wmf.3; 2026-11-03), 13Patch-For-Review, 06Test Platform (San Miguel de Allende 31): Test the cache proxy on integration VPS - https://phabricator.wikimedia.org/T439116#12534771 (10AMarkossyan-WMF) 05Open→03Resolved [16:29:44] (03CR) 10Dduvall: [C:03+2] dockerfiles: remove buildctl image [integration/config] - 10https://gerrit.wikimedia.org/r/1361876 (https://phabricator.wikimedia.org/T443253) (owner: 10Hashar) [16:30:19] (03CR) 10Dduvall: [C:03+2] ci: update releng/gradle to latest [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/1363239 (https://phabricator.wikimedia.org/T383337) (owner: 10Hashar) [16:31:01] (03Merged) 10jenkins-bot: ci: update releng/gradle to latest [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/1363239 (https://phabricator.wikimedia.org/T383337) (owner: 10Hashar) [16:31:23] (03Merged) 10jenkins-bot: dockerfiles: remove buildctl image [integration/config] - 10https://gerrit.wikimedia.org/r/1361876 (https://phabricator.wikimedia.org/T443253) (owner: 10Hashar) [16:58:30] (03update) 10mhurd: Add a dashboard for moving from Castor to the cache proxy [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/103 [17:04:30] (03PS1) 10Snwachukwu: Add Thanks to WikimediaEvents dependency list [integration/config] - 10https://gerrit.wikimedia.org/r/1367888 (https://phabricator.wikimedia.org/T441426) [17:44:54] (03PS1) 10Dduvall: build: Build images with buildx [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/1367902 (https://phabricator.wikimedia.org/T440668) [17:44:56] (03PS1) 10Dduvall: build: Support provenance and SBOM attestations [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/1367903 (https://phabricator.wikimedia.org/T440668) [19:20:07] (03update) 10catrope: Add a script for checking which security patches can't be backported [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/282 [19:23:06] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-webperf31 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:05:07] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [20:15:26] (03CR) 10Jeena Huneidi: [C:03+2] Add Thanks to WikimediaEvents dependency list [integration/config] - 10https://gerrit.wikimedia.org/r/1367888 (https://phabricator.wikimedia.org/T441426) (owner: 10Snwachukwu) [20:16:41] (03Merged) 10jenkins-bot: Add Thanks to WikimediaEvents dependency list [integration/config] - 10https://gerrit.wikimedia.org/r/1367888 (https://phabricator.wikimedia.org/T441426) (owner: 10Snwachukwu) [20:19:09] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1367888 [20:19:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:33:08] 06Release-Engineering-Team, 06MediaWiki-API-Platform-Team, 06ServiceOps, 05FY2025-26 KR 5.1, and 2 others: rest-gateway: run system tests via helm test - https://phabricator.wikimedia.org/T424825#12535806 (10KineticPelagic) 05Open→03In progress [21:08:00] 10Phabricator: Disable H481 - https://phabricator.wikimedia.org/T444227 (10Aklapper) 03NEW [21:22:41] (03PS1) 10Snwachukwu: Add Thanks Extension to WikimediaEvents Phan Dependency List [integration/config] - 10https://gerrit.wikimedia.org/r/1367952 [21:23:06] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-webperf31 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [21:26:33] (03CR) 10Ahmon Dancy: [C:03+2] Add Thanks Extension to WikimediaEvents Phan Dependency List [integration/config] - 10https://gerrit.wikimedia.org/r/1367952 (owner: 10Snwachukwu) [21:27:49] (03Merged) 10jenkins-bot: Add Thanks Extension to WikimediaEvents Phan Dependency List [integration/config] - 10https://gerrit.wikimedia.org/r/1367952 (owner: 10Snwachukwu) [21:30:38] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1367952 [21:30:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:04:05] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-webperf31 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:31:15] (03open) 10aklapper: Remove ancient unused Security task access permission code [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/72 (https://phabricator.wikimedia.org/T443243) [22:34:06] (03update) 10aklapper: Remove ancient unused Security task access permission code [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/72 (https://phabricator.wikimedia.org/T443243) [22:38:08] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 10Wikimedia-Phabricator-Extensions, 13Patch-For-Review: Decouple setting #permanentlyprivate from setting #security and #security-team project tags - https://phabricator.wikimedia.org/T443243#12536279 (10Aklapper) a:03Aklapper Plan is to put that propo... [22:51:07] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [22:55:52] !log taking integration-castor06 down for iops upgrade [22:55:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:03:12] 06Release-Engineering-Team (Radar), 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 32): Determine if we can expand TK experiment deployment windows - https://phabricator.wikimedia.org/T440612#12536449 (10KReid-WMF) > I do think we need some reliable way to look up whether an experiment is ready... [23:21:51] !log castor back at 23:03 with 4x-iops volume [23:21:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:35:53] 06Release-Engineering-Team (Radar), 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 32): Determine if we can expand TK experiment deployment windows - https://phabricator.wikimedia.org/T440612#12536655 (10DLynch) > Or do we just do away with experiment deployment windows entirely? This would see... [23:51:32] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (San Miguel de Allende 31): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12536725 (10Mhurd) @thcipriani landed this using the [[ https://phabricator.wikimed... [23:55:51] (03update) 10catrope: Add a script for checking which security patches can't be backported [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/282