[03:38:12] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: scap backport due to expired debian snapshot - https://phabricator.wikimedia.org/T437829 (10tstarling) 03NEW [03:38:50] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12315688 (10tstarling) [03:44:00] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12315690 (10tstarling) Full terminal output: P96435 [04:38:50] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12315749 (10tstarling) The base image is broken. Locally I can do ` $ docker run -it docker-registry.wikimedia.org/bookworm:latest root@9efc76ae3ac4:/# cat /etc/ap... [05:10:59] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12315783 (10tstarling) Base image build gave warnings: ` warning: no apparent 'bookworm/updates/main' for 'amd64' on any of the following; skipping - http://snap... [05:51:12] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/PageProperties] Add WikiSEO Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1340607 (owner: 10Fomafix) [05:52:48] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PageProperties] Add WikiSEO Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1340607 (owner: 10Fomafix) [05:52:55] (03CR) 10Hashar: [C:03+2] Zuul: Add Linedwell's email to the CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1340565 (owner: 10Mahveotm) [05:55:05] (03Merged) 10jenkins-bot: Zuul: Add Linedwell's email to the CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1340565 (owner: 10Mahveotm) [06:54:12] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12315884 (10MoritzMuehlenhoff) >>! In T437069#12304969, @Scott_French wrote: > @MoritzMuehlenhoff - Do you happen to have any... [07:46:39] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/PageCheckout] Add Workflows Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1340502 (owner: 10Fomafix) [07:47:53] 10GitLab, 06Collaboration-Services, 06SRE: "apt update" failing in CI - https://phabricator.wikimedia.org/T437836 (10MatthewVernon) 03NEW [07:48:17] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PageCheckout] Add Workflows Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1340502 (owner: 10Fomafix) [07:48:23] 10GitLab, 06Collaboration-Services, 06SRE: "apt update" failing in CI - https://phabricator.wikimedia.org/T437836#12315981 (10MatthewVernon) [08:02:01] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337#12316017 (10hashar) [08:12:26] 10Gerrit: Gerrit UI/API reports the wrong running server version - https://phabricator.wikimedia.org/T437837 (10Volans) 03NEW [08:13:10] (03PS2) 10Hslater: Zuul: Add StandardDialogs as transitive dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1338146 [08:35:41] 10Gerrit: Gerrit UI/API reports the wrong running server version - https://phabricator.wikimedia.org/T437837#12316144 (10hashar) 05Open→03Declined Thanks for the report, this is known and a transient issue. Some patches were applied last week and led to a botched version number. I plan to upgrade again... [08:40:25] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Always get logged out when creating new patch demo - https://phabricator.wikimedia.org/T437616#12316166 (10jnuche) a:03jnuche [08:41:22] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Always get logged out when creating new patch demo - https://phabricator.wikimedia.org/T437616#12316182 (10jnuche) 05Open→03Resolved Fix is in prod. Closing tentatively [08:41:56] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Always get logged out when creating new patch demo - https://phabricator.wikimedia.org/T437616#12316206 (10jnuche) [09:08:54] 10GitLab, 06Collaboration-Services, 06SRE: "apt update" failing in CI - https://phabricator.wikimedia.org/T437836#12316587 (10Mvolz) Dupe of T437069 maybe? [09:15:02] 10GitLab, 06Collaboration-Services, 06SRE: "apt update" failing in CI - https://phabricator.wikimedia.org/T437836#12316631 (10Mvolz) Ticket says it's about bullseye and because it's EOL, but above is bookworm and I have builds failing on trixie, too... ` > [2/7] RUN apt-get update && apt-get install -y "g... [09:17:51] 10Gerrit, 10GitLab, 06Collaboration-Services, 06SRE: "apt update" failing in CI, on both bullseye and trixie, and both gerrit and gitlab - https://phabricator.wikimedia.org/T437836#12316665 (10Mvolz) [09:18:00] 10Gerrit, 10GitLab, 06Collaboration-Services, 06SRE: "apt update" failing in CI, on both bookworm and trixie, and both gerrit and gitlab - https://phabricator.wikimedia.org/T437836#12316667 (10Mvolz) [09:25:48] (03PS1) 10DCausse: add semantic highlighting build config [integration/config] - 10https://gerrit.wikimedia.org/r/1341083 (https://phabricator.wikimedia.org/T433872) [09:29:20] (03CR) 10Kevin Bazira: [C:03+1] add semantic highlighting build config [integration/config] - 10https://gerrit.wikimedia.org/r/1341083 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [09:40:49] (03CR) 10Hashar: [C:03+2] add semantic highlighting build config [integration/config] - 10https://gerrit.wikimedia.org/r/1341083 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [09:41:40] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate wikimedia/developer-portal to GitLab - https://phabricator.wikimedia.org/T343261#12316778 (10Mooeypoo) [09:42:32] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate wikimedia/developer-portal to GitLab - https://phabricator.wikimedia.org/T343261#12316784 (10Mooeypoo) I've started moving the interface and the basic behaviour from the prototype into the production repository, to get it ready for... [09:42:35] (03Merged) 10jenkins-bot: add semantic highlighting build config [integration/config] - 10https://gerrit.wikimedia.org/r/1341083 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [09:42:56] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate wikimedia/developer-portal to GitLab - https://phabricator.wikimedia.org/T343261#12316787 (10Mooeypoo) a:03Mooeypoo [09:49:12] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate wikimedia/developer-portal to GitLab - https://phabricator.wikimedia.org/T343261#12316810 (10Mooeypoo) [09:51:15] 10Gerrit, 10GitLab, 06Collaboration-Services, 06SRE: "apt update" failing in CI, on both bookworm and trixie, and both gerrit and gitlab - https://phabricator.wikimedia.org/T437836#12316817 (10hashar) →14Duplicate dup:03T437069 [09:51:16] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12316814 (10hashar) [09:52:19] 10GitLab, 06Collaboration-Services, 06SRE: "apt update" failing in CI, on both bookworm and trixie, and both gerrit and gitlab - https://phabricator.wikimedia.org/T437836#12316821 (10hashar) This is a dupe of T437069 / T437829 (both have more informations and attached patches). It is not related to Gerr... [09:54:34] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12316829 (10hashar) There was a related issue happening this morning mentioning expired release files. T... [09:54:55] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12316833 (10hashar) There was a similar issue (Release file expired) that surfaced on Friday 4th and filed as T437069 [09:56:04] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate wikimedia/developer-portal to GitLab - https://phabricator.wikimedia.org/T343261#12316838 (10Mooeypoo) [09:56:06] 10Scap: scap deploy --init on deployment server fails on first puppet run - https://phabricator.wikimedia.org/T257317#12316839 (10hashar) [09:58:37] 10Gerrit, 06Release-Engineering-Team, 06Collaboration-Services, 10Wikimedia-GitHub, and 2 others: gerrit: document replication re-runs to Github - https://phabricator.wikimedia.org/T424172#12316844 (10hashar) [09:58:38] 10Gerrit, 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#12316845 (10hashar) [10:00:00] 10Gerrit: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465#12316850 (10hashar) [10:07:45] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12316871 (10tstarling) I think it's mostly a bug in debuerreotype. The purpose of using snapshot.debian.org is for reproducible builds. The build isn't reproducible... [10:10:19] dcausse: for the Debian expired release files there is https://phabricator.wikimedia.org/T437069 from Sep 4th and from early this morning https://phabricator.wikimedia.org/T437829 [10:10:25] I don't know what are the proper fixes to have [10:12:17] (03CR) 10Hashar: [C:03+2] build: Updating npm dependencies [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1340481 (owner: 10Libraryupgrader) [10:32:09] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12316988 (10MoritzMuehlenhoff) >>! In T437829#12316871, @tstarling wrote: > I think it's mostly a bug in debuerreotype. The purpose of using snapshot.debian.org is... [10:40:37] 10Phabricator, 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Move aphlict to k8s - https://phabricator.wikimedia.org/T436657#12317039 (10ABran-WMF) [10:46:12] docker-registry.wikimedia.org/ml/amd-vllm014 gfx90agfx942rocm7.0.0pytorch2.10.0mori0.1flash-attn2.8.3aiter0.1.7vllm0.14-3 d8b3882bb03f 2 months ago 29.4GB [10:46:12] docker-registry.wikimedia.org/ml/amd-vllm022 gfx90agfx942rocm7.2.0pytorch2.10.0flash-attn2.8.3aiter0.1.13vllm0.22.1-2 09f180018504 2 months ago 37.2GB [10:46:12] Pytorch images being 29.4 and 37.2G *cough* [10:56:18] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 13Patch-For-Review: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12317118 (10MoritzMuehlenhoff) >>! In T437829#12316871, @tstarling wrote: > I think it's mostly a bug in debuerreotype. The purpose of using s... [11:30:24] Project ArticlePlaceholder-phpmetrics build #134: 04FAILURE in 28 min: https://integration.wikimedia.org/ci/job/ArticlePlaceholder-phpmetrics/134/ [12:07:05] (03PS1) 10Arendpieter: castor: Stop blocking builds on the cache save [integration/config] - 10https://gerrit.wikimedia.org/r/1341139 [12:14:07] (03PS2) 10Samtar: layout.yaml: Add ModeratorToolkit `in-wikimedia-production` job [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) [12:14:20] (03CR) 10Samtar: "This can be reviewed & merged afaict" [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [12:14:29] (03PS3) 10Samtar: layout.yaml: Add ModeratorToolkit `in-wikimedia-production` job [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) [12:22:18] (03PS1) 10Ozge: inference-services: Stop config.yaml changes from triggering all pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/1341162 (https://phabricator.wikimedia.org/T437882) [12:22:45] (03PS2) 10Ozge: inference-services: Stop config.yaml changes from triggering all pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/1341162 (https://phabricator.wikimedia.org/T437882) [12:33:09] (03CR) 10Kevin Bazira: [C:03+1] inference-services: Stop config.yaml changes from triggering all pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/1341162 (https://phabricator.wikimedia.org/T437882) (owner: 10Ozge) [12:54:25] (03open) 10phedenskog: Count only green runs in the core feedback time tables [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/65 [12:54:44] (03merge) 10phedenskog: Count only green runs in the core feedback time tables [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/65 [12:55:28] (03update) 10phedenskog: core: Show cache hit counts in the gate execution time tooltip [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/64 [13:06:01] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Helm deleting resources during environment update causes false failure - https://phabricator.wikimedia.org/T437744#12317610 (10jnuche) a:03jnuche [13:18:12] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI - Env Watcher: Handle optional deployments correctly - https://phabricator.wikimedia.org/T437893#12317689 (10jnuche) [13:22:03] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Helm deleting resources during environment update causes false failure - https://phabricator.wikimedia.org/T437744#12317717 (10jnuche) 05Open→03Resolved I've merged a quickfix to unblock yo... [13:39:12] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 13Patch-For-Review: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12317806 (10taavi) p:05Triage→03Unbreak! Not being able to deploy any MW patches sounds like an UBN, so boldly setting priority. [14:18:05] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12318249 (10ABran-WMF) Following up on: >>! In T425441#12285450, @ABran-WMF wrote: > thanks for mentioning it @jnuche > I sent 2 patches, one that will... [14:44:32] (03PS2) 10Hashar: Zuul: [BlueSpiceProDistributionConnector] Add OAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1338144 (owner: 10Hslater) [14:45:14] (03CR) 10Hashar: [C:03+2] "The OAuth requirement is introduced by Iac7a24c8ae0bf5c1ae37f27835bc82bb43492217" [integration/config] - 10https://gerrit.wikimedia.org/r/1338144 (owner: 10Hslater) [14:46:09] 06Release-Engineering-Team, 06Infrastructure-Foundations, 13Patch-For-Review: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297#12318371 (10elukey) p:05Triage→03Medium [14:47:50] (03Merged) 10jenkins-bot: Zuul: [BlueSpiceProDistributionConnector] Add OAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1338144 (owner: 10Hslater) [14:57:31] (03CR) 10Hashar: [C:03+1] "@jforrester@wikimedia.org the `in-wikimedia-production` job template is purely informational isn't it?" [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [14:59:31] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 13Patch-For-Review: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12318444 (10MoritzMuehlenhoff) The weekly rebuild from yesterday failed as "buster" was still listed, but had been garbage-collected from the... [15:17:53] (03CR) 10Jforrester: [C:03+1] "Yes, this is good to land now if the team are happy with it." [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [15:18:21] (03PS4) 10Jforrester: Zuul: [mediawiki/extensions/ModeratorToolkit] +in-wikimedia-production [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [15:23:15] 10Gerrit (Gerrit 3.10): Upgrade Gerrit from 3.10.6 to 3.10.9 - https://phabricator.wikimedia.org/T400688#12318503 (10hashar) We have upgraded 3.10.6 to 3.10.9 last week. That should fix T343471 & T405625, I am going to ping those tasks individually. The two other upstream issues should be looked at/confirmed. [15:25:36] 10Gerrit, 07Upstream: Gerrit does not send email when a new patchset is created using the web interface - https://phabricator.wikimedia.org/T343471#12318507 (10hashar) We have upgraded Gerrit from 3.10.6 to 3.10.9 last week (T400688). It should have fixed this specific issue. May one attempt to reproduce and c... [15:25:37] 10Gerrit, 07Upstream: Gerrit fudges multiple suggestions in comments - https://phabricator.wikimedia.org/T405625#12318511 (10hashar) We have upgraded Gerrit from 3.10.6 to 3.10.9 last week (T400688). It should have fixed this specific issue. May one attempt to reproduce and confirm the resolution? Thank you! [15:31:29] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12318562 (10Raymond_Ndibe) We noticed this problem today while deploy to toolforge k8s cluster. Builds... [15:36:19] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 13Patch-For-Review: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12318588 (10Raymond_Ndibe) This is also an issue for us at toolforge. we started getting similar failure messages while trying to build trixie... [15:38:37] 10Gerrit, 07Upstream: Gerrit fudges multiple suggestions in comments - https://phabricator.wikimedia.org/T405625#12318592 (10Paladox) 05Open→03Resolved Works for me. {F102918654} [15:49:48] 10Gerrit, 06Release-Engineering-Team, 06Collaboration-Services: Create alerts using the new replication metrics exposed in Gerrit - https://phabricator.wikimedia.org/T418216#12318651 (10LSobanski) →14Duplicate dup:03T418084 [15:49:49] 10Gerrit, 06Collaboration-Services: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#12318653 (10LSobanski) [15:51:19] 10Gerrit, 06Release-Engineering-Team, 06Collaboration-Services, 07Upstream: Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10 - https://phabricator.wikimedia.org/T407844#12318670 (10LSobanski) 05Open→03Stalled Pending {T379714} [15:53:54] 10Phabricator, 06Collaboration-Services, 06SRE, 06Traffic, 13Patch-For-Review: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228#12318700 (10LSobanski) 05Open→03Declined We're not planning to work on this task. [15:56:15] 06Release-Engineering-Team, 06Collaboration-Services: Update sync-gitlab-group-with-ldap to handle syncing users who have no email address - https://phabricator.wikimedia.org/T414428#12318719 (10LSobanski) 05Open→03Stalled Stalled on {T423984} [16:00:14] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 13Patch-For-Review: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12318763 (10Joe) >>! In T437829#12317806, @taavi wrote: > Not being able to deploy any MW patches sounds like an UBN, so boldly setting priori... [16:14:45] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839#12318852 (10Aklapper) Note that {T437829} could make this train rather cumbersome. [16:27:15] 06Release-Engineering-Team, 10dev-images, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Migrate Patchdemo & Catalyst to PHP 8.5, Node 24 - https://phabricator.wikimedia.org/T435368#12318923 (10thcipriani) a:03brennen [16:33:18] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12318952 (10elukey) The base images are being rebuilt as we speak and from some preliminary tests the sn... [17:03:49] (03CR) 10Samtar: "Happy for this to be merged" [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [17:10:17] (03CR) 10Majavah: [C:04-1] "the list is alphabetically sorted per section, and this entry is not in the right place" [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [17:13:09] (03PS5) 10Samtar: Zuul: [mediawiki/extensions/ModeratorToolkit] +in-wikimedia-production [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) [17:14:19] (03CR) 10Samtar: "whoops - resolved :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [17:15:52] (03CR) 10Majavah: [C:03+2] Zuul: [mediawiki/extensions/ModeratorToolkit] +in-wikimedia-production [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [17:16:01] thaaanks taavi [17:17:00] meow :3 [17:17:32] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ModeratorToolkit] +in-wikimedia-production [integration/config] - 10https://gerrit.wikimedia.org/r/1337606 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [17:18:11] !log relaoding zuul to deploy https://gerrit.wikimedia.org/r/1337606 [17:18:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:39:06] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06Collaboration-Services: Deploy Phab/Phorge 2026-09-15 - https://phabricator.wikimedia.org/T437930 (10brennen) 03NEW [17:44:04] taavi: couldn't bother you for a quick completeness +1 on https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1337611 could I? Just adding ModeratorToolkit to `extension-list` for T431000 [17:44:04] T431000: Deploy the ModeratorToolkit extension to Beta Cluster - https://phabricator.wikimedia.org/T431000 [17:44:46] ship it [17:45:11] thaanks, I owe you a +1 /j [17:46:03] TheresNoTime: redeemably on any patch, or just on those with similar size? [17:46:06] :P [17:46:30] any patch, but you only get 30s of me looking at it [18:32:49] (03update) 10dancy: group-management: add retry to gitlab helper _get method [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/82 (https://phabricator.wikimedia.org/T423984) (owner: 10lenap) [18:35:02] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 13Patch-For-Review: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12319469 (10dancy) 05Open→03Resolved a:03dancy [18:49:26] (03CR) 10Cklimas: "Hi, the App Growth team is hoping to get this patch landed for this week's release train. If you're able prioritize reviewing this, it wou" [integration/config] - 10https://gerrit.wikimedia.org/r/1339930 (https://phabricator.wikimedia.org/T437736) (owner: 10DLynch) [19:02:22] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate Developer Portal prototype to Gitlab production version - https://phabricator.wikimedia.org/T343261#12319564 (10Mooeypoo) [19:07:04] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [19:16:04] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [19:44:57] (03CR) 10Jforrester: "This isn't in the train? This is production CI config. Did you mean this comment for a different patch?" [integration/config] - 10https://gerrit.wikimedia.org/r/1339930 (https://phabricator.wikimedia.org/T437736) (owner: 10DLynch) [19:45:26] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/MobileApp] Add VisualEditor phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1339930 (https://phabricator.wikimedia.org/T437736) (owner: 10DLynch) [19:45:42] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/MobileApp] Add VisualEditor phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1339930 (https://phabricator.wikimedia.org/T437736) (owner: 10DLynch) [19:45:59] (03CR) 10DLynch: "It's blocking Ia6111c3cbb02c747360cb2be09f0160ec430eecf which needs the phan config to know about VisualEditor so CI can pass, which does " [integration/config] - 10https://gerrit.wikimedia.org/r/1339930 (https://phabricator.wikimedia.org/T437736) (owner: 10DLynch) [19:46:42] (03CR) 10Jforrester: [C:03+2] "Aha, right. Let's see if this works." [integration/config] - 10https://gerrit.wikimedia.org/r/1339930 (https://phabricator.wikimedia.org/T437736) (owner: 10DLynch) [19:50:42] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/MobileApp] Add VisualEditor phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1339930 (https://phabricator.wikimedia.org/T437736) (owner: 10DLynch) [19:50:59] !log Zuul: [mediawiki/extensions/MobileApp] Add VisualEditor phan dep, for T437736 [19:51:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:51:02] T437736: Visual Editor: Create public edit tag highlighting an edit came from VE in the app - https://phabricator.wikimedia.org/T437736 [20:08:51] (03CR) 10Mhurd: [C:03+1] castor: load the cache without the delayed rename pass (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1330226 (https://phabricator.wikimedia.org/T436173) (owner: 10Phedenskog) [21:00:29] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 13Patch-For-Review: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12320080 (10tstarling) >>! In T437829#12317118, @MoritzMuehlenhoff wrote: > Looking at https://github.com/debuerreotype/debuerreotype/issu... [21:11:49] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services, 13Patch-For-Review: Back up Zuul encryption keys - https://phabricator.wikimedia.org/T436131#12320112 (10Dzahn) @dduvall I am making sure /etc/zuul/backup exists outside the container, is only readable by the zuul user itself,... [21:35:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cp-upload09 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:35:10] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cp-upload09 on project deployment-prep - https://phabricator.wikimedia.org/T437958 (10wmcs-alerts) 03NEW [22:15:03] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cp-upload09 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:48:28] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12320355 (10Southparkfan) NOTE: TL;DR: Beta Clu... [23:13:03] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cp-upload09 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:35:44] 10Beta-Cluster-Infrastructure, 06SRE, 06Traffic: Name new CDN servers deployment-cp-(text|upload)0x instead of deployment-cache-(text|upload)0x - https://phabricator.wikimedia.org/T280393#12320486 (10BCornwall) 05Open→03Resolved a:03BCornwall deployment-cp-text09 and deployment-cp-upload09 are up a... [23:53:03] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-cp-upload09 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:54:41] 10Beta-Cluster-Infrastructure, 06Traffic, 10Cloud-VPS (Debian Bullseye Deprecation): Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12320526 (10BCornwall) deployment-cp-text09 and deployment-cp-upload09 are provisioned and ready for...