[00:07:00] 10Beta-Cluster-Infrastructure, 06Traffic, 10Cloud-VPS (Debian Bullseye Deprecation): Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12320556 (10BCornwall) a:03BCornwall [00:13:40] (03PS1) 10Pwangai: Only run QUnit when the change can affect it [integration/quibble] - 10https://gerrit.wikimedia.org/r/1341421 (https://phabricator.wikimedia.org/T437963) [01:49:03] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 13Patch-For-Review: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12320689 (10tstarling) I filed an issue and a PR https://github.com/debuerreotype/debuerreotype/issues/201 [05:19:01] !log devel-stats upgraded datasette-dashboards from 0.7.1 to 0.8.0 for releng-data.wmcloud.org [05:19:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:28:35] (03open) 10phedenskog: Show rolling p50/p75 for easier understanding how we are doing [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/66 [05:28:46] (03merge) 10phedenskog: Show rolling p50/p75 for easier understanding how we are doing [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/66 [05:49:58] (03merge) 10phedenskog: core: Show cache hit counts in the gate execution time tooltip [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/64 [06:31:38] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/ThemeToggle] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1341630 [06:31:38] (03CR) 10QChris: [V:03+2 C:03+2] Allow “Gerrit Managers” to import history [extensions/ThemeToggle] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1341630 (owner: 10QChris) [06:31:44] (03PS1) 10QChris: Import done. Revoke import grants [extensions/ThemeToggle] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1341631 [06:31:44] (03CR) 10QChris: [V:03+2 C:03+2] Import done. Revoke import grants [extensions/ThemeToggle] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1341631 (owner: 10QChris) [06:33:46] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839#12320913 (10Nikerabbit) Testwiki is still on wmf.19. Something failed? [06:47:12] (03open) 10phedenskog: Add index to speedup browser/core dashboards [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/67 [06:50:53] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839#12320921 (10Aklapper) Yes, `train-presync` failed with `No space left on device`. [06:52:42] (03merge) 10phedenskog: Add index to speedup browser/core dashboards [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/67 [06:58:52] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839#12320924 (10jnuche) After `train-presync` failed, our image cleanup kicked in and freed up some space. Hopefully that's enough nex... [07:10:10] (03open) 10phedenskog: Show 7-day time to green next to the 14-day tiles for core dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/68 [07:10:14] (03merge) 10phedenskog: Show 7-day time to green next to the 14-day tiles for core dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/68 [08:14:56] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839#12321114 (10jnuche) [08:26:26] 10Phabricator, 06Collaboration-Services: consider moving aphlict admin port to https / envoy - https://phabricator.wikimedia.org/T340169#12321153 (10ABran-WMF) [08:26:31] 10Phabricator, 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Move aphlict to k8s - https://phabricator.wikimedia.org/T436657#12321152 (10ABran-WMF) [08:35:47] (03open) 10jelto: digitalocean: create dedicated vpc and nat_gateway for each cluster [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/631 (https://phabricator.wikimedia.org/T425441) [08:46:12] (03open) 10phedenskog: Add time to green histogram and run counters to the exporter [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/69 [09:00:12] (03CR) 10Hashar: [C:03+2] Zuul: Add StandardDialogs as transitive dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1338146 (owner: 10Hslater) [09:03:15] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839#12321356 (10jnuche) [09:03:28] (03Merged) 10jenkins-bot: Zuul: Add StandardDialogs as transitive dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1338146 (owner: 10Hslater) [09:04:03] ok, blocker has been removed. I'm going ahead with the train [09:07:37] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1338146 (owner: 10Hslater) [09:07:47] jnuche: !!! :-] [09:10:17] (03update) 10jelto: digitalocean: create dedicated vpc and nat_gateway for each cluster [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/631 (https://phabricator.wikimedia.org/T425441) [09:12:30] (03update) 10jelto: digitalocean: create dedicated vpc and nat_gateway for each cluster [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/631 (https://phabricator.wikimedia.org/T425441) [09:12:53] hashar: ? [09:13:19] 10Phabricator, 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Move aphlict to k8s - https://phabricator.wikimedia.org/T436657#12321388 (10ABran-WMF) [09:13:37] ah, wrong channel [09:17:01] jnuche: sorry I was just celebrating the train blocker got disposed off and you can continue rolling it :) [09:17:58] hashar: magnifique, merci :) [09:19:14] (03update) 10phedenskog: Add time to green histogram and run counters to the exporter [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/69 [09:26:36] (03CR) 10Hashar: [C:03+2] "Had we wanted to keep them, we could have set `logs-dir = /log`  which, on CI, is bind mounted on the host workspace and collected at end " [integration/config] - 10https://gerrit.wikimedia.org/r/1338389 (https://phabricator.wikimedia.org/T437376) (owner: 10Mhurd) [09:28:24] (03Merged) 10jenkins-bot: dockerfiles: keep npm debug logs out of the castor cache [integration/config] - 10https://gerrit.wikimedia.org/r/1338389 (https://phabricator.wikimedia.org/T437376) (owner: 10Mhurd) [09:30:20] (03PS1) 10Hashar: jjb: update npm jobs to drop debug logs from cache [integration/config] - 10https://gerrit.wikimedia.org/r/1341713 (https://phabricator.wikimedia.org/T437376) [09:34:46] Hi, I have a question about kokkuri and docker images building. As far as I understood, `.kokkuri:build-and-publish-image` publishes the image with the name that equals the name of the GitLab repository. [09:36:01] We now have a situation where we prepare the upgrade for a piece of software, and rebuilding not-yet-production image a lot (with a separate tag) https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/.gitlab-ci.yml?ref_type=heads#L42 [09:37:34] Production images are build with the version and the `latest` tags [09:37:41] but not as frequently [09:38:46] I worry that we can accidentally evict all old `airflow-2.*`images from registry by continuing building new `airflow-3* images here https://docker-registry.wikimedia.org/repos/data-engineering/airflow-dags/tags/ [09:40:06] so the questions are: 1) is this worry warranted, or the registry will keep at least one old image because it is tagged as latest, and 2) should it be better to have major versions as different images, and does kokkuri support it? [09:43:06] I'm tracking it in https://phabricator.wikimedia.org/T437994 [09:56:30] pinging andre hashar jnuche [09:57:02] atsukoito: so it is not a kokkuri issue but a question how images are garbage collected from the docker registry isn't it? [09:57:21] hashar: it is both, i guess :3 [09:57:45] if docker images are garbage-collected, I'll need to update the image name [09:57:58] well you generate images airflow-2 / airflow-3 with tags latest and some date. The production images are not build often but you expect them to be available for deployment once built [09:58:43] and if you worry the old stable airflow-2 could be garbage collected, I guess there might be rules added to prevent that somehow [09:58:53] i generate airflow-2 and airflow-3 images as a tags, not as different images, is this approach even correct? [09:58:56] but I don't know how they are garbage collected [09:59:42] (in terms of kokkuri use case i mean.. does kokkuri even allow images to be named differently?) [09:59:51] no clue :) [10:00:07] I see, I'll look into it [10:00:55] !log Updating node based Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/1341713 | update npm jobs to drop debug logs from cache # T437376 T426741 [10:00:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:01:00] T437376: Keep npm debug logs out of the castor cache - https://phabricator.wikimedia.org/T437376 [10:01:00] T426741: Only save/sync caches if something has changed - https://phabricator.wikimedia.org/T426741 [10:01:12] (03CR) 10Hashar: [C:03+2] jjb: update npm jobs to drop debug logs from cache [integration/config] - 10https://gerrit.wikimedia.org/r/1341713 (https://phabricator.wikimedia.org/T437376) (owner: 10Hashar) [10:01:42] (03CR) 10Hashar: [C:03+2] "Jobs updated by I8b063a151e888ca6b63c23f0efa12ca6e4798dbf" [integration/config] - 10https://gerrit.wikimedia.org/r/1338389 (https://phabricator.wikimedia.org/T437376) (owner: 10Mhurd) [10:02:11] hashar: btullis told me kokkuri can do multiple image names and in fact we do it for other project, https://gitlab.wikimedia.org/repos/data-engineering/spark/-/blob/main/.gitlab-ci.yml#L28 [10:02:13] atsukoito: for the garbage collection, I think the registry is managed by SRE Infrastructure Foundations ( IRC: #wikimedia-sre-foundationsconnect ) [10:02:20] ah great [10:02:22] thanks, I'll ask them [10:02:50] I think they did some cleanup of /releng/* images based on access logs [10:03:06] and are currently disposing of any production images based on the old Debian Buster [10:03:41] (03CR) 10Hashar: [C:03+2] Zuul: [IssueTrackerLinks] Add OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/1338148 (owner: 10Hslater) [10:04:37] (03CR) 10CI reject: [V:04-1] jjb: update npm jobs to drop debug logs from cache [integration/config] - 10https://gerrit.wikimedia.org/r/1341713 (https://phabricator.wikimedia.org/T437376) (owner: 10Hashar) [10:06:09] (03CR) 10Hashar: [C:03+2] "Build failed to IO / docker registry slowness" [integration/config] - 10https://gerrit.wikimedia.org/r/1341713 (https://phabricator.wikimedia.org/T437376) (owner: 10Hashar) [10:06:25] (03Merged) 10jenkins-bot: Zuul: [IssueTrackerLinks] Add OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/1338148 (owner: 10Hslater) [10:07:29] (03Merged) 10jenkins-bot: jjb: update npm jobs to drop debug logs from cache [integration/config] - 10https://gerrit.wikimedia.org/r/1341713 (https://phabricator.wikimedia.org/T437376) (owner: 10Hashar) [10:08:15] (03CR) 10Hashar: "Deployed and I did a recheck of I362bec3ad0bc72e651f6fad390c8b08b03f04220" [integration/config] - 10https://gerrit.wikimedia.org/r/1338148 (owner: 10Hslater) [10:08:43] (03CR) 10Hashar: [V:03+2 C:03+2] jjb: update npm jobs to drop debug logs from cache [integration/config] - 10https://gerrit.wikimedia.org/r/1341713 (https://phabricator.wikimedia.org/T437376) (owner: 10Hashar) [10:09:30] (03CR) 10Hashar: [C:03+2] build: Updating npm dependencies [integration/docroot] - 10https://gerrit.wikimedia.org/r/1338318 (owner: 10Libraryupgrader) [10:38:32] !log ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/Cypress/ # corrupt Cypress cache? T438002 [10:38:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:38:35] T438002: WikibaseQualityConstraints wdio/cypress tests fail - https://phabricator.wikimedia.org/T438002 [11:47:26] (03open) 10phedenskog: Fold the core intro into a plain-words explanation [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/70 [11:47:31] (03merge) 10phedenskog: Fold the core intro into a plain-words explanation [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/70 [11:47:50] (03merge) 10phedenskog: Add time to green histogram and run counters to the exporter [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/69 [13:32:45] (03CR) 10Hashar: [C:03+2] inference-services: Stop config.yaml changes from triggering all pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/1341162 (https://phabricator.wikimedia.org/T437882) (owner: 10Ozge) [13:34:56] (03Merged) 10jenkins-bot: inference-services: Stop config.yaml changes from triggering all pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/1341162 (https://phabricator.wikimedia.org/T437882) (owner: 10Ozge) [14:06:28] (03open) 10brennen: Update submodules for 2026-09-15 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/120 (https://phabricator.wikimedia.org/T437930) [14:06:47] andre: https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/120 - did i miss anything? [14:07:35] brennen, thanks. should be correct [14:11:58] (03CR) 10Hashar: [C:03+2] Remove the per-project browser tests duration [integration/quibble] - 10https://gerrit.wikimedia.org/r/1340319 (https://phabricator.wikimedia.org/T437752) (owner: 10Phedenskog) [14:33:55] 10GitLab, 06Release-Engineering-Team, 06Data-Engineering: Trusted-runner access for developer-activity-pipelines (publish a container image) - https://phabricator.wikimedia.org/T438042 (10Antoine_Quhen) 03NEW [14:34:16] (03Merged) 10jenkins-bot: Remove the per-project browser tests duration [integration/quibble] - 10https://gerrit.wikimedia.org/r/1340319 (https://phabricator.wikimedia.org/T437752) (owner: 10Phedenskog) [14:34:17] (03PS1) 10Matthias Mullie: zuul: Add CirrusSearch phan dependency for ReaderExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1341911 [14:35:40] (03PS2) 10Vaughn Walters: jjb: [selenium-daily-beta-*] Delete the betacluster daily Selenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1341908 (https://phabricator.wikimedia.org/T435647) [14:38:04] (03CR) 10Jforrester: "<3" [integration/config] - 10https://gerrit.wikimedia.org/r/1341908 (https://phabricator.wikimedia.org/T435647) (owner: 10Vaughn Walters) [14:38:47] (03open) 10aqu: Add developer-activity-pipelines [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/189 (https://phabricator.wikimedia.org/T433696 https://phabricator.wikimedia.org/T438042) [14:38:51] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade): Delete unused Jenkins jobs - https://phabricator.wikimedia.org/T383134#12323309 (10hashar) 05Open→03Resolved a:03hashar That was a routine/yearly cleanup. [14:39:05] (03PS3) 10Vaughn Walters: jjb: [selenium-daily-beta-*] Delete the betacluster daily Selenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1341908 (https://phabricator.wikimedia.org/T435647) [14:39:13] (03update) 10aqu: Add developer-activity-pipelines [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/189 (https://phabricator.wikimedia.org/T433696 https://phabricator.wikimedia.org/T438042) [14:39:20] 10Continuous-Integration-Config, 10QS-Test-Automation, 10ST5.2.1 Daily Test Environment, 13Patch-For-Review: Delete betacluster daily selenium tests - https://phabricator.wikimedia.org/T435647#12323316 (10hashar) [14:39:27] (03CR) 10Vaughn Walters: jjb: [selenium-daily-beta-*] Delete the betacluster daily Selenium jobs (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1341908 (https://phabricator.wikimedia.org/T435647) (owner: 10Vaughn Walters) [14:40:09] (03update) 10aqu: Add developer-activity-pipelines [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/189 (https://phabricator.wikimedia.org/T433696 https://phabricator.wikimedia.org/T438042) [14:46:08] (03CR) 10Hashar: jjb: [selenium-daily-beta-*] Delete the betacluster daily Selenium jobs (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1341908 (https://phabricator.wikimedia.org/T435647) (owner: 10Vaughn Walters) [14:46:23] (03CR) 10Hashar: [C:03+2] jjb: [selenium-daily-beta-*] Delete the betacluster daily Selenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1341908 (https://phabricator.wikimedia.org/T435647) (owner: 10Vaughn Walters) [14:46:31] 10Phabricator, 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Move aphlict to k8s - https://phabricator.wikimedia.org/T436657#12323359 (10Dzahn) There was also https://gerrit.wikimedia.org/r/c/operations/puppet/+/1080823 but I abandoned it assuming it's pointless once on k8s. [14:47:36] (03CR) 10Hashar: [C:03+2] zuul: Add CirrusSearch phan dependency for ReaderExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1341911 (owner: 10Matthias Mullie) [14:49:54] (03Merged) 10jenkins-bot: jjb: [selenium-daily-beta-*] Delete the betacluster daily Selenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1341908 (https://phabricator.wikimedia.org/T435647) (owner: 10Vaughn Walters) [14:50:21] (03Merged) 10jenkins-bot: zuul: Add CirrusSearch phan dependency for ReaderExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1341911 (owner: 10Matthias Mullie) [15:01:00] 06Release-Engineering-Team, 10WikibaseMediaInfo: Consider enabling partial git clones from gerrit, to reduce load when doing sparse partial clones - https://phabricator.wikimedia.org/T437725#12323427 (10hashar) Can you tell more about the use case? WikibaseMediaInfo is a MediaWiki extension. As such in CI the... [15:22:06] 06Release-Engineering-Team, 10WikibaseMediaInfo: Consider enabling partial git clones from gerrit, to reduce load when doing sparse partial clones - https://phabricator.wikimedia.org/T437725#12323554 (10Jdforrester-WMF) > Can you tell more about the use case? https://gerrit.wikimedia.org/r/plugins/gitiles/med... [15:25:35] 10Continuous-Integration-Config, 10QS-Test-Automation, 10ST5.2.1 Daily Test Environment, 13Patch-For-Review: Delete betacluster daily selenium tests - https://phabricator.wikimedia.org/T435647#12323591 (10Jdforrester-WMF) 05Open→03Resolved https://integration.wikimedia.org/ci/ no longer has any tra... [15:26:55] 10Phabricator (2026-09-15), 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Deploy Phab/Phorge 2026-09-15 - https://phabricator.wikimedia.org/T437930#12323602 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=023da60b-d48a-42e2-b744-9acc3359ec22) set by jelto@cum... [15:30:26] (03merge) 10brennen: Update submodules for 2026-09-15 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/120 (https://phabricator.wikimedia.org/T437930) [15:30:57] (03PS2) 10Arendpieter: castor: Stop blocking builds on the cache save [integration/config] - 10https://gerrit.wikimedia.org/r/1341139 (https://phabricator.wikimedia.org/T352319) [15:31:24] (03PS1) 10Jforrester: jjb: Remove unused 'archive-tox-logs' macro (no-op) [integration/config] - 10https://gerrit.wikimedia.org/r/1341928 [15:31:39] (03CR) 10Jforrester: [C:03+2] jjb: Remove unused 'archive-tox-logs' macro (no-op) [integration/config] - 10https://gerrit.wikimedia.org/r/1341928 (owner: 10Jforrester) [15:33:12] (03Merged) 10jenkins-bot: jjb: Remove unused 'archive-tox-logs' macro (no-op) [integration/config] - 10https://gerrit.wikimedia.org/r/1341928 (owner: 10Jforrester) [15:34:41] 10Phabricator (2026-09-15), 06Release-Engineering-Team, 06Collaboration-Services: Deploy Phab/Phorge 2026-09-15 - https://phabricator.wikimedia.org/T437930#12323651 (10Malyacko) [15:34:44] 10Diffusion, 10Phabricator, 06Release-Engineering-Team, 07Performance Issue: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#12323652 (10Malyacko) [15:34:55] 10Diffusion, 10Phabricator, 06Release-Engineering-Team, 07Performance Issue: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#12323654 (10Malyacko) [15:34:55] 10Phabricator (2026-09-15), 06Release-Engineering-Team, 06Collaboration-Services: Deploy Phab/Phorge 2026-09-15 - https://phabricator.wikimedia.org/T437930#12323653 (10Malyacko) [15:36:38] 10Phabricator (2026-09-15), 06Release-Engineering-Team (Doing 😎), 06Collaboration-Services: Deploy Phab/Phorge 2026-09-15 - https://phabricator.wikimedia.org/T437930#12323691 (10Aklapper) 05Open→03Resolved Phab seems to be still up. [15:37:51] 10Phabricator (2026-09-15), 06Release-Engineering-Team (Doing 😎): Add actor membership condition for "Restrict maximum number of connected objects when editing edges" - https://phabricator.wikimedia.org/T375122#12323702 (10Aklapper) 05Open→03Resolved This should be resolved now. [15:38:23] 10Phabricator (2026-09-15), 06Release-Engineering-Team (Doing 😎): Can't edit subtasks: "Too many relationships (25, of type "task.has-subtask"). Must be less than 20." - https://phabricator.wikimedia.org/T376301#12323712 (10Aklapper) 05Open→03Resolved This should be resolved now. [15:48:42] 10Phabricator (2026-09-15), 06Release-Engineering-Team (Doing 😎), 06Collaboration-Services, 07Essential-Work: Deploy Phab/Phorge 2026-09-15 - https://phabricator.wikimedia.org/T437930#12323809 (10brennen) [16:00:27] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12323927 (10komla) Hello @Jdforrester-WMF and @hashar any update on this? [16:20:56] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12324089 (10Jdforrester-WMF) Sorry, no, I've not had time to work on this. @hashar, can... [16:50:44] (03update) 10dancy: group-management: add retry to gitlab helper _get method [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/82 (https://phabricator.wikimedia.org/T423984) (owner: 10lenap) [16:51:17] (03merge) 10dancy: group-management: add retry to gitlab helper _get method [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/82 (https://phabricator.wikimedia.org/T423984) (owner: 10lenap) [17:06:19] Not sure which is the right IRC channel, but CI is currently blocked for gated extensions due to corrupted npm-data or npm-cache or something: https://phabricator.wikimedia.org/T438061 [17:28:12] (03merge) 10addshore: Keep existing linux/arm fallback behavior, but fix detection so it actually [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/662 (https://phabricator.wikimedia.org/T435037) [17:29:12] (03merge) 10addshore: Improve MWDD dashboard service detection performance. [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/663 (https://phabricator.wikimedia.org/T413248) [17:30:18] (03merge) 10addshore: Fix various update command failures and make improvements [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/661 [17:56:04] !log ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -fr /srv/castor/castor-mw-ext-and-skins/master/quibble-with-gated-extensions-vendor-mysql-php83 (T438061) [18:08:07] MichaelG_WMF: Cache cleared [18:09:52] 10Continuous-Integration-Config, 07Technical-Debt: integration/config Spring cleaning, September 2026 - https://phabricator.wikimedia.org/T438076 (10Jdforrester-WMF) 03NEW p:05Triage→03Medium [18:09:55] (03open) 10dancy: .gitlab-ci.yml: lint: Use python3-trixie image [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/632 [18:10:46] (03PS1) 10Jforrester: jjb, Zuul: [mediawiki/services/wikispeech/*] Archive the services [integration/config] - 10https://gerrit.wikimedia.org/r/1341986 (https://phabricator.wikimedia.org/T344741) [18:10:50] (03PS1) 10Jforrester: jjb, Zuul: [wikidata/query-builder, wikidata/query/gui] Archive repos [integration/config] - 10https://gerrit.wikimedia.org/r/1341987 (https://phabricator.wikimedia.org/T438076) [18:10:52] (03PS1) 10Jforrester: jjb, Zuul: [mediawiki/services/recommendation-api] Archive service [integration/config] - 10https://gerrit.wikimedia.org/r/1341988 (https://phabricator.wikimedia.org/T429123) [18:10:57] (03PS1) 10Jforrester: jjb, Zuul: [operations/container/miscweb] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341989 (https://phabricator.wikimedia.org/T438076) [18:10:59] (03PS1) 10Jforrester: jjb, Zuul: [node-rdkafka-factory, node-rdkafka-statsd] Archive repos [integration/config] - 10https://gerrit.wikimedia.org/r/1341990 (https://phabricator.wikimedia.org/T366611) [18:11:04] (03PS1) 10Jforrester: jjb, Zuul: [cloud/toolforge/*] Archive two Toolforge repositories [integration/config] - 10https://gerrit.wikimedia.org/r/1341991 (https://phabricator.wikimedia.org/T438076) [18:11:07] (03PS1) 10Jforrester: jjb, Zuul: [research/landing-page] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341992 (https://phabricator.wikimedia.org/T438076) [18:11:09] (03PS1) 10Jforrester: jjb, Zuul: [integration/gear] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341993 (https://phabricator.wikimedia.org/T289512) [18:11:14] (03PS1) 10Jforrester: jjb, Zuul: [labs/tools/ldap] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341994 (https://phabricator.wikimedia.org/T438076) [18:11:16] hashar: You closing T383134 inspired me to do a new pass. :-) [18:11:16] T383134: Delete unused Jenkins jobs - https://phabricator.wikimedia.org/T383134 [18:13:17] (03merge) 10dancy: .gitlab-ci.yml: lint: Use python3-trixie image [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/632 [18:13:57] (03PS1) 10SomeRandomDeveloper: Zuul: [mediawiki/extensions/ThemeToggle] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1341995 [18:14:34] (03CR) 10Jforrester: [C:03+2] jjb, Zuul: [mediawiki/services/wikispeech/*] Archive the services [integration/config] - 10https://gerrit.wikimedia.org/r/1341986 (https://phabricator.wikimedia.org/T344741) (owner: 10Jforrester) [18:16:08] (03Merged) 10jenkins-bot: jjb, Zuul: [mediawiki/services/wikispeech/*] Archive the services [integration/config] - 10https://gerrit.wikimedia.org/r/1341986 (https://phabricator.wikimedia.org/T344741) (owner: 10Jforrester) [18:16:27] !log Zuul: [mediawiki/services/wikispeech/*] Archive the services, for T344741 and T438076 [18:16:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:16:31] T344741: Migrate Wikispeech services to GitLab - https://phabricator.wikimedia.org/T344741 [18:16:32] T438076: integration/config Spring cleaning, September 2026 - https://phabricator.wikimedia.org/T438076 [18:17:09] (03CR) 10Jforrester: [C:03+2] jjb, Zuul: [wikidata/query-builder, wikidata/query/gui] Archive repos [integration/config] - 10https://gerrit.wikimedia.org/r/1341987 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [18:17:27] Could someone update https://en.wikipedia.beta.wmcloud.org/wiki/Main_Page and change "runs PHP 8.3?" to "runs PHP 8.5"? [18:18:53] (03Merged) 10jenkins-bot: jjb, Zuul: [wikidata/query-builder, wikidata/query/gui] Archive repos [integration/config] - 10https://gerrit.wikimedia.org/r/1341987 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [18:19:09] (03update) 10dancy: digitalocean: create dedicated vpc and nat_gateway for each cluster [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/631 (https://phabricator.wikimedia.org/T425441) (owner: 10jelto) [18:19:16] !log Zuul: [wikidata/query-builder, wikidata/query/gui] Archive repos, for T438076 [18:19:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:20:20] (03CR) 10Jforrester: [C:03+2] jjb, Zuul: [mediawiki/services/recommendation-api] Archive service [integration/config] - 10https://gerrit.wikimedia.org/r/1341988 (https://phabricator.wikimedia.org/T429123) (owner: 10Jforrester) [18:22:11] (03Merged) 10jenkins-bot: jjb, Zuul: [mediawiki/services/recommendation-api] Archive service [integration/config] - 10https://gerrit.wikimedia.org/r/1341988 (https://phabricator.wikimedia.org/T429123) (owner: 10Jforrester) [18:22:18] !log Zuul: [mediawiki/services/recommendation-api] Archive service, for T429123 and T438076 [18:22:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:22:23] T429123: Archive mediawiki/services/recommendation-api - https://phabricator.wikimedia.org/T429123 [18:22:23] T438076: integration/config Spring cleaning, September 2026 - https://phabricator.wikimedia.org/T438076 [18:22:27] (03CR) 10Jforrester: [C:03+2] jjb, Zuul: [operations/container/miscweb] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341989 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [18:24:07] (03Merged) 10jenkins-bot: jjb, Zuul: [operations/container/miscweb] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341989 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [18:24:22] !log Zuul: [operations/container/miscweb] Archive repository, for T438076 [18:24:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:24:31] (03CR) 10Jforrester: [C:03+2] jjb, Zuul: [node-rdkafka-factory, node-rdkafka-statsd] Archive repos [integration/config] - 10https://gerrit.wikimedia.org/r/1341990 (https://phabricator.wikimedia.org/T366611) (owner: 10Jforrester) [18:26:03] (03Merged) 10jenkins-bot: jjb, Zuul: [node-rdkafka-factory, node-rdkafka-statsd] Archive repos [integration/config] - 10https://gerrit.wikimedia.org/r/1341990 (https://phabricator.wikimedia.org/T366611) (owner: 10Jforrester) [18:26:06] (03CR) 10Jforrester: [C:03+2] jjb, Zuul: [cloud/toolforge/*] Archive two Toolforge repositories [integration/config] - 10https://gerrit.wikimedia.org/r/1341991 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [18:26:07] !log Zuul: [node-rdkafka-factory, node-rdkafka-statsd] Archive repos, for T366611 and T438076. [18:26:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:26:12] T366611: Migrate Data Engineering NodeJS library repos to GitLab - https://phabricator.wikimedia.org/T366611 [18:27:46] (03Merged) 10jenkins-bot: jjb, Zuul: [cloud/toolforge/*] Archive two Toolforge repositories [integration/config] - 10https://gerrit.wikimedia.org/r/1341991 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [18:28:07] !log Zuul: [cloud/toolforge/*] Archive two Toolforge repositories, for T438076 [18:28:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:28:10] T438076: integration/config Spring cleaning, September 2026 - https://phabricator.wikimedia.org/T438076 [18:28:12] (03CR) 10Jforrester: [C:03+2] jjb, Zuul: [research/landing-page] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341992 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [18:29:38] (03Merged) 10jenkins-bot: jjb, Zuul: [research/landing-page] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341992 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [18:29:47] !log Zuul: [research/landing-page] Archive repository, for T438076 [18:29:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:29:55] (03CR) 10Jforrester: [C:03+2] jjb, Zuul: [integration/gear] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341993 (https://phabricator.wikimedia.org/T289512) (owner: 10Jforrester) [18:32:02] (03Merged) 10jenkins-bot: jjb, Zuul: [integration/gear] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341993 (https://phabricator.wikimedia.org/T289512) (owner: 10Jforrester) [18:32:07] !log Zuul: [integration/gear] Archive repository, for T289512 and T438076 [18:32:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:32:12] T289512: Upgrade zuul gearman when upstream releases it - https://phabricator.wikimedia.org/T289512 [18:32:12] (03CR) 10Jforrester: [C:03+2] jjb, Zuul: [labs/tools/ldap] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341994 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [18:33:48] (03Merged) 10jenkins-bot: jjb, Zuul: [labs/tools/ldap] Archive repository [integration/config] - 10https://gerrit.wikimedia.org/r/1341994 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [18:33:53] !log Zuul: [labs/tools/ldap] Archive repository, for T438076 [18:33:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:33:55] T438076: integration/config Spring cleaning, September 2026 - https://phabricator.wikimedia.org/T438076 [18:34:30] (03CR) 10Ahmon Dancy: [C:03+1] castor: Stop blocking builds on the cache save [integration/config] - 10https://gerrit.wikimedia.org/r/1341139 (https://phabricator.wikimedia.org/T352319) (owner: 10Arendpieter) [18:45:32] (03PS1) 10Phedenskog: jjb: delete the workspace in the background [integration/config] - 10https://gerrit.wikimedia.org/r/1342006 (https://phabricator.wikimedia.org/T437022) [19:07:04] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [19:26:17] James_F: thx for the cleanups! [19:27:58] 10Beta-Cluster-Infrastructure, 06Traffic, 10Cloud-VPS (Debian Bullseye Deprecation): Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12324916 (10Southparkfan) Hi @BCornwall, thank you for your help! I have added the servers to `cache_... [19:29:27] (03merge) 10dancy: digitalocean: create dedicated vpc and nat_gateway for each cluster [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/631 (https://phabricator.wikimedia.org/T425441) (owner: 10jelto) [19:30:11] (03PS1) 10Jforrester: jjb: [quibble-*-selenium] Build only the MySQL PHP 8.3 and 8.5 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342018 (https://phabricator.wikimedia.org/T438076) [19:30:13] (03PS1) 10Jforrester: jjb: [quibble-*] Build PostgreSQL and SQLite only for PHP 8.3 and 8.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1342019 (https://phabricator.wikimedia.org/T438076) [19:30:15] (03PS1) 10Jforrester: jjb: [quibble-*-phpunit-standalone] Build only the MySQL jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342020 (https://phabricator.wikimedia.org/T438076) [19:30:18] (03PS1) 10Jforrester: jjb: [quibble-*-browser-tests-only] Build only the MySQL jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342021 (https://phabricator.wikimedia.org/T438076) [19:30:20] (03PS1) 10Jforrester: jjb: [quibble-apitests-only-*] Build only the PHP 8.3 and 8.5 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342022 (https://phabricator.wikimedia.org/T438076) [19:30:23] (03PS1) 10Jforrester: jjb: [quibble-for-mediawiki-core-*] Limit PostgreSQL and SQLite versions [integration/config] - 10https://gerrit.wikimedia.org/r/1342023 (https://phabricator.wikimedia.org/T438076) [19:33:02] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace Acme-chief hosts: deployment-acme-chief0[56] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436472#12324940 (10Southparkfan) Hi @BCornwall you have created `deployment-acme-chief07` and `deployment-acme-chief08`, and... [19:33:31] 06Project-Admins: Create project tag for ThemeToggle - https://phabricator.wikimedia.org/T438081 (10SomeRandomDeveloper) 03NEW [19:33:36] 06Project-Admins: Create project tag for ThemeToggle - https://phabricator.wikimedia.org/T438081#12324952 (10SomeRandomDeveloper) [19:34:32] (03CR) 10Jforrester: [C:03+2] jjb: [quibble-*-selenium] Build only the MySQL PHP 8.3 and 8.5 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342018 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:36:10] (03Merged) 10jenkins-bot: jjb: [quibble-*-selenium] Build only the MySQL PHP 8.3 and 8.5 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342018 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:36:11] (03CR) 10Jforrester: [C:03+2] jjb: [quibble-*] Build PostgreSQL and SQLite only for PHP 8.3 and 8.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1342019 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:36:17] (03CR) 10Jforrester: [C:03+2] jjb: [quibble-*-phpunit-standalone] Build only the MySQL jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342020 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:36:29] (03CR) 10Jforrester: [C:03+2] jjb: [quibble-apitests-only-*] Build only the PHP 8.3 and 8.5 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342022 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:36:36] (03CR) 10Jforrester: [C:03+2] jjb: [quibble-for-mediawiki-core-*] Limit PostgreSQL and SQLite versions [integration/config] - 10https://gerrit.wikimedia.org/r/1342023 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:37:30] 10Beta-Cluster-Infrastructure, 06Traffic, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12324962 (10BCornwall) Thanks, @Southparkfan ! I went ahead and moved the float... [19:37:45] (03Merged) 10jenkins-bot: jjb: [quibble-*] Build PostgreSQL and SQLite only for PHP 8.3 and 8.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1342019 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:37:51] (03Merged) 10jenkins-bot: jjb: [quibble-*-phpunit-standalone] Build only the MySQL jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342020 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:37:58] (03CR) 10Jforrester: [C:03+2] jjb: [quibble-*-browser-tests-only] Build only the MySQL jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342021 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:38:03] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-mwmaint03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [19:38:09] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-mwmaint03 in project deployment-prep - https://phabricator.wikimedia.org/T438082 (10wmcs-alerts) 03NEW [19:39:45] (03Merged) 10jenkins-bot: jjb: [quibble-*-browser-tests-only] Build only the MySQL jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342021 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:39:50] (03Merged) 10jenkins-bot: jjb: [quibble-apitests-only-*] Build only the PHP 8.3 and 8.5 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342022 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:39:52] (03Merged) 10jenkins-bot: jjb: [quibble-for-mediawiki-core-*] Limit PostgreSQL and SQLite versions [integration/config] - 10https://gerrit.wikimedia.org/r/1342023 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [19:41:03] (03PS1) 10Jforrester: Add a 'jjb-delete-job' script to delete jobs by name [integration/config] - 10https://gerrit.wikimedia.org/r/1342028 (https://phabricator.wikimedia.org/T438076) [19:41:53] That’s 142 Jenkins jobs dropped so far. [19:43:14] James_F: and Vaughn got rid of the selenium*-daily jobs that run browser tests against the beta cluster :party: [19:43:19] that got moved to Catalyst [19:43:56] 06Project-Admins: Create project tag for ThemeToggle - https://phabricator.wikimedia.org/T438081#12324983 (10Peachey88) 05Open→03Resolved a:03Peachey88 Requested project has been created: #MediaWiki-extensions-ThemeToggle [19:45:05] hashar: Yeah! Lots of slimming down. [19:45:26] hashar: Do you still need the hashar-dump-params and hashar-env-variables jobs? They’re defined only in CI, not in git. [19:45:46] you can drop those [19:45:56] I sometime create jobs manually for some debugging/testing [19:46:05] always prefix those with `hashar-` [19:46:19] Ack, will do. What about `T2999995 - cancel and postbuild`? [19:46:19] and then I sometime delete them but often forget [19:47:04] that is for https://phabricator.wikimedia.org/T299995 [19:47:12] I think upstream eventually got my patch merged [19:47:16] it is in my backlog [19:47:25] Too many 9s. :-) But OK. [19:47:42] yeah [19:47:50] just keep it around, it is handy to have around when I revisit the task [19:48:46] yeah that is https://phabricator.wikimedia.org/T352319 [19:48:50] and https://github.com/jenkinsci/postbuildscript-plugin/pull/219 ! [19:49:09] Ack. [19:49:19] Good luck getting upstream. [19:50:23] ah and https://github.com/jenkinsci/parameterized-trigger-plugin/pull/400 [19:50:30] there is a mix up between the plugins [19:50:45] and Java / Jenkins / plugins have slightly different semantics [19:50:55] Of course they do. [19:50:58] aka what are Interrupted / Cancelled / Aborted? :b [19:51:36] If only we could have more states of result, and fewer of process. :-) [19:53:05] OH [19:53:06] and https://github.com/jenkinsci/postbuildscript-plugin/pull/233 [19:53:14] that is the one that got merged recently [19:53:56] and I don't remember whether I got that one deployed [19:55:10] $ git tag --contains handle-cancelled-steps [19:55:10] 3.4.1.821.v08352159216f [19:55:58] I'll upgrade the plugin tomoror [19:55:59] ww [19:56:09] Yay. [19:56:49] “hashar marked this pull request as ready for review 2 years ago” // “Thanks so much for the contribution and sorry for the huge delay. I didn't get any notifications for this PR. Will change my settings.” Ah well. [19:57:06] We’ve definitely done much worse for MediaWiki patches. [19:57:07] yeah shit happens :] [19:57:24] I had excellent interactions with Jenkins upstream [19:57:31] their community is quite similar to us [19:57:40] Cloudbee (the company) maintaining a bunch of pro plugins [19:57:51] and offering CI / shared maintenance / security patches etc [19:57:56] hashar: maintenance-sample-workspace-sizes has been disabled since 2020. Can we drop it? [19:58:09] and a bunch of wild volunteers here and there maintaining a large set of plugins with different grade of quality [19:58:33] that job is dancy using Groovy to garbage collect stuff on the agents [19:58:47] maybe that specific job is no more used. It is a .groovy script in jjb [19:58:54] or maybe there is another job replacing it nowadays [19:59:01] Maybe? [20:00:01] no idea [20:00:03] oh [20:00:18] and on an other topic, we still have `jsduck` floating around! [20:00:29] Yeah. :-( [20:01:30] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate Developer Portal prototype to Gitlab production version - https://phabricator.wikimedia.org/T343261#12325053 (10Mooeypoo) [20:03:35] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate Developer Portal prototype to Gitlab production version - https://phabricator.wikimedia.org/T343261#12325065 (10Mooeypoo) [20:04:38] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate Developer Portal prototype to Gitlab production version - https://phabricator.wikimedia.org/T343261#12325068 (10Mooeypoo) Note: Decision was officially made to *not* have to import any wiki pages into markdown for the prototype. I r... [20:08:04] (03PS1) 10Jforrester: jjb: [wikidata-query-flink-swift-plugin] Delete the jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342034 (https://phabricator.wikimedia.org/T438076) [20:08:06] (03PS1) 10Jforrester: jjb: Delete two unused per-project tox jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342035 (https://phabricator.wikimedia.org/T438076) [20:08:46] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate Developer Portal prototype to Gitlab production version - https://phabricator.wikimedia.org/T343261#12325089 (10Mooeypoo) [20:08:59] (03PS2) 10Jforrester: jjb: Delete two unused per-project tox jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342035 (https://phabricator.wikimedia.org/T438076) [20:09:06] (03CR) 10Jforrester: [C:03+2] jjb: Delete two unused per-project tox jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342035 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [20:09:38] (03PS2) 10Jforrester: jjb: [wikidata-query-flink-swift-plugin] Delete the jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342034 (https://phabricator.wikimedia.org/T438076) [20:09:41] (03CR) 10Jforrester: [C:03+2] jjb: [wikidata-query-flink-swift-plugin] Delete the jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342034 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [20:10:41] (03Merged) 10jenkins-bot: jjb: Delete two unused per-project tox jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342035 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [20:11:08] (03Merged) 10jenkins-bot: jjb: [wikidata-query-flink-swift-plugin] Delete the jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342034 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [20:13:58] (03CR) 10Jforrester: [C:03+2] Add a 'jjb-delete-job' script to delete jobs by name [integration/config] - 10https://gerrit.wikimedia.org/r/1342028 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [20:15:00] James_F: then jsduck does not support ES5 so I guess it is mostly useless at this point :) [20:15:05] Indeed. [20:15:29] I’ll drop it some time this weekend. [20:15:39] (03Merged) 10jenkins-bot: Add a 'jjb-delete-job' script to delete jobs by name [integration/config] - 10https://gerrit.wikimedia.org/r/1342028 (https://phabricator.wikimedia.org/T438076) (owner: 10Jforrester) [20:15:41] https://codesearch.wmcloud.org/things/?q=jsduck&files=&excludeFiles=&repos= [20:15:57] includes DonationInterface or WikimediaCustomizations bah [20:17:05] then I don't know whether any of those docs are functionals or whether they are published anywhere [20:17:08] hashar: https://codesearch.wmcloud.org/things/?q=jsduck&files=&excludeFiles=%28%5C%2Flib%5C%2F%7C%5C.md%24%29&repos= makes it look better. But yes, DI’s code is bad, shock etc. [20:17:53] yeah it is never ending [20:18:09] anyway I am off, I gotta sleep a bit :] [20:18:28] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate Developer Portal prototype to Gitlab production version - https://phabricator.wikimedia.org/T343261#12325113 (10Mooeypoo) [20:32:20] 10Continuous-Integration-Infrastructure, 10LibUp, 10MediaWiki-Vagrant: Uninstall jsduck and related material from CI images etc., once it's no longer needed - https://phabricator.wikimedia.org/T363905#12325137 (10Jdforrester-WMF) [20:34:05] (03PS1) 10Mhurd: jjb: run extension browser tests without npm --prefix [integration/config] - 10https://gerrit.wikimedia.org/r/1342041 (https://phabricator.wikimedia.org/T437376) [20:35:46] 10Continuous-Integration-Infrastructure, 10Castor, 13Patch-For-Review, 06Test Platform (Arusha 30): Keep npm debug logs out of the castor cache - https://phabricator.wikimedia.org/T437376#12325146 (10Mhurd) Four namespaces on castor06 kept receiving npm debug logs after this morning's switch: the Wikibase... [20:36:06] (03PS1) 10Jforrester: Zuul: […/OOJsUIAjaxLogin] Drop the JavaScript documentation job [integration/config] - 10https://gerrit.wikimedia.org/r/1342043 (https://phabricator.wikimedia.org/T391706) [20:37:40] (03CR) 10Jforrester: [C:03+2] Zuul: […/OOJsUIAjaxLogin] Drop the JavaScript documentation job [integration/config] - 10https://gerrit.wikimedia.org/r/1342043 (https://phabricator.wikimedia.org/T391706) (owner: 10Jforrester) [20:39:20] (03Merged) 10jenkins-bot: Zuul: […/OOJsUIAjaxLogin] Drop the JavaScript documentation job [integration/config] - 10https://gerrit.wikimedia.org/r/1342043 (https://phabricator.wikimedia.org/T391706) (owner: 10Jforrester) [20:39:35] !log Zuul: […/OOJsUIAjaxLogin] Drop the JavaScript documentation job, for T391706 [20:39:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:39:38] T391706: Replace jsduck with JSDoc (or just drop it) in remaining extensions, skins, etc. - https://phabricator.wikimedia.org/T391706 [20:42:14] (03PS1) 10Jforrester: Docker: [quibble-bookworm, node24-test, node26-test] Drop jsduck and Ruby supports [integration/config] - 10https://gerrit.wikimedia.org/r/1342045 (https://phabricator.wikimedia.org/T363905) [20:42:24] (03CR) 10Jforrester: [C:03+2] Docker: [quibble-bookworm, node24-test, node26-test] Drop jsduck and Ruby supports [integration/config] - 10https://gerrit.wikimedia.org/r/1342045 (https://phabricator.wikimedia.org/T363905) (owner: 10Jforrester) [20:43:32] (03CR) 10CI reject: [V:04-1] Docker: [quibble-bookworm, node24-test, node26-test] Drop jsduck and Ruby supports [integration/config] - 10https://gerrit.wikimedia.org/r/1342045 (https://phabricator.wikimedia.org/T363905) (owner: 10Jforrester) [20:44:06] (03PS2) 10Jforrester: Docker: [quibble-bookworm, node24-test, node26-test] Drop jsduck etc. [integration/config] - 10https://gerrit.wikimedia.org/r/1342045 (https://phabricator.wikimedia.org/T363905) [20:44:09] (03CR) 10Jforrester: "…" [integration/config] - 10https://gerrit.wikimedia.org/r/1342045 (https://phabricator.wikimedia.org/T363905) (owner: 10Jforrester) [20:45:49] (03Merged) 10jenkins-bot: Docker: [quibble-bookworm, node24-test, node26-test] Drop jsduck etc. [integration/config] - 10https://gerrit.wikimedia.org/r/1342045 (https://phabricator.wikimedia.org/T363905) (owner: 10Jforrester) [20:46:42] !log Docker: [quibble-bookworm, node24-test, node26-test] Drop jsduck etc. for T363905 [20:46:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:46:46] T363905: Uninstall jsduck and related material from CI images etc., once it's no longer needed - https://phabricator.wikimedia.org/T363905 [20:47:06] 10Continuous-Integration-Infrastructure, 10LibUp, 10MediaWiki-Vagrant, 13Patch-For-Review: Uninstall jsduck and related material from CI images etc., once it's no longer needed - https://phabricator.wikimedia.org/T363905#12325187 (10Jdforrester-WMF) [20:47:28] 10Continuous-Integration-Infrastructure, 10LibUp, 10MediaWiki-Vagrant, 13Patch-For-Review: Uninstall jsduck and related material from CI images etc., once it's no longer needed - https://phabricator.wikimedia.org/T363905#12325188 (10Jdforrester-WMF) [20:54:11] 10Beta-Cluster-Infrastructure, 06Traffic, 10Cloud-VPS (Debian Bullseye Deprecation): Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12325202 (10BCornwall) Yay, thanks for the help deploying it. Regarding tofu integration, I'm not see... [20:59:32] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12325210 (10Dzahn) I have merged the following: [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1329228 | add spec tests ]] by hashar [[ h... [21:02:17] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace Acme-chief hosts: deployment-acme-chief0[56] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436472#12325213 (10BCornwall) 05Open→03Resolved a:03BCornwall [21:12:10] (03CR) 10Jforrester: [C:03+2] "Successfully published image docker-registry.discovery.wmnet/releng/node26-test:26.4.0-s4" [integration/config] - 10https://gerrit.wikimedia.org/r/1342045 (https://phabricator.wikimedia.org/T363905) (owner: 10Jforrester) [21:14:33] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [21:17:30] (03PS1) 10Jforrester: jjb: Update node and Quibble jobs for the jsduck removal [integration/config] - 10https://gerrit.wikimedia.org/r/1342051 (https://phabricator.wikimedia.org/T363905) [21:17:45] (03CR) 10Jforrester: [C:03+2] jjb: Update node and Quibble jobs for the jsduck removal [integration/config] - 10https://gerrit.wikimedia.org/r/1342051 (https://phabricator.wikimedia.org/T363905) (owner: 10Jforrester) [21:18:27] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12325235 (10Dzahn) Also amended https://gerrit.wikimedia.org/r/1322828 by Scott which intends to remove the zookeeper34 special case and class path... [21:18:38] 10Continuous-Integration-Infrastructure, 10MediaWiki-Vagrant, 13Patch-For-Review: Uninstall jsduck and related material from CI images etc., once it's no longer needed - https://phabricator.wikimedia.org/T363905#12325236 (10Jdforrester-WMF) [21:19:41] (03Merged) 10jenkins-bot: jjb: Update node and Quibble jobs for the jsduck removal [integration/config] - 10https://gerrit.wikimedia.org/r/1342051 (https://phabricator.wikimedia.org/T363905) (owner: 10Jforrester) [21:24:01] 10Continuous-Integration-Infrastructure, 10MediaWiki-Vagrant, 13Patch-For-Review: Uninstall jsduck and related material from CI images etc., once it's no longer needed - https://phabricator.wikimedia.org/T363905#12325242 (10Jdforrester-WMF) 05Stalled→03Resolved a:03Jdforrester-WMF Let's call this d... [21:26:26] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 22 to Node 24 - https://phabricator.wikimedia.org/T395926#12325260 (10Jdforrester-WMF) [21:26:53] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 22 to Node 24 - https://phabricator.wikimedia.org/T395926#12325267 (10Jdforrester-WMF) This has been done since April, oops. [21:26:59] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 22 to Node 24 - https://phabricator.wikimedia.org/T395926#12325268 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [21:40:41] (03PS1) 10Jforrester: Docker: [node26] Update Node from 26.4.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1342055 [21:40:41] (03PS1) 10Jforrester: jjb: Update Node from 26.4.0 to 26.8.2 for generic Node jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342056 [21:40:41] (03PS1) 10Jforrester: Docker: [quibble-bookworm] Update Node from 24.18.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1342057 (https://phabricator.wikimedia.org/T438085) [21:40:43] (03PS1) 10Jforrester: jjb: Update Node from 24.18.0 to 26.8.2 for Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342058 (https://phabricator.wikimedia.org/T438085) [21:40:45] (03PS1) 10Jforrester: Zuul: Migrate MediaWiki-land independent CI Node from 24 to 26 [integration/config] - 10https://gerrit.wikimedia.org/r/1342059 (https://phabricator.wikimedia.org/T438085) [21:42:28] (03CR) 10CI reject: [V:04-1] Docker: [quibble-bookworm] Update Node from 24.18.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1342057 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [21:43:12] (03CR) 10CI reject: [V:04-1] jjb: Update Node from 24.18.0 to 26.8.2 for Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342058 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [21:43:26] (03CR) 10CI reject: [V:04-1] Zuul: Migrate MediaWiki-land independent CI Node from 24 to 26 [integration/config] - 10https://gerrit.wikimedia.org/r/1342059 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [22:01:52] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12325363 (10Dzahn) zookeeper restarted on zuul1001 and it's now logging: snippet from `/var/log/zookeeper/zookeeper.log` ` 2026-09-15 22:01:03,07... [22:02:22] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12325364 (10Dzahn) 05Open→03Resolved [22:04:49] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12325367 (10Dzahn) restarted zookeeper on zuul1001,zuul1004,zuul2001 and all three have non-empty logfile [23:01:03] (03CR) 10Finchgold: [C:03+1] Zuul: [mediawiki/extensions/ThemeToggle] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1341995 (owner: 10SomeRandomDeveloper) [23:29:55] 06Project-Admins: Create project tag for DataMaps - https://phabricator.wikimedia.org/T438099 (10SomeRandomDeveloper) 03NEW