[01:12:13] (03CR) 10Jforrester: [C:03+2] Docker: [helm-linter] Bump for Envoy 1.35.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1250710 (https://phabricator.wikimedia.org/T419637) (owner: 10RLazarus) [01:13:59] (03Merged) 10jenkins-bot: Docker: [helm-linter] Bump for Envoy 1.35.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1250710 (https://phabricator.wikimedia.org/T419637) (owner: 10RLazarus) [01:14:23] !log Docker: [helm-linter] Bump for Envoy 1.35.9, for T419637 [01:14:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:14:26] T419637: Upgrade Envoy to v1.35.9 - https://phabricator.wikimedia.org/T419637 [01:19:13] 06Project-Admins: Create project tag for zhwikisource CJOPP - https://phabricator.wikimedia.org/T419782 (10Supergrey1) 03NEW [01:26:05] (03CR) 10Jforrester: [C:03+2] "Deployed and I re-ran https://integration.wikimedia.org/ci/job/helm-lint/30672/console as https://integration.wikimedia.org/ci/job/helm-li" [integration/config] - 10https://gerrit.wikimedia.org/r/1250711 (https://phabricator.wikimedia.org/T419637) (owner: 10RLazarus) [01:28:07] (03Merged) 10jenkins-bot: jjb: [helm-lint] Update to helm-linter:0.7.8 to pick up envoy-future 1.35.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1250711 (https://phabricator.wikimedia.org/T419637) (owner: 10RLazarus) [01:29:00] (03PS1) 10Jforrester: Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest [integration/config] - 10https://gerrit.wikimedia.org/r/1250762 [01:29:54] (03CR) 10Jforrester: "@hashar@free.fr, wanted your thoughts on this before landing it just to check." [integration/config] - 10https://gerrit.wikimedia.org/r/1250762 (owner: 10Jforrester) [01:42:53] 06Project-Admins, 07Chinese-Sites: Create project tag for zhwikisource CJOPP - https://phabricator.wikimedia.org/T419782#11700495 (10Supergrey1) [07:18:37] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for GinnyTheCat - https://phabricator.wikimedia.org/T419776#11700762 (10Aklapper) 05Open→03Resolved a:03Aklapper Approved. [07:19:20] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Kengkong1 - https://phabricator.wikimedia.org/T419773#11700767 (10Aklapper) Thanks for your interest! Which project(s) hosted in Wikimedia GitLab do you plan to work on? [07:21:29] 06Project-Admins, 06Release-Engineering-Team (Doing 😎), 07Chinese-Sites: Create project tag for zhwikisource CJOPP - https://phabricator.wikimedia.org/T419782#11700781 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #china-judgments-online-preservation-program has been created:... [07:48:00] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: ProbeDown - Service gerrit2003:443 has failed probes - https://phabricator.wikimedia.org/T419714#11700854 (10ABran-WMF) I've tried to repeatedly fetch (with a 1s interval) [[ https://gerrit.wikimedia.org/r/changes/?O=10000a&q=status%3Aopen%20con... [08:10:50] (03CR) 10Hashar: Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1250762 (owner: 10Jforrester) [08:15:56] (03PS2) 10Hashar: Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest [integration/config] - 10https://gerrit.wikimedia.org/r/1250762 (owner: 10Jforrester) [11:20:20] (03PS5) 10Zfilipin: Add devcontainer.json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T418234) [11:20:21] (03CR) 10Zfilipin: "PS5 adds setenv to tox.ini." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T418234) (owner: 10Zfilipin) [11:21:32] (03CR) 10Zfilipin: Add devcontainer.json (033 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T418234) (owner: 10Zfilipin) [11:21:47] (03CR) 10Zfilipin: Add devcontainer.json (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T418234) (owner: 10Zfilipin) [13:42:45] (03PS1) 10Hashar: dockerfiles: quibble-bullseye: fix run as different user [integration/config] - 10https://gerrit.wikimedia.org/r/1251084 (https://phabricator.wikimedia.org/T397810) [13:50:33] (03CR) 10Zfilipin: "🎉" [integration/config] - 10https://gerrit.wikimedia.org/r/1251084 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [13:50:45] (03CR) 10Jforrester: "Don't you need to cascade this to all the Quibble images?" [integration/config] - 10https://gerrit.wikimedia.org/r/1251084 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [13:58:57] (03open) 10sebastian-berlin-wmse: Add Pronlex to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/151 [14:22:08] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T413810#11702400 (10cscott) [14:22:53] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T413810#11702403 (10cscott) [14:23:38] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T413810#11702412 (10cscott) [14:36:50] 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.20; 2026-03-17), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11702462 (10Tgr) IIUIC this is su... [14:48:37] 06Release-Engineering-Team (Priority Backlog 📥), 10Release Pipeline (Blubber): blubber: Pin Python to setuptools 81 for now? - https://phabricator.wikimedia.org/T418253#11702594 (10MoritzMuehlenhoff) @dduvall @thcipriani Thanks so much for the quick solution. I've merged Tyler's patch and can confirm that it f... [14:55:53] 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.20; 2026-03-17), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11702604 (10Daimona) >>! In T4191... [14:57:31] 06Release-Engineering-Team (Priority Backlog 📥), 10Release Pipeline (Blubber): blubber: Allow users to specify version constraints for core python packages - https://phabricator.wikimedia.org/T418253#11702626 (10thcipriani) [14:59:03] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 10Release Pipeline (Blubber): blubber: Allow users to specify version constraints for core python packages - https://phabricator.wikimedia.org/T418253#11702642 (10thcipriani) 05Open→03Resolved a:03dduvall >>! In T418253#11702594, @MoritzMuehle... [15:01:49] (03merge) 10dancy: Add Pronlex to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/151 (owner: 10sebastian-berlin-wmse) [15:05:31] 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.20; 2026-03-17), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11702678 (10Tgr) Is the fix easy... [15:09:44] 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.20; 2026-03-17), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11702721 (10Daimona) >>! In T4191... [15:41:28] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:41:37] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T419876 (10wmcs-alerts) 03NEW [15:46:28] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:13:44] 10GitLab (Project Migration), 10Wikispeech-Jobrunner, 10Wikispeech-Text-to-Speech: Move Speechoid components to Gitlab - https://phabricator.wikimedia.org/T360758#11703186 (10Sebastian_Berlin-WMSE) [16:21:08] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Put a limit on demos created by ci - https://phabricator.wikimedia.org/T417304#11703214 (10thcipriani) [16:23:02] 10Continuous-Integration-Config, 10MobileFrontend, 06Reader Growth Team: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043#11703218 (10HSwan-WMF) p:05Triage→03Low [17:13:17] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: NDA ticket access for user kreid_WMF - https://phabricator.wikimedia.org/T419602#11703427 (10Aklapper) 05Open→03Resolved a:03Aklapper Added. [17:37:35] 06Release-Engineering-Team (Radar), 06SRE Observability (FY2025/2026-Q3): Upgrade Observability mwlog hosts to trixie - https://phabricator.wikimedia.org/T417002#11703513 (10herron) Thanks! I've updated the docs at https://wikitech.wikimedia.org/wiki/Wikimedia_binaries#mwlog_host from "probably mwlog[12]002"... [18:24:26] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T413810#11703685 (10brennen) [19:01:26] (03open) 10fab: dding: traffic pattern streaming [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/152 (https://phabricator.wikimedia.org/T415637) [20:16:59] (03merge) 10dancy: dding: traffic pattern streaming [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/152 (https://phabricator.wikimedia.org/T415637) (owner: 10fab) [20:59:11] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11704383 (10brennen) [21:20:17] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T419876#11704453 (10bd808) `lang=shell-session $ ssh deployment-sessionstore06.deployment-prep.eqiad1.wikimedia.cloud Linux deployment-sessionstore06 5.10.0-35-cloud-amd64 #1 SMP... [21:22:14] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T419876#11704462 (10bd808) The instance went non-responsive on me. The Horizon console shows: ` [13099164.309067] Out of memory: Killed process 1648239 (java) total-vm:2922196kB,... [21:22:35] 10Beta-Cluster-Infrastructure: Caassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11704464 (10bd808) [21:22:37] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T419876#11704466 (10bd808) →14Duplicate dup:03T415021 [21:23:38] !log Hard reboot deployment-sessionstore06 (T415021) [21:23:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:23:41] T415021: Caassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021 [21:24:28] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:24:34] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T419909 (10wmcs-alerts) 03NEW [21:24:43] sigh [21:28:04] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T419909#11704511 (10bd808) 05Open→03Invalid This was caused by my hard reboot of the system while it was struggling with OOM issues. [21:29:28] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:41:03] 10Beta-Cluster-Infrastructure: Caassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11704550 (10bd808) Something strange looking in the `sudo journalctl --since "2026-03-12 15:35:00" --until "2026-03-12 21:23:00"` o... [23:32:25] 10GitLab (Account Approval): Request to unlock GitLab account for contributing to CampWiz (GSoC 2026) - https://phabricator.wikimedia.org/T419896#11704807 (10brennen) 05Open→03Resolved a:03brennen Approved - happy hacking! [23:33:58] 06Release-Engineering-Team (Radar), 06SRE Observability (FY2025/2026-Q3): Upgrade Observability mwlog hosts to trixie - https://phabricator.wikimedia.org/T417002#11704811 (10herron) [23:34:07] 06Release-Engineering-Team (Radar), 06SRE Observability (FY2025/2026-Q3): Upgrade Observability mwlog hosts to trixie - https://phabricator.wikimedia.org/T417002#11704812 (10herron) 05Open→03Resolved