[00:02:05] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: navtiming is broken on the Beta Cluster because kafka-jumbo has been removed - https://phabricator.wikimedia.org/T445011#12585465 (10Southparkfan) After switching to `main-deployment-prep`, navtiming no longer spams the logs. How to get this patch merged? [00:03:31] Yippee, build fixed! [00:03:31] Project mediawiki-core-doxygen build #24455: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/24455/ [00:18:58] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12585530 (10bd808) Follow up tasks: * {T445599} * {T445600} * {T445602} * {T445601} [00:19:44] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12585535 (10bd808) 05In progress→03Resolved p:05Triage→03High [00:32:46] !log Change deployment-docker-mobileapps02 hiera to point to deployment-mediawiki16 instead of deployment-mediawiki14 [00:32:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:48:51] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T442224#12585585 (10SecurityPatchBot) [00:48:57] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T442224#12585594 (10SecurityPatchBot) [01:08:31] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics: Backfill data missed while systems were down October-November 2026 - https://phabricator.wikimedia.org/T445599#12585659 (10bd808) a:03thcipriani [04:06:49] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for tduk - https://phabricator.wikimedia.org/T445610 (10Pharos) 03NEW [06:34:46] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Requesting access to WMF-NDA project in Phabricator for cjming (Clare Ming) - https://phabricator.wikimedia.org/T445138#12586006 (10Aklapper) 05Open→03Resolved a:03Aklapper Indeed, added. [06:36:20] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Add CMassaro to #WMF-NDA in Phab - https://phabricator.wikimedia.org/T445264#12586015 (10Aklapper) 05Open→03Resolved a:03Aklapper [06:51:27] 10Gerrit (Gerrit 3.12), 13Patch-For-Review: Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#12586039 (10hashar) [06:52:06] 10Gerrit (Gerrit 3.12), 13Patch-For-Review: Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#12586051 (10hashar) [07:03:06] 10GitLab (Project and group requests), 06Release-Engineering-Team: Create new GitLab project group:  - https://phabricator.wikimedia.org/T444993#12586055 (10Aklapper) 05Open→03Invalid a:05Nelutu.balu→03None No rationale provided. [07:03:36] 10GitLab (Project and group requests), 06Release-Engineering-Team: Create new GitLab project group:  - https://phabricator.wikimedia.org/T444993#12586058 (10Aklapper) 05Invalid→03Declined [08:28:44] (03merge) 10addshore: build(deps): bump github.com/go-sql-driver/mysql from 1.8.1 to 1.10.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/825 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [08:45:28] 10Gerrit (Gerrit 3.12), 13Patch-For-Review: Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#12586244 (10hashar) [08:57:31] GitLab needs a short maintenance in around 20 minutes [09:16:52] (03update) 10addshore: Draft: Agents, guardrails, MCP and command metadata [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/821 [09:37:43] 10Phabricator, 06Collaboration-Services, 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: data lake: phabricator events - https://phabricator.wikimedia.org/T434047#12586366 (10Antoine_Quhen) Thanks all for the input. This is where we are before merging and running full back... [11:28:21] 06Release-Engineering-Team (Radar), 10Scap, 10Citoid, 10CXServer, and 18 others: Migrate Event Platform to node26 - https://phabricator.wikimedia.org/T445574#12586921 (10aputhin) [14:00:40] 06Release-Engineering-Team, 06Data-Platform-SRE (2026-11-20 - 2026-12-11): archiva1002 has stale jobs in /var/cache/archiva that uses all the disk space - https://phabricator.wikimedia.org/T425083#12587533 (10Gehel) [15:05:12] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: Stop supporting streaming from multiple containers - https://phabricator.wikimedia.org/T445678 (10jnuche) 03NEW [15:05:37] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: Stop supporting streaming from multiple containers - https://phabricator.wikimedia.org/T445678#12588125 (10jnuche) [15:13:55] (03update) 10dondersmooi: make-release: Reject incompatible bundled Composer dependencies [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/286 (https://phabricator.wikimedia.org/T445593) [15:13:55] (03open) 10dondersmooi: make-release: Reject incompatible bundled Composer dependencies [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/286 (https://phabricator.wikimedia.org/T445593) [15:31:44] 10Gerrit (Gerrit 3.15): Remove RegisterNewEmailHtml.soy once upgraded to Gerrit 3.15 - https://phabricator.wikimedia.org/T445682 (10hashar) 03NEW [15:39:26] Yippee, build fixed! [15:39:26] Project mediawiki-core-phpmetrics build #924: 09FIXED in 5 min 25 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics/924/ [15:54:01] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: CAPI: Stop supporting streaming from multiple containers - https://phabricator.wikimedia.org/T445678#12588408 (10jnuche) 05Open→03Resolved [16:06:19] (03open) 10dancy: Chart 1.12.4: Update nginx to 1.31.9 [repos/releng/docker-hub-mirror] - 10https://gitlab.wikimedia.org/repos/releng/docker-hub-mirror/-/merge_requests/47 [16:06:40] (03update) 10dancy: Chart 1.12.4: Update nginx to 1.31.9 [repos/releng/docker-hub-mirror] - 10https://gitlab.wikimedia.org/repos/releng/docker-hub-mirror/-/merge_requests/47 [16:07:05] (03merge) 10dancy: Chart 1.12.4: Update nginx to 1.31.9 [repos/releng/docker-hub-mirror] - 10https://gitlab.wikimedia.org/repos/releng/docker-hub-mirror/-/merge_requests/47 [16:10:57] (03open) 10dancy: docker-hub-mirror: Use chart version 1.12.4 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/668 [16:11:23] (03update) 10dancy: docker-hub-mirror: Use chart version 1.12.4 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/668 [16:12:08] (03merge) 10dancy: docker-hub-mirror: Use chart version 1.12.4 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/668 [17:09:41] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/go-sql-driver/mysql from 1.10.1 to 1.10.2 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/826 [18:01:18] 10Development-Metrics, 06Release-Engineering-Team (Doing 😎): Backfill data missed while systems were down October-November 2026 - https://phabricator.wikimedia.org/T445599#12589046 (10thcipriani) My earlier question: Will this fix persist? Yes. - The mordred box is in `control-node:~zhquan/bap-deployment-tool... [18:05:10] (03PS2) 10Vaughn Walters: jjb: [catalyst-daily-Cite] Add Cite job [integration/config] - 10https://gerrit.wikimedia.org/r/1364933 (https://phabricator.wikimedia.org/T443844) [18:15:30] (03CR) 10Vaughn Walters: "mfirst run is green, https://integration.wikimedia.org/ci/view/selenium-daily/job/catalyst-daily-Cite/1/ Merging" [integration/config] - 10https://gerrit.wikimedia.org/r/1364933 (https://phabricator.wikimedia.org/T443844) (owner: 10Vaughn Walters) [18:15:38] (03CR) 10Vaughn Walters: [C:03+2] jjb: [catalyst-daily-Cite] Add Cite job [integration/config] - 10https://gerrit.wikimedia.org/r/1364933 (https://phabricator.wikimedia.org/T443844) (owner: 10Vaughn Walters) [18:16:43] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-Cite] Add Cite job [integration/config] - 10https://gerrit.wikimedia.org/r/1364933 (https://phabricator.wikimedia.org/T443844) (owner: 10Vaughn Walters) [18:30:21] 10Development-Metrics, 06Release-Engineering-Team (Doing 😎): Backfill data missed while systems were down October-November 2026 - https://phabricator.wikimedia.org/T445599#12589133 (10thcipriani) Done. mordred logs look good. Will let it run and check after a while. Still TODO: update gitlab repos...possibly... [19:45:03] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Doing 😎): Build and deploy Zuul 14.3.0 - https://phabricator.wikimedia.org/T445248#12589426 (10Dzahn) Deployed the 14.3.0-1 images (1371904) on zuul1004 and zuul1005. zuul-executor on zuul1005 then failed with "No module named... [19:53:04] !log Beta Cluster: Attempt to repair RESTBase config to declutter access logs, in order to find out what's gonna break if we decommission/replace it - T436477 [19:53:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:53:07] T436477: Replace or decommission RESTBase hosts: deployment-restbase(05|-bullseye) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436477 [20:18:43] 10Gerrit (Gerrit 3.12), 06Release-Engineering-Team, 06Collaboration-Services, 07Upstream: Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10 - https://phabricator.wikimedia.org/T407844#12589547 (10Dzahn) :) nice! can we close this as resolved? @hashar [20:19:22] TIL scap3 supports *local* configuration of services by erb templating files such as /etc/restbase/config-vars.yaml [20:55:42] Southparkfan: huh. I may have known that at some point. [20:56:44] The good ol' RESTBase :) [21:00:05] it is truly a a technical gift that just keeps on giving. [21:01:53] Definitely [21:03:11] I'm at a turning point in my "career" as a technical volunteer, getting closer to understanding the bits in https://wikitech.wikimedia.org/wiki/File:MediaWiki_infrastructure_2022.png day by day. [21:04:02] so what you're saying is that K.rinkle needs to make an updated version of that diagram? :P [21:04:24] Yes :) [21:06:38] By the way, I genuinely appreciate Timo's style of drawing diagrams. At $job, I tried using drawio to explain traffic flows in a hybrid cloud environment (something something multiple CSPs and a legacy on-premises environment). I have managed to draw *something*, but compared to these WMF/MW infrastructure diagrams, it's a mess. [21:07:47] 06Release-Engineering-Team, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: archiva1002 has stale jobs in /var/cache/archiva that uses all the disk space - https://phabricator.wikimedia.org/T425083#12589789 (10Gehel) [21:24:20] (03merge) 10addshore: build(deps): bump github.com/go-sql-driver/mysql from 1.10.1 to 1.10.2 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/826 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [21:44:24] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace or decommission RESTBase hosts: deployment-restbase(05|-bullseye) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436477#12590041 (10Southparkfan) After fixing RESTBase - only recommendation-api is broken now, but I d... [22:09:52] (03update) 10addshore: Draft: Agents, guardrails, MCP and command metadata [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/821 [22:16:37] (03merge) 10addshore: build(deps): bump github.com/itchyny/gojq from 0.12.19 to 0.12.20 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/795 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [22:17:42] (03open) 10addshore: Grafana service, and dev dashboard [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/827 [22:18:05] (03update) 10addshore: Agents, guardrails, MCP and command metadata [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/821 [22:28:15] (03merge) 10addshore: Agents, guardrails, MCP and command metadata [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/821 [22:38:10] (03merge) 10addshore: Grafana service, and dev dashboard [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/827 [22:41:33] (03open) 10addshore: build(deps): update PHP images for November security releases [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/828 [22:53:20] (03merge) 10addshore: build(deps): update PHP images for November security releases [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/828