[00:01:13] (03update) 10thcipriani: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 (https://phabricator.wikimedia.org/T347207) (owner: 10sandeeps) [00:06:57] (03merge) 10thcipriani: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 (https://phabricator.wikimedia.org/T347207) (owner: 10sandeeps) [00:48:55] (03CR) 10Pppery: "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/1217260 (https://phabricator.wikimedia.org/T411558) (owner: 10Kimberly Sarabia) [02:11:28] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:11:35] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T412774 (10wmcs-alerts) 03NEW [02:16:28] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:52:23] (03PS1) 10Clare Ming: Add references to Test Kitchen [integration/config] - 10https://gerrit.wikimedia.org/r/1218396 (https://phabricator.wikimedia.org/T407906) [03:01:17] (03PS2) 10Clare Ming: Add references to Test Kitchen [integration/config] - 10https://gerrit.wikimedia.org/r/1218396 (https://phabricator.wikimedia.org/T407906) [06:23:01] 10Gerrit, 06collaboration-services: Gerrit failover process - https://phabricator.wikimedia.org/T412779#11462804 (10ABran-WMF) [06:28:11] 10Gerrit, 06collaboration-services: Use encrypted rsync for Gerrit - https://phabricator.wikimedia.org/T412780 (10ABran-WMF) 03NEW [07:20:58] 10Gerrit, 06collaboration-services: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#11462853 (10ABran-WMF) 05Openβ†’03Resolved marking this as resolved, feel free to reopen if needed [07:47:08] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit switchover process - https://phabricator.wikimedia.org/T387833#11462886 (10A_smart_kitten) [09:50:25] 10GitLab (Auth & Access), 10Phabricator, 06Release-Engineering-Team (Radar), 06cloud-services-team, 10Toolforge: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#11463141 (10Bugreporter) > a member of Trusted-Contributors was being unintentional... [09:54:58] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.46.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T408277#11463166 (10hashar) >>! In T408277#11462645, @gerritbot wrote: > Change #1218393 had a related patch set uplo... [10:48:53] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T408277#11463352 (10Dreamy_Jazz) The train still has not moved to 1.46.0-wmf.7 and is blocking deployments? Is this a known issue? [11:04:46] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T408277#11463386 (10hashar) Yes it failed overnight and I have restarted the images building this morning. It was stuck. The image was cre... [11:09:00] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T408277#11463392 (10Dreamy_Jazz) I wonder if it makes sense for scap to be locked during this? `scap` proceeded to merge the patches but th... [11:53:42] 06Project-Admins, 10Tool-curator: Create a workboard for Tool-curator - https://phabricator.wikimedia.org/T412686#11463494 (10DaxServer) 05Openβ†’03Resolved a:03DaxServer Thank you! [11:58:41] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for marius851000 - https://phabricator.wikimedia.org/T412629#11463508 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Done. [12:06:31] (03PS1) 10Krinkle: zuul: Add php85 experimental to mediawiki/libs/less.php [integration/config] - 10https://gerrit.wikimedia.org/r/1218742 (https://phabricator.wikimedia.org/T411213) [12:12:52] (03CR) 10Krinkle: [C:03+2] zuul: Add php85 experimental to mediawiki/libs/less.php [integration/config] - 10https://gerrit.wikimedia.org/r/1218742 (https://phabricator.wikimedia.org/T411213) (owner: 10Krinkle) [12:15:27] (03Merged) 10jenkins-bot: zuul: Add php85 experimental to mediawiki/libs/less.php [integration/config] - 10https://gerrit.wikimedia.org/r/1218742 (https://phabricator.wikimedia.org/T411213) (owner: 10Krinkle) [12:16:36] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1218742 [12:16:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:17:10] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project for Phabricator translations - https://phabricator.wikimedia.org/T412647#11463615 (10Aklapper) 05Openβ†’03Resolved p:05Triageβ†’03Medium a:03Aklapper Makes sense, thanks for all your work! Done in https://phabricator.wikimedia.org... [12:19:16] 10Phabricator, 10Phabricator Translations Extension, 06Release-Engineering-Team (Yakisfaction): Translations: Remove and gitignore .phutil_module_cache - https://phabricator.wikimedia.org/T365716#11463624 (10Aklapper) [12:19:32] 10Phabricator, 10Phabricator Antivandalism Extension, 10Phabricator Translations Extension, 06translatewiki.net, 07Unplanned-Sprint-Work: Add Phabricator antivandalism extension to Phabricator translations - https://phabricator.wikimedia.org/T365858#11463625 (10Aklapper) [12:19:48] 10Phabricator, 10Phabricator Translations Extension, 06translatewiki.net: Restore translations lost when libphutil was split - https://phabricator.wikimedia.org/T366377#11463626 (10Aklapper) [12:38:58] 06Release-Engineering-Team, 06serviceops, 06SRE Observability: Proof of Concept: Train Health Dashboard - https://phabricator.wikimedia.org/T412801 (10jijiki) 03NEW [12:41:28] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project for Phabricator translations - https://phabricator.wikimedia.org/T412647#11463771 (10A_smart_kitten) Out of interest, is there any interest in doing a silent batch-edit to add the #phabricator_translations_extension tag to closed tasks... [12:54:05] 10Phabricator (2024-05-05), 10Phabricator Translations Extension, 10Wikimedia-Phabricator-Extensions, 07Epic, 07PHP 8.0 support: Make WMF custom Phab extensions and code work with PHP8 - https://phabricator.wikimedia.org/T342244#11463853 (10A_smart_kitten) [12:59:40] 10Phabricator Translations Extension, 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project for Phabricator translations - https://phabricator.wikimedia.org/T412647#11463886 (10A_smart_kitten) Ah, there's [[https://phabricator.wikimedia.org/T412646#11463659|already been a silent batch-ed... [14:01:05] a valuable lesson learned this morning: adding another factor to phab's 2fa isn't "require this key OR this key", it is "require this key AND this key". [14:01:09] 10Phabricator (Upstream), 10Phabricator Antivandalism Extension, 07Upstream: If the Antivandalism Extension is disabling an account in response to a given action, also prevent that action from taking place - https://phabricator.wikimedia.org/T412583#11464298 (10Aklapper) >>! In T412583#11462286, @A_smart_kit... [14:03:11] brennen: hmm, wasn't there a warning? https://we.phorge.it/rPa7f94af9cf5b6e557378f23ca2a987ba08bfa9ae should have added that :-/ [14:08:24] (03PS2) 10Jsn.sherman: Zuul: [mediawiki/extensions/PersonalDashboard] Add prod template [integration/config] - 10https://gerrit.wikimedia.org/r/1217775 (https://phabricator.wikimedia.org/T412528) [14:09:51] yeah, I tried to be smart and get around the cooldown on a single factor one time and ended up in a situation where I had to enter like five different 2fa codes in the time frame to log in [14:11:03] haha [14:11:55] andre: it's possible there was a warning and i just obliviously skimmed right past it; i was hurriedly doing various new-hardware-right-before-travel things. [14:12:03] and as we all know, "people don't read things" includes me, i guess. [14:12:50] eh :) [14:12:55] brennen: also, do we wanna deploy today? [14:13:30] i should be able to if there's anything you want to get out before the holiday. [14:13:59] i'll be afk from friday on, or maybe earlier. [14:16:12] Would love to finally get https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/90/diffs out but should probably do that early January as people WILL have opinions, and sounds like we'll both be mostly out soon [14:16:23] but yeah there are a few smaller patches to maybe get out today [14:16:38] 10Phabricator (Upstream), 10Phabricator Antivandalism Extension, 07Upstream: If the Antivandalism Extension is disabling an account in response to a given action, also prevent that action from taking place - https://phabricator.wikimedia.org/T412583#11464377 (10A_smart_kitten) Basically, I 'stole'/repurposed... [14:16:49] k, i can cut a branch. [14:16:52] (03update) 10aklapper: Related GitLab patches: Ignore partial task ID matches [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/55 (https://phabricator.wikimedia.org/T395728) [14:17:15] brennen: let me first also merge https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/55 if you're not completely against that :) [14:17:20] then we should be ready to go [14:17:56] sounds good [14:19:08] (03merge) 10aklapper: Related GitLab patches: Ignore partial task ID matches [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/55 (https://phabricator.wikimedia.org/T395728) [14:19:36] 10GitLab (Upstream pit of despair πŸ•³οΈ), 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎), 10Wikimedia-Phabricator-Extensions: 'Related GitLab patches' box includes patches for unrelated tasks (that begin with the current task's ID) - https://phabricator.wikimedia.org/T395728#11464387 (1... [14:19:41] alright. GO CUT THEM ALL!!!1 [14:23:10] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phab/Phorge 2025-12-16 - https://phabricator.wikimedia.org/T412825 (10brennen) 03NEW [14:28:04] (03open) 10brennen: update submodules for 2025-12-16 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/90 (https://phabricator.wikimedia.org/T412825) [14:28:39] ^ andre [14:30:05] brennen: yupp, that looks good and complete to me. Thanks [14:42:51] (03CR) 10LWatson: [C:03+1] "Done" [integration/config] - 10https://gerrit.wikimedia.org/r/1217260 (https://phabricator.wikimedia.org/T411558) (owner: 10Kimberly Sarabia) [14:53:03] 10Gerrit, 06collaboration-services: Use encrypted rsync for Gerrit - https://phabricator.wikimedia.org/T412780#11464645 (10Dzahn) Where `rsync::quickdatacopy` is being used (like for `lfs_replica_sync` in modules/gerrit/manifest/init.pp) it should work by adding the `server_uses_stunnel` parameter. I noticed... [15:03:05] 06Release-Engineering-Team: versions.toolforge.org mixes up testwikis and group 0 - https://phabricator.wikimedia.org/T412834 (10hashar) 03NEW [15:05:19] 10Gerrit, 06collaboration-services: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#11464747 (10Dzahn) My thought here is just that the O'Reilly SRE book reminded me of "**data recovery plans should not rely on replication**". [15:35:05] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T408277#11464912 (10hashar) @dancy 1.46.0-wmf.7 has been rolled to testwikis. The pretrain Jenkins job failed due to some npm connection i... [15:37:35] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T408277#11464914 (10dancy) >>! In T408277#11464912, @hashar wrote: > @dancy 1.46.0-wmf.7 has been rolled to testwikis. > > The pretrain Je... [15:42:53] (03merge) 10brennen: update submodules for 2025-12-16 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/90 (https://phabricator.wikimedia.org/T412825) [15:43:54] (03open) 10pppery: Remove invalid username rewording [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/91 [15:44:13] (03update) 10pppery: Remove invalid username rewording [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/91 [15:47:32] !log Restarting CI Jenkins [15:47:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:02:03] 10Phabricator (2025-12-16), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phab/Phorge 2025-12-16 - https://phabricator.wikimedia.org/T412825#11464993 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=767d0770-34be-4080-b717-6fcd10af3517) set by jelto@cum... [16:02:50] 10Phabricator (2025-12-16), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phab/Phorge 2025-12-16 - https://phabricator.wikimedia.org/T412825#11464994 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=6fbc002b-f58a-4a1e-8c1b-328bb8dc684d) set by jelto@cum... [16:07:27] 10Phabricator (2025-12-16), 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phab/Phorge 2025-12-16 - https://phabricator.wikimedia.org/T412825#11465015 (10Aklapper) 05Openβ†’03Resolved a:03brennen This deployment included: * Translations (thanks to Pppery for all patches!): **... [16:09:33] 10GitLab (Upstream pit of despair πŸ•³οΈ), 10Phabricator (2025-12-16), 06Release-Engineering-Team (Doing 😎), 10Wikimedia-Phabricator-Extensions: 'Related GitLab patches' box includes patches for unrelated tasks (that begin with the current task's ID) - https://phabricator.wikimedia.org/T395728#11465023 (10A... [16:32:17] 10Phabricator (2025-12-16), 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phab/Phorge 2025-12-16 - https://phabricator.wikimedia.org/T412825#11465107 (10Pppery) Note that none of the translation stuff has a direct effect on Phabricator; it's all internal refactoring of scripts or... [16:35:50] 06Release-Engineering-Team (Priority Backlog πŸ“₯): Automatically handle manual release notes for MediaWiki - https://phabricator.wikimedia.org/T347207#11465131 (10thcipriani) Now, every Tuesday at 2am utc (so Monday night) we automatically create: - `MediaWiki 1.XX/wmf.XX/Changelog` for the branch it just cut - `M... [16:40:03] maintenance-disconnect-full-disks build 763803 integration-agent-docker-1041 (/: 37%, /srv: 98%, /var/lib/docker: 27%): OFFLINE due to disk space [16:44:38] (03open) 10dancy: Improve reliability of the CAS server [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/173 [16:44:42] (03update) 10dancy: Improve reliability of the CAS server [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/173 [16:45:03] maintenance-disconnect-full-disks build 763804 integration-agent-docker-1041 (/: 37%, /srv: 26%, /var/lib/docker: 26%): RECOVERY disk space OK [16:45:47] (03merge) 10dancy: Improve reliability of the CAS server [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/173 [16:52:05] 10Phabricator (2025-12-16), 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phab/Phorge 2025-12-16 - https://phabricator.wikimedia.org/T412825#11465239 (10Aklapper) Right, sorry. I need to get better in simply copying your commit message titles. :) [17:30:03] maintenance-disconnect-full-disks build 763813 integration-agent-docker-1051 (/: 36%, /srv: 100%, /var/lib/docker: 29%): OFFLINE due to disk space [17:35:04] maintenance-disconnect-full-disks build 763814 integration-agent-docker-1051 (/: 36%, /srv: 71%, /var/lib/docker: 29%): RECOVERY disk space OK [17:55:21] 10Continuous-Integration-Config, 05MW-1.39-release, 07PHP 8.1 support: Make Phan on PHP 8.1 voting on REL1_39 - https://phabricator.wikimedia.org/T326374#11465632 (10Reedy) 05Openβ†’03Declined [18:17:52] (03CR) 10Eric Gardner: [C:03+1] Decomissioning Article Summaries [integration/config] - 10https://gerrit.wikimedia.org/r/1217260 (https://phabricator.wikimedia.org/T411558) (owner: 10Kimberly Sarabia) [21:15:52] (03open) 10dancy: Fix incremental image build with newer docker [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/230 [21:15:55] (03update) 10dancy: Fix incremental image build with newer docker [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/230 [21:20:35] (03CR) 10Thcipriani: [C:03+2] Zuul: [mediawiki/extensions/PersonalDashboard] Add prod template [integration/config] - 10https://gerrit.wikimedia.org/r/1217775 (https://phabricator.wikimedia.org/T412528) (owner: 10Jsn.sherman) [21:22:04] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PersonalDashboard] Add prod template [integration/config] - 10https://gerrit.wikimedia.org/r/1217775 (https://phabricator.wikimedia.org/T412528) (owner: 10Jsn.sherman) [21:24:37] (03open) 10dancy: Define BUILD_DIR and fix run_minikube [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/174 [21:24:40] (03update) 10dancy: Define BUILD_DIR and fix run_minikube [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/174 [21:25:57] (03update) 10dancy: Define BUILD_DIR and fix run_minikube [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/174 [21:26:01] (03update) 10dancy: Define BUILD_DIR and fix run_minikube [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/174 [21:26:29] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1217775 [21:26:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:27:38] (03merge) 10dancy: Define BUILD_DIR and fix run_minikube [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/174 [21:50:50] 10Scap (SpiderPig πŸ•ΈοΈ): Act as if testwikis is not part of group0 - https://phabricator.wikimedia.org/T412868 (10dancy) 03NEW [21:54:44] 10Scap (SpiderPig πŸ•ΈοΈ): Act as if testwikis are not part of group0 - https://phabricator.wikimedia.org/T412868#11466553 (10dancy) [21:56:11] (03open) 10dancy: train: Act as if testwikis are not part of group0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1056 (https://phabricator.wikimedia.org/T412868) [21:56:13] (03update) 10dancy: train: Act as if testwikis are not part of group0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1056 (https://phabricator.wikimedia.org/T412868) [21:58:46] (03update) 10dancy: train: Act as if testwikis are not part of group0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1056 (https://phabricator.wikimedia.org/T412868) [22:01:18] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.26.3 released - https://phabricator.wikimedia.org/T412869 (10dancy) 03NEW [22:01:21] (03PS1) 10Subramanya Sastry: Bump puppeteer to latest version [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1218841 [22:01:21] (03PS1) 10Subramanya Sastry: Update package-lock.json based on npm install and audit fix [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1218842 [22:01:21] (03PS1) 10Subramanya Sastry: Add scripts to gather metrics (size, time, memory) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1218843 [22:01:46] (03CR) 10CI reject: [V:04-1] Bump puppeteer to latest version [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1218841 (owner: 10Subramanya Sastry) [22:01:55] (03CR) 10CI reject: [V:04-1] Add scripts to gather metrics (size, time, memory) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1218843 (owner: 10Subramanya Sastry) [22:11:56] Could somebody mark https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GlobalWatchlist/+/1218844 as Work-in-Progress, please? The patch that allows to do this to trusted users was not merged yet. Thank you in advance. [22:12:38] IKhitron: marked as WIP [22:13:02] Thanks a lot.