[00:07:08] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12249667 (10Lars) @Andrew 1077 has run between 21-24 minutes sin... [01:42:51] 10Beta-Cluster-Infrastructure, 07Wikimedia-extension-review-queue: Deploy RecentPageStats extension to Beta Cluster - https://phabricator.wikimedia.org/T435869#12249738 (10Bugreporter) 05Open→03Stalled Step 3 of https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment#Preparing_for_deployment mus... [03:06:32] 10Beta-Cluster-Infrastructure, 10Application Security Reviews, 06secscrum, 07Security, 07Wikimedia-extension-review-queue: Application Security Review for RecentPageStats extension - https://phabricator.wikimedia.org/T435890 (10Jnanaranjan_sahu) 03NEW [03:09:16] 10Beta-Cluster-Infrastructure, 07Wikimedia-extension-review-queue: Deploy RecentPageStats extension to Beta Cluster - https://phabricator.wikimedia.org/T435869#12249803 (10Jnanaranjan_sahu) [03:12:14] 10Beta-Cluster-Infrastructure, 07Wikimedia-extension-review-queue: Deploy RecentPageStats extension to Beta Cluster - https://phabricator.wikimedia.org/T435869#12249806 (10Jnanaranjan_sahu) 05Stalled→03Open Thanks @Bugreporter! I have updated the task description with the pre-deployment checklist and creat... [03:56:48] 10Phabricator, 06Collaboration-Services: phstats user lacks database permissions on phab1005 - https://phabricator.wikimedia.org/T435442#12249892 (10Marostegui) If you are going to do just RO operations, then do not user m3-master but m3-slave, so no need to change that. I've added the grant. ` root@phab1005:~... [03:57:43] 10Phabricator, 06Collaboration-Services, 06DBA: phstats user lacks database permissions on phab1005 - https://phabricator.wikimedia.org/T435442#12249893 (10Marostegui) 05Open→03Resolved a:05Arnoldokoth→03Marostegui The scope of this ticket is fixed by adding the grants. [04:25:43] 10GitLab, 06Release-Engineering-Team (Radar), 10a Wikimedia CLI: Possibly rename mwcli and mwcli gitlab repo - https://phabricator.wikimedia.org/T384781#12249950 (10Samwilson) It sounds like this is now called "Wikimedia CLI". Should https://www.mediawiki.org/wiki/Cli be updated to match? [05:23:51] (03open) 10phedenskog: Add date filters and readability fixes to the core and selenium dashboards [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/39 [05:23:59] (03merge) 10phedenskog: Add date filters and readability fixes to the core and selenium dashboards [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/39 [05:31:43] (03open) 10phedenskog: Add a daily castor save time chart for the last 30 days [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/40 [05:31:57] (03merge) 10phedenskog: Add a daily castor save time chart for the last 30 days [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/40 [06:25:16] (03CR) 10Hashar: [C:03+2] Docker: [quibble-bookworm] Switch quibble to venv for easier installs [integration/config] - 10https://gerrit.wikimedia.org/r/1327120 (owner: 10Jforrester) [06:27:37] (03Merged) 10jenkins-bot: Docker: [quibble-bookworm] Switch quibble to venv for easier installs [integration/config] - 10https://gerrit.wikimedia.org/r/1327120 (owner: 10Jforrester) [06:35:15] (03CR) 10Hashar: [C:03+2] docker: Use venv for Quibble installation [integration/quibble] - 10https://gerrit.wikimedia.org/r/1327130 (owner: 10Jforrester) [06:42:14] James_F: thx for the patches to have Quibble installed within a virtual env [06:52:07] (03Merged) 10jenkins-bot: docker: Use venv for Quibble installation [integration/quibble] - 10https://gerrit.wikimedia.org/r/1327130 (owner: 10Jforrester) [07:35:56] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: irc.beta.wmflabs.org is unresponsive - https://phabricator.wikimedia.org/T396088#12250167 (10taavi) 05Open→03Resolved a:03taavi [07:52:40] (03PS1) 10Hashar: jjb: update Quibble jobs (venv install) [integration/config] - 10https://gerrit.wikimedia.org/r/1329169 [07:53:10] (03CR) 10Hashar: [C:03+2] "Follows-up I2d8125081b21cb1a0028d42d2c7ec73c30efd9bd" [integration/config] - 10https://gerrit.wikimedia.org/r/1329169 (owner: 10Hashar) [07:53:32] (03CR) 10Hashar: [C:03+2] "I have updated the Jenkins jobs via If0c7765bd624b680d077a471f8940188b62ea2f1" [integration/config] - 10https://gerrit.wikimedia.org/r/1327120 (owner: 10Jforrester) [07:54:57] (03Merged) 10jenkins-bot: jjb: update Quibble jobs (venv install) [integration/config] - 10https://gerrit.wikimedia.org/r/1329169 (owner: 10Hashar) [08:00:07] 10Phabricator, 10Striker, 06tools-platform-team: Set corresponding project tag when creating a Diffusion repository via Striker - https://phabricator.wikimedia.org/T435906 (10Aklapper) 03NEW [08:00:39] 10Phabricator, 10Striker, 06tools-platform-team: Set corresponding project tag when creating a Diffusion repository via Striker - https://phabricator.wikimedia.org/T435906#12250242 (10Aklapper) p:05Triage→03Low [08:03:59] 10Phabricator, 10Striker, 06tools-platform-team: Set corresponding project tag when creating a Diffusion repository via Striker - https://phabricator.wikimedia.org/T435906#12250249 (10Aklapper) [08:22:33] 10Beta-Cluster-Infrastructure, 07Wikimedia-extension-review-queue: Deploy RecentPageStats extension to Beta Cluster - https://phabricator.wikimedia.org/T435869#12250303 (10Aklapper) Please thoroughly check https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment. I cannot find any Phabricator project... [08:46:52] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12250373 (10hashar) Sorry I got distracted last week and I did n... [09:10:19] 06Project-Admins: Create project tag for RecentPageStats - https://phabricator.wikimedia.org/T435910 (10Jnanaranjan_sahu) 03NEW [09:16:49] 10Beta-Cluster-Infrastructure, 07Wikimedia-extension-review-queue: Deploy RecentPageStats extension to Beta Cluster - https://phabricator.wikimedia.org/T435869#12250481 (10Jnanaranjan_sahu) Thanks @Aklapper! I have requested the Phabricator project tag for RecentPageStats in {T435910}. Once created, I will add... [09:33:18] 10Diffusion, 10Phabricator (Upstream), 07Upstream: Attempting to view a non-existent file through Diffusion causes an unhandled CommandException - https://phabricator.wikimedia.org/T435627#12250566 (10Aklapper) Upstreamed as https://we.phorge.it/T16787 [09:37:46] 10Diffusion, 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Attempting to view a non-existent file through Diffusion causes an unhandled CommandException - https://phabricator.wikimedia.org/T435627#12250581 (10Aklapper) a:03Aklapper [10:04:29] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-RecentPageStats, 07Wikimedia-extension-review-queue: Deploy RecentPageStats extension to Beta Cluster - https://phabricator.wikimedia.org/T435869#12250672 (10Aklapper) [10:04:42] 10Beta-Cluster-Infrastructure, 10Application Security Reviews, 10MediaWiki-extensions-RecentPageStats, 06secscrum, and 2 others: Application Security Review for RecentPageStats extension - https://phabricator.wikimedia.org/T435890#12250673 (10Aklapper) [10:05:12] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for RecentPageStats - https://phabricator.wikimedia.org/T435910#12250675 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project tag #mediawiki-extensions-recentpagestats has been created: https://phabricator.wiki... [10:17:56] 10Phabricator, 06Collaboration-Services, 06DBA: phstats user lacks database permissions on phab1005 - https://phabricator.wikimedia.org/T435442#12250721 (10Arnoldokoth) Thank you @Marostegui [10:35:13] 10Phabricator, 06Collaboration-Services, 06DBA: phstats user lacks database permissions on phab1005 - https://phabricator.wikimedia.org/T435442#12250788 (10Aklapper) > I've added the grant. For `phstats` (no idea what's that good for) or for `phuser` (which worked for me until last Thursday)? Sorry, st... [10:56:12] 10Phabricator, 06Collaboration-Services, 06DBA: phstats user lacks database permissions on phab1005 - https://phabricator.wikimedia.org/T435442#12250950 (10Marostegui) I will add it in a bit - in meetings now. [11:05:30] 10Phabricator, 06Collaboration-Services, 06DBA: phstats user lacks database permissions on phab1005 - https://phabricator.wikimedia.org/T435442#12251019 (10Marostegui) should be fixed now, can you try again? [11:07:20] 10Phabricator, 06Collaboration-Services, 06DBA: phstats user lacks database permissions on phab1005 - https://phabricator.wikimedia.org/T435442#12251040 (10Aklapper) Thank you a lot! Yes, `mysql -h m3-slave.eqiad.wmnet -P 3323 -u phuser -p --skip-ssl` works now. [11:31:05] 10GitLab: Avatars missing in several places - https://phabricator.wikimedia.org/T435934 (10Sebastian_Berlin-WMSE) 03NEW [13:36:11] 10GitLab (Administration, Settings & Policy): Archive GitLab repo commtech/wishlist-intake - https://phabricator.wikimedia.org/T432059#12251651 (10Aklapper) Does one have to be an "owner" to do this? [13:40:42] hey folks, I just moved /v2/repos/releng to its new s3 bucket [13:41:06] not all the images stored on S3 were copied over, so if you spot a CI job that ends up in a docker pull failure or similar please ping me [13:41:38] Ack [14:04:44] ack [14:06:53] (03open) 10phedenskog: Add the 10 minute goal line to the gated selenium run time chart [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/41 [14:14:50] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12251947 (10hashar) Thank you for the subtask! On the parent task I emitted the hypotheses that maybe the... [14:16:48] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Wilfredor (GitLab username wilfredor2) - https://phabricator.wikimedia.org/T435957 (10Wilfredor) 03NEW [14:18:50] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06Collaboration-Services: Deploy Phab/Phorge 2026-08-25 - https://phabricator.wikimedia.org/T435958 (10brennen) 03NEW [14:20:24] (03open) 10brennen: update submodules for 2026-08-25 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/119 (https://phabricator.wikimedia.org/T435958) [14:20:27] andre: ^ [14:25:47] (03merge) 10phedenskog: Add the 10 minute goal line to the gated selenium run time chart [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/41 [14:31:14] (03open) 10phedenskog: Show percentile spread in core dashboard chart hovers [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/42 [14:31:48] (03merge) 10phedenskog: Show percentile spread in core dashboard chart hovers [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/42 [14:42:28] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T430836#12252194 (10Zabe) ^ Moving bswiktionary since I moved it from group1 to group0 as part of T435954. [14:47:16] (03approved) 10jnuche: catalyst-base: update to PHP 8.5, Node 24 [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/121 (https://phabricator.wikimedia.org/T435368) (owner: 10brennen) [14:53:30] brennen: lgtm [14:54:41] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Deploy Phab/Phorge 2026-08-25 - https://phabricator.wikimedia.org/T435958#12252292 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c7a76720-e507-4e44-ac47-0d3c3ff5479a) set by jel... [14:55:04] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Deploy Phab/Phorge 2026-08-25 - https://phabricator.wikimedia.org/T435958#12252294 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0bb82a05-b43c-46f8-83bc-1ade517c5b75) set by jel... [14:55:37] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for Wilfredor (GitLab username wilfredor2) - https://phabricator.wikimedia.org/T435957#12252298 (10Aklapper) 05Open→03Resolved a:03Aklapper I added you to #Trusted-Contributors instead given your c... [14:55:39] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Deploy Phab/Phorge 2026-08-25 - https://phabricator.wikimedia.org/T435958#12252297 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=fd427404-2e43-4128-b05a-a687319d60fa) set by jel... [14:57:58] 06Release-Engineering-Team: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12252310 (10dancy) ` dancy@deploy1003:~$ scap analyze-logstash --scope pretrain canaries 14:55:32 Analyzing logstash history for scope: pretrain, stage: canari... [14:58:34] (03merge) 10brennen: update submodules for 2026-08-25 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/119 (https://phabricator.wikimedia.org/T435958) [15:04:46] andre: well train looks fine as far as I can tell [15:04:52] I'll review the logs a bit later tonight [15:05:12] yay [15:05:12] stay tuned for tomorrow morning! 🚂 [15:05:53] (03merge) 10brennen: catalyst-base: update to PHP 8.5, Node 24 [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/121 (https://phabricator.wikimedia.org/T435368) [15:06:52] 10Phabricator (2026-08-25), 06Release-Engineering-Team (Doing 😎), 06Collaboration-Services: Deploy Phab/Phorge 2026-08-25 - https://phabricator.wikimedia.org/T435958#12252368 (10Aklapper) 05Open→03Resolved Changes included in today's deployment (or merged over the last days): * Fix {T435466} via htt... [15:06:58] !log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/121 (T435368) [15:07:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:07:00] T435368: Migrate Patchdemo & Catalyst to PHP 8.5, Node 24 - https://phabricator.wikimedia.org/T435368 [15:10:32] 10Phabricator (2026-08-25), 06Release-Engineering-Team (Doing 😎): PHP 8.1 "Implicit conversion from float 36.5 to int loses precision" in PhabricatorSearchResultView - https://phabricator.wikimedia.org/T435466#12252385 (10Aklapper) 05Open→03Resolved This should reduce our pretty high Logstash noise lev... [15:10:41] (03open) 10dancy: logstash_checker.py: Do not update the release repository [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1278 (https://phabricator.wikimedia.org/T435419) [15:11:52] (03update) 10dancy: logstash_checker.py: Do not update the release repository [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1278 (https://phabricator.wikimedia.org/T435419) [15:12:33] 06Release-Engineering-Team, 10dev-images, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Migrate Patchdemo & Catalyst to PHP 8.5, Node 24 - https://phabricator.wikimedia.org/T435368#12252422 (10brennen) ` == Step 2: publishing == Successfully published image docker-registry.disco... [15:13:06] 06Release-Engineering-Team, 13Patch-For-Review: Bad interaction between sssd auto-shutdown and wmf-beta-update-all on deployment-deploy04 - https://phabricator.wikimedia.org/T434809#12252424 (10dancy) 05Open→03Resolved a:03dancy [15:14:36] (03merge) 10dancy: logstash_checker.py: Do not update the release repository [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1278 (https://phabricator.wikimedia.org/T435419) [15:15:43] (03open) 10dancy: Release 4.285.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1279 [15:20:52] (03merge) 10dancy: Release 4.285.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1279 [15:23:50] (03open) 10dancy: deploy-promote: Verify the promotion with deployment-info.php [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1280 (https://phabricator.wikimedia.org/T434726) [15:33:26] 10Phabricator, 06Collaboration-Services, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: data lake: phabricator events - https://phabricator.wikimedia.org/T434047#12252565 (10LSobanski) [16:11:54] 06Release-Engineering-Team, 10MediaWiki-extensions-StopForumSpam, 06Product Safety and Integrity, 10Catalyst (PatchDemo): StopForumSpam explodes on missing configuration in Patchdemo - https://phabricator.wikimedia.org/T435978 (10brennen) 03NEW [16:15:29] (03open) 10dancy: deploy-promote: Check en.wiktionary.org for group1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1281 (https://phabricator.wikimedia.org/T421796 https://phabricator.wikimedia.org/T435915) [16:15:55] (03update) 10dancy: deploy-promote: Check en.wiktionary.org for group1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1281 (https://phabricator.wikimedia.org/T421796 https://phabricator.wikimedia.org/T435915) [16:16:45] (03update) 10dancy: deploy-promote: Verify the promotion with deployment-info.php [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1280 (https://phabricator.wikimedia.org/T434726) [16:22:29] 06Release-Engineering-Team, 10MediaWiki-extensions-StopForumSpam, 06Product Safety and Integrity, 10Catalyst (PatchDemo), 13Patch-For-Review: StopForumSpam explodes on missing configuration in Patchdemo - https://phabricator.wikimedia.org/T435978#12252835 (10brennen) →14Duplicate dup:03T420302 [16:26:19] 06Release-Engineering-Team, 10Catalyst (PatchDemo): "wgSFSIPListLocation does not appear to be a valid URL" when trying to create a patchdemo with all extensions - https://phabricator.wikimedia.org/T420302#12252875 (10brennen) [16:28:54] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12252894 (10BLiviero-WMF) consulting with Claude, there should be no reason for intel_pstate to not recogn... [16:37:37] (03merge) 10dancy: deploy-promote: Check en.wiktionary.org for group1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1281 (https://phabricator.wikimedia.org/T421796 https://phabricator.wikimedia.org/T435915) [16:39:14] (03open) 10dancy: Release 4.285.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1282 [16:41:50] (03merge) 10dancy: Release 4.285.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1282 [16:47:19] (03update) 10dancy: deploy-promote: Verify the promotion with deployment-info.php [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1280 (https://phabricator.wikimedia.org/T434726) [16:50:53] (03update) 10dancy: deploy-promote: Verify the promotion with deployment-info.php [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1280 (https://phabricator.wikimedia.org/T434726) [17:01:29] (03update) 10dancy: deploy-service: Run helmfile commands, add --confirm-diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1273 (https://phabricator.wikimedia.org/T412941) [17:01:38] (03update) 10dancy: deploy-service: Run helmfile commands, add --confirm-diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1273 (https://phabricator.wikimedia.org/T412941) [17:02:17] (03update) 10dancy: deploy-service: Run helmfile commands, add --confirm-diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1273 (https://phabricator.wikimedia.org/T412941) [17:03:03] FIRING: [2x] PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-acme-chief07 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:04:33] I can't find a ticket but all PG jobs are failing since this morning so nothing can be merged on core https://integration.wikimedia.org/ci/job/quibble-for-mediawiki-core-vendor-postgres-php83/ [17:04:51] 10Beta-Cluster-Infrastructure: irc.beta.wmflabs.org is unresponsive - https://phabricator.wikimedia.org/T396088#12253080 (10bd808) [[https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/4961e8f45f0fa49b64da8a8d71008c179444d93a|4961e8f4: Delete data for deployment-prep deployment-irc]] [17:06:07] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319#12253096 (10dancy) 05Resolved→03Open `run-puppet-agent` on `integration-cumin-01.integration.eqiad1.wikimedia.cloud` is complaining: ` $ sudo run-pupp... [17:11:06] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-RecentPageStats, 07Wikimedia-extension-review-queue: Deploy RecentPageStats extension to Beta Cluster - https://phabricator.wikimedia.org/T435869#12253129 (10bd808) @Jnanaranjan_sahu This step from the checklist is honestly the most important these days:... [17:13:56] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-RecentPageStats, 07Wikimedia-extension-review-queue: Deploy RecentPageStats extension to Beta Cluster - https://phabricator.wikimedia.org/T435869#12253159 (10bd808) > MIT License — Copyright (c) 2026 Jnanaranjan Sahu MediaWiki is licensed under GPL-2.0-o... [17:16:41] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-RecentPageStats, 07Wikimedia-extension-review-queue: Deploy RecentPageStats extension to Beta Cluster - https://phabricator.wikimedia.org/T435869#12253171 (10bd808) 05Open→03Stalled Blocked on {T435890} [17:17:20] (03approved) 10brennen: deploy-promote: Verify the promotion with deployment-info.php [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1280 (https://phabricator.wikimedia.org/T434726) (owner: 10dancy) [17:19:00] 10Beta-Cluster-Infrastructure, 10Application Security Reviews, 10MediaWiki-extensions-RecentPageStats, 06secscrum, and 2 others: Application Security Review for RecentPageStats extension - https://phabricator.wikimedia.org/T435890#12253184 (10taavi) This request does not seem to follow the template require... [17:29:28] 10Beta-Cluster-Infrastructure, 10Shellbox, 10SyntaxHighlight: SyntaxHighlight not working on Beta Cluster wikis (firejail creation failure) - https://phabricator.wikimedia.org/T435866#12253219 (10bd808) [17:43:03] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-acme-chief07 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:51:09] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-acme-chief{05,07} - https://phabricator.wikimedia.org/T435991 (10bd808) 03NEW [18:02:57] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-acme-chief{05,07} - https://phabricator.wikimedia.org/T435991#12253408 (10bd808) I can't find anything related to this hiera setting that has obviously changed recently. This includes any sign of why this puppet module would have applied cleanly before n... [18:04:27] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-acme-chief{07,08} - https://phabricator.wikimedia.org/T435991#12253412 (10bd808) [18:08:40] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Zuul: ci: gate-and-submit not triggering - https://phabricator.wikimedia.org/T435995 (10Les4353) 03NEW [18:09:17] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Zuul: ci: gate-and-submit not triggering - https://phabricator.wikimedia.org/T435995#12253449 (10Les4353) p:05Triage→03Unbreak! [18:13:13] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Zuul: ci: gate-and-submit not triggering - https://phabricator.wikimedia.org/T435995#12253455 (10Jdforrester-WMF) Can you point to a patchset for which this is the case? For example https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1328342 is going through... [18:14:24] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Zuul: ci: gate-and-submit not triggering - https://phabricator.wikimedia.org/T435995#12253458 (10Les4353) >>! 在T435995#12253455中，@Jdforrester-WMF写道： > Can you point to a patchset for which this is the case? For example https://gerrit.wikimedia.org/r/c/pywi... [18:14:54] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul: ci: gate-and-submit not triggering - https://phabricator.wikimedia.org/T435995#12253459 (10Les4353) [18:17:55] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul: ci: gate-and-submit not triggering - https://phabricator.wikimedia.org/T435995#12253466 (10taavi) p:05Unbreak!→03Triage Please show a specific example. [18:18:33] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul: ci: gate-and-submit not triggering - https://phabricator.wikimedia.org/T435995#12253470 (10Aklapper) @Les4353: Hi, please use the bug report form (linked from the top of the task creation page) to create bug reports, and... [18:18:46] Amir1: seems you and obvious-dust are seeing the same error. [18:18:56] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul: ci: gate-and-submit not triggering - https://phabricator.wikimedia.org/T435995#12253472 (10Les4353) >>! 在T435995#12253466中，@taavi写道： > Please show a specific example. https://gerrit.wikimedia.org/r/c/1329156 [18:19:59] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul: ci: gate-and-submit not triggering - https://phabricator.wikimedia.org/T435995#12253473 (10taavi) 05Open→03Invalid That patch has an open parent patch without a +2 vote. [18:48:07] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Zuul: ci: gate-and-submit not triggering - https://phabricator.wikimedia.org/T435995#12253651 (10Aklapper) [19:01:20] hashar: Alas the Postgres venv is broken with my Quibble venv. :-( Will push a revert here and a fix there. [19:03:50] (03PS1) 10Jforrester: jjb: Revert Quibble jobs to the pre-venv images [integration/config] - 10https://gerrit.wikimedia.org/r/1329353 [19:04:40] (03PS2) 10Jforrester: jjb: Revert Quibble jobs to the pre-venv images [integration/config] - 10https://gerrit.wikimedia.org/r/1329353 [19:06:27] (03PS1) 10Jforrester: backend: Use venv for Quibble on Postgres [integration/quibble] - 10https://gerrit.wikimedia.org/r/1329356 [19:06:43] (03CR) 10Jforrester: [C:03+2] jjb: Revert Quibble jobs to the pre-venv images [integration/config] - 10https://gerrit.wikimedia.org/r/1329353 (owner: 10Jforrester) [19:08:24] (03Merged) 10jenkins-bot: jjb: Revert Quibble jobs to the pre-venv images [integration/config] - 10https://gerrit.wikimedia.org/r/1329353 (owner: 10Jforrester) [19:12:58] (03PS2) 10Vaughn Walters: jjb: [catalyst-daily-*] Publish JUnit test results [integration/config] - 10https://gerrit.wikimedia.org/r/1329349 (https://phabricator.wikimedia.org/T435988) [19:12:58] (03CR) 10Vaughn Walters: [C:03+2] "@phed" [integration/config] - 10https://gerrit.wikimedia.org/r/1329349 (https://phabricator.wikimedia.org/T435988) (owner: 10Vaughn Walters) [19:15:47] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-*] Publish JUnit test results [integration/config] - 10https://gerrit.wikimedia.org/r/1329349 (https://phabricator.wikimedia.org/T435988) (owner: 10Vaughn Walters) [19:29:36] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-RecentPageStats, 07Wikimedia-extension-review-queue: Deploy RecentPageStats extension to Beta Cluster - https://phabricator.wikimedia.org/T435869#12253864 (10Jnanaranjan_sahu) Thanks @bd808! 1. **Licensing**: I've updated the license to `GPL-2.0-or-later... [19:37:13] 06Release-Engineering-Team, 10dev-images, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Migrate Patchdemo & Catalyst to PHP 8.5, Node 24 - https://phabricator.wikimedia.org/T435368#12253900 (10brennen) [19:50:16] James_F: I guess we can't test everything! :/ If you have a trace/error to put in a task, I am pretty sure it is an easy fix [19:51:07] most probably Quibble invokes `pg_virtualenv` via subprocess.run without the parent environment passed to it [19:51:46] yes that is it [19:52:26] quibble/backend.py has the `subprocess.Popen()` which set an env for QUIBBLE_TMPFILE and TMPDIR [19:52:51] we would need a dict populated with os.environ then update it with the *TMP* variables and pass the result to Popen [19:53:12] I am happy to do it tomorrow if I don't forget about it ;) [20:07:50] hashar: Fix already written. [20:07:59] hashar: No rush. :-) See you tomorrow. [20:08:09] <3 [20:35:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [20:35:09] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/labs/private on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T436020 (10wmcs-alerts) 03NEW [20:38:03] FIRING: [2x] PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-acme-chief07 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:38:03] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-acme-chief07 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [20:38:08] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-acme-chief08 in project deployment-prep - https://phabricator.wikimedia.org/T436021 (10wmcs-alerts) 03NEW [20:38:09] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-acme-chief08 in project deployment-prep - https://phabricator.wikimedia.org/T436022 (10wmcs-alerts) 03NEW [20:39:27] James_F: We do have some folks blocked by the postgresql problem [20:46:48] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-acme-chief08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [20:47:00] 06Release-Engineering-Team, 10Catalyst (PatchDemo): "wgSFSIPListLocation does not appear to be a valid URL" when trying to create a patchdemo with all extensions - https://phabricator.wikimedia.org/T420302#12254198 (10brennen) Seems like the above change worked. {F100098317} [20:47:01] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎), 10Cloud-VPS (Debian Bullseye Deprecation): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12254200 (10Southparkfan) @bd808 and I have The plan still needs work before it can be shared and used.... [20:48:03] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-acme-chief08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:49:00] (03open) 10dancy: train.py: Initialize self.train_is_at at top of update() [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1283 [20:49:07] 06Release-Engineering-Team, 10Catalyst (PatchDemo), 07Essential-Work: "wgSFSIPListLocation does not appear to be a valid URL" when trying to create a patchdemo with all extensions - https://phabricator.wikimedia.org/T420302#12254205 (10brennen) 05Open→03Resolved a:03brennen [20:50:17] dancy: No-one should be blocked by the PG issue; I reverted the change. The jobs probably just need re-C+2-ing? [20:51:09] James_F: Thanks! I'll pass that information along [20:51:38] dancy: I could go hunt down patches and re-queue them, but I’d probably miss some. [20:53:14] (03merge) 10dancy: train.py: Initialize self.train_is_at at top of update() [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1283 [20:54:59] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-acme-chief{07,08} - https://phabricator.wikimedia.org/T435991#12254228 (10BCornwall) I just created these instances, as they're going to be replacing 05/06 (bullseye→bookworm) [20:55:54] Thank you @James_F [20:57:07] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-acme-chief{07,08} - https://phabricator.wikimedia.org/T435991#12254231 (10BCornwall) Fixed with https://gerrit.wikimedia.org/r/plugins/gitiles/labs/private/+/cdc68fb6637c27d5893879d5c65478bc68845f26 [20:57:14] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-acme-chief{07,08} - https://phabricator.wikimedia.org/T435991#12254232 (10BCornwall) 05Open→03Resolved a:03BCornwall [21:08:06] James_F: Confirmed that a fresh +2 has moved people on. Thanks! [21:50:31] elukey: got a not-found for docker pull docker-registry.wikimedia.org/releng/composer-test-php84:8.4.7-s3 [21:57:03] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [21:57:05] (this may not be an image that should be used, i'm pulling on a thread in patchdemo tests) [21:57:09] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T436024 (10wmcs-alerts) 03NEW [21:58:45] 06Release-Engineering-Team: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12254395 (10dancy) ` dancy@deploy1003:~$ scap analyze-logstash --scope pretrain production 21:57:48 Analyzing logstash history for scope: pretrain, stage: prod... [22:25:23] 06Release-Engineering-Team (Doing 😎): Configure additional httpbb checks to perform during a deployment - https://phabricator.wikimedia.org/T428972#12254524 (10bd808) 05Open→03Resolved {T435419} will follow up with the new limits [22:42:03] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [22:47:03] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-deploy06 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [22:47:10] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-deploy06 in project deployment-prep - https://phabricator.wikimedia.org/T436030 (10wmcs-alerts) 03NEW [22:47:22] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12254573 (10Lars) I see that I've mistaken the instance from the... [22:57:03] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-deploy06 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [23:19:28] (03PS1) 10Pwangai: castor: Print the load outcome only once [integration/config] - 10https://gerrit.wikimedia.org/r/1329398 (https://phabricator.wikimedia.org/T436032) [23:26:06] (03PS2) 10Pwangai: castor: Print the load outcome only once [integration/config] - 10https://gerrit.wikimedia.org/r/1329398 (https://phabricator.wikimedia.org/T436032) [23:37:07] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12254725 (10bd808) When I [[https://gitlab.wiki... [23:39:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-deploy06 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:39:15] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-deploy06 on project deployment-prep - https://phabricator.wikimedia.org/T436038 (10wmcs-alerts) 03NEW [23:47:31] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-deploy06 in project deployment-prep - https://phabricator.wikimedia.org/T436030#12254769 (10bd808) 05Open→03Resolved a:03bd808 T435393#12254723 [23:48:01] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-deploy06 on project deployment-prep - https://phabricator.wikimedia.org/T436038#12254777 (10bd808) 05Open→03Resolved a:03bd808 T435393#12254723 [23:51:27] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/labs/private on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T436020#12254798 (10bd808) a:03BCornwall I think this was Brett working through things on deployment-acme-chief{07,08}....