[00:33:25] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 10MW-Interfaces-Team (MWI-Sprint-11 (2025-06-03 to 2025-06-17)): Release MW 1.44.0-rc.0 - https://phabricator.wikimedia.org/T390697#10945473 (10Reedy) 05Open→03Resolved While this was posted to wikitech-l [1] and mediawiki-l [2], it wasn't,... [00:33:55] 10MediaWiki-Releasing, 10CheckUser, 06Trust and Safety Product Team, 05MW-1.44-release: Bundle CheckUser extension with MediaWiki - https://phabricator.wikimedia.org/T317147#10945476 (10Reedy) 05Open→03Resolved a:03Reedy [00:34:11] 10MediaWiki-Releasing, 10TemplateStyles, 05MW-1.44-release: Bundle Extension:TemplateStyles with MediaWiki core - https://phabricator.wikimedia.org/T327006#10945479 (10Reedy) 05Open→03Resolved a:03Reedy [00:34:25] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10945483 (10Reedy) 05Open→03Resolved a:03Reedy [00:45:24] (03CR) 10BryanDavis: [C:03+1] Remove unused grant for tools.wmflabs.org [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163056 (https://phabricator.wikimedia.org/T227626) (owner: 10Krinkle) [00:56:12] 10MediaWiki-Releasing, 10TemplateStyles, 05MW-1.44-release: Bundle Extension:TemplateStyles with MediaWiki core - https://phabricator.wikimedia.org/T327006#10945513 (10Bugreporter2) There's still no mention in the release notes! [00:56:18] 10MediaWiki-Releasing, 10CheckUser, 06Trust and Safety Product Team, 05MW-1.44-release: Bundle CheckUser extension with MediaWiki - https://phabricator.wikimedia.org/T317147#10945514 (10Bugreporter2) There's still no mention in the release notes! [00:56:55] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10945515 (10Bugreporter2) Why aren't these mentioned in the 1.44 release notes? https://github.c... [01:00:24] 10GitLab (Infrastructure), 10Cloud-VPS, 06collaboration-services, 10cloud-services-team (FY2024/2025-Q3-Q4): Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10945531 (10Andrew) [01:33:24] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10945600 (10Reedy) You’re very welcome to submit a patch if you’re enthusiastic about this issue. [02:08:33] (03PS2) 10Krinkle: Remove unused grant for tools.wmflabs.org [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163056 (https://phabricator.wikimedia.org/T227626) [02:09:44] (03PS2) 10Krinkle: Add support for Beta Cluster via "beta.wmcloud.org" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163057 (https://phabricator.wikimedia.org/T289318) [02:09:52] (03CR) 10Krinkle: "Fixed duplicate." [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163056 (https://phabricator.wikimedia.org/T227626) (owner: 10Krinkle) [02:12:37] (03PS1) 10Krinkle: Update logstash-beta.wmflabs.org to beta-logs.wmcloud.org [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163499 (https://phabricator.wikimedia.org/T289318) [02:19:05] !log Add mapping for performance.wikimedia.beta.wmcloud.org to profile::trafficserver::backend::mapping_rules in Hiera under deployment-cache-text prefix. Same mapping as the wmflabs version. T289318 [02:19:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:19:07] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [02:22:11] (03PS2) 10Krinkle: Update logstash and xhgui links from wmflabs.org to wmcloud.org [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163499 (https://phabricator.wikimedia.org/T289318) [03:59:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-docker-cxserver02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [03:59:34] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-cxserver02 on project deployment-prep - https://phabricator.wikimedia.org/T397806 (10wmcs-alerts) 03NEW [04:44:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-docker-cxserver02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [04:44:34] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-cxserver02 on project deployment-prep - https://phabricator.wikimedia.org/T397806#10945768 (10wmcs-alerts) [07:17:09] 10Beta-Cluster-Infrastructure, 10GitLab (CI & Job Runners), 06collaboration-services, 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#10945896 (10Jelto) >>! In T397591#10943336, @LucasWerkmeister wrote: > According to https://wikitech.wikim... [08:04:56] 10Continuous-Integration-Infrastructure, 07Jenkins: Jenkins jobs running on production agents should support switch over - https://phabricator.wikimedia.org/T397815 (10hashar) 03NEW [08:05:26] 10Continuous-Integration-Infrastructure, 07Jenkins: Jenkins jobs running on production agents should support switch over - https://phabricator.wikimedia.org/T397815#10946006 (10hashar) [08:43:03] (03PS1) 10Hashar: jjb: change train-deploy-notes label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163705 (https://phabricator.wikimedia.org/T225031) [08:47:28] (03PS1) 10Hashar: jjb: change fail-archived-repositories label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163706 (https://phabricator.wikimedia.org/T397815) [08:50:14] (03PS2) 10Hashar: jjb: change fail-archived-repositories label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163706 (https://phabricator.wikimedia.org/T397815) [08:50:14] (03PS1) 10Hashar: jjb: change trigger-pipeline jobs label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163708 (https://phabricator.wikimedia.org/T397815) [08:52:00] !log jenkins: updated jobs fail-archived-repositories, train-deploy-notes and trigger-* to use label productionAgents rather than contint1002 # T397815 [08:52:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:52:02] T397815: Jenkins jobs running on production agents should support switch over - https://phabricator.wikimedia.org/T397815 [08:52:09] (03CR) 10Hashar: [C:03+2] jjb: change trigger-pipeline jobs label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163708 (https://phabricator.wikimedia.org/T397815) (owner: 10Hashar) [08:52:12] (03CR) 10Hashar: [C:03+2] jjb: change fail-archived-repositories label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163706 (https://phabricator.wikimedia.org/T397815) (owner: 10Hashar) [08:52:17] (03CR) 10Hashar: [C:03+2] jjb: change train-deploy-notes label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163705 (https://phabricator.wikimedia.org/T225031) (owner: 10Hashar) [08:54:05] (03PS1) 10Hashar: jjb: change publish-to-doc label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163709 (https://phabricator.wikimedia.org/T397815) [08:55:00] (03Merged) 10jenkins-bot: jjb: change train-deploy-notes label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163705 (https://phabricator.wikimedia.org/T225031) (owner: 10Hashar) [08:55:03] (03CR) 10Hashar: [C:03+2] jjb: change publish-to-doc label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163709 (https://phabricator.wikimedia.org/T397815) (owner: 10Hashar) [08:55:14] (03Merged) 10jenkins-bot: jjb: change fail-archived-repositories label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163706 (https://phabricator.wikimedia.org/T397815) (owner: 10Hashar) [08:55:15] (03Merged) 10jenkins-bot: jjb: change trigger-pipeline jobs label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163708 (https://phabricator.wikimedia.org/T397815) (owner: 10Hashar) [08:55:45] !log jenkins: updated job publish-to-doc to use label productionAgents rather than contint1002 # T397815 [08:55:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:56:28] (03Merged) 10jenkins-bot: jjb: change publish-to-doc label to productionAgents [integration/config] - 10https://gerrit.wikimedia.org/r/1163709 (https://phabricator.wikimedia.org/T397815) (owner: 10Hashar) [08:57:42] 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Jenkins jobs running on production agents should support switch over - https://phabricator.wikimedia.org/T397815#10946257 (10hashar) 05Open→03Resolved a:03hashar Jobs are now using the [[ https://integration.wikimedia.org/ci/la... [09:10:21] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T392177#10946304 (10thiemowmde) [10:08:48] (03PS1) 10Hashar: jjb: move publish-to-doc fetching to a macro [integration/config] - 10https://gerrit.wikimedia.org/r/1163720 (https://phabricator.wikimedia.org/T392894) [10:08:49] (03PS1) 10Hashar: jjb: doc-publish support retrieving specific files [integration/config] - 10https://gerrit.wikimedia.org/r/1163721 (https://phabricator.wikimedia.org/T392894) [10:08:51] (03PS1) 10Hashar: jjb: create publisher for Prometheus metrics [integration/config] - 10https://gerrit.wikimedia.org/r/1163722 (https://phabricator.wikimedia.org/T392894) [10:10:41] (03CR) 10CI reject: [V:04-1] jjb: create publisher for Prometheus metrics [integration/config] - 10https://gerrit.wikimedia.org/r/1163722 (https://phabricator.wikimedia.org/T392894) (owner: 10Hashar) [10:12:04] (03PS2) 10Hashar: jjb: create publisher for Prometheus metrics [integration/config] - 10https://gerrit.wikimedia.org/r/1163722 (https://phabricator.wikimedia.org/T392894) [10:20:38] 10Scap (SpiderPig 🕸️): Don’t obscure the typed OTP in SpiderPig - https://phabricator.wikimedia.org/T397505#10946584 (10Lucas_Werkmeister_WMDE) Works for me now, thanks a lot! [10:53:12] 06Release-Engineering-Team, 10Scap (SpiderPig 🕸️): SpiderPig session / login broken - https://phabricator.wikimedia.org/T397457#10946659 (10Lucas_Werkmeister_WMDE) FWIW, during one backport today ([SpiderPig #249](https://spiderpig.wikimedia.org/jobs/249)) I had to provide a new OTP three times. (Before the la... [11:19:30] (03PS3) 10Hashar: jjb: create publisher for Prometheus metrics [integration/config] - 10https://gerrit.wikimedia.org/r/1163722 (https://phabricator.wikimedia.org/T392894) [12:48:46] 06Release-Engineering-Team, 07Epic, 05MW-1.43-release, 05MW-1.44-release, and 2 others: [REST Sandbox] Remove SwaggerUI from MediaWiki Releases - https://phabricator.wikimedia.org/T397470#10946912 (10Atieno) a:03Atieno [13:25:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:35:39] (03PS1) 10Hashar: jjb: send seelenium-daily test results to Prometheus [integration/config] - 10https://gerrit.wikimedia.org/r/1163781 (https://phabricator.wikimedia.org/T392894) [13:50:56] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10947228 (10Tgr) The easy way to make private security patches was to make the edit on the deployment host in `/srv/mediawiki-sta... [13:53:12] (03CR) 10Hashar: [C:03+2] Drop six dependency by using Python 3 dict.items() [integration/quibble] - 10https://gerrit.wikimedia.org/r/1163388 (owner: 10Hashar) [13:56:34] (03CR) 10Hashar: [C:03+2] zuul: [mediawiki/extensions/CampaignEvents] add CLDR as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1163347 (https://phabricator.wikimedia.org/T397271) (owner: 10Cmelo) [13:57:01] (03CR) 10Hashar: [C:04-2] "It is all lower cases :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1163347 (https://phabricator.wikimedia.org/T397271) (owner: 10Cmelo) [13:57:46] (03PS3) 10Hashar: zuul: [mediawiki/extensions/CampaignEvents] add cldr as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1163347 (https://phabricator.wikimedia.org/T397271) (owner: 10Cmelo) [13:57:58] (03CR) 10Hashar: [C:03+2] zuul: [mediawiki/extensions/CampaignEvents] add cldr as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1163347 (https://phabricator.wikimedia.org/T397271) (owner: 10Cmelo) [13:58:18] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Re-create GitLab test instance - https://phabricator.wikimedia.org/T396622#10947262 (10Jelto) I was able to re-create the test instance at https://gitlab.devtools.wmcloud.org/ using the backup which was sto... [13:59:20] (03Merged) 10jenkins-bot: zuul: [mediawiki/extensions/CampaignEvents] add cldr as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1163347 (https://phabricator.wikimedia.org/T397271) (owner: 10Cmelo) [14:07:47] 10WikimediaDebug, 10Arc-Lamp, 10observability, 07Regression, 10SRE Observability (FY2025/2026-Q1): https://performance.wikimedia.org/php-profiling/ leads to 404 for all listed sources - https://phabricator.wikimedia.org/T391516#10947315 (10lmata) [14:10:56] (03Merged) 10jenkins-bot: Drop six dependency by using Python 3 dict.items() [integration/quibble] - 10https://gerrit.wikimedia.org/r/1163388 (owner: 10Hashar) [14:11:05] (03CR) 10Hashar: [C:03+2] Zuul: Add Arendpieter to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1163334 (owner: 10Majavah) [14:12:16] (03Merged) 10jenkins-bot: Zuul: Add Arendpieter to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1163334 (owner: 10Majavah) [14:16:03] (03CR) 10Hashar: [C:03+2] "I have updated https://integration.wikimedia.org/ci/job/publish-to-doc/" [integration/config] - 10https://gerrit.wikimedia.org/r/1163720 (https://phabricator.wikimedia.org/T392894) (owner: 10Hashar) [14:16:21] should be a noop [14:16:55] (03CR) 10Hashar: [C:03+2] jjb: doc-publish support retrieving specific files [integration/config] - 10https://gerrit.wikimedia.org/r/1163721 (https://phabricator.wikimedia.org/T392894) (owner: 10Hashar) [14:17:22] (03Merged) 10jenkins-bot: jjb: move publish-to-doc fetching to a macro [integration/config] - 10https://gerrit.wikimedia.org/r/1163720 (https://phabricator.wikimedia.org/T392894) (owner: 10Hashar) [14:17:47] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS: ZuulDevOpsBot user can create but not delete a cluster template - https://phabricator.wikimedia.org/T396932#10947384 (10joanna_borun) p:05Triage→03Medium [14:18:09] (03Merged) 10jenkins-bot: jjb: doc-publish support retrieving specific files [integration/config] - 10https://gerrit.wikimedia.org/r/1163721 (https://phabricator.wikimedia.org/T392894) (owner: 10Hashar) [14:18:34] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Re-create GitLab test instance - https://phabricator.wikimedia.org/T396622#10947393 (10Jelto) >>! In T396622#10947262, @Jelto wrote: > Login/OIDC seems broken The test instance used the `gitlab_replica_oid... [14:19:09] (03PS7) 10Hashar: jjb: create publisher for Prometheus metrics [integration/config] - 10https://gerrit.wikimedia.org/r/1163722 (https://phabricator.wikimedia.org/T392894) [14:23:23] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10947437 (10Jelto) [14:23:27] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Re-create GitLab test instance - https://phabricator.wikimedia.org/T396622#10947438 (10Jelto) [14:23:39] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10947440 (10Jelto) [14:23:45] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Re-create GitLab test instance - https://phabricator.wikimedia.org/T396622#10947441 (10Jelto) [14:25:21] (03PS8) 10Hashar: jjb: create publisher for Prometheus metrics [integration/config] - 10https://gerrit.wikimedia.org/r/1163722 (https://phabricator.wikimedia.org/T392894) [14:25:23] (03CR) 10Zuul test: "Merge Failed." [integration/config] - 10https://gerrit.wikimedia.org/r/1163722 (https://phabricator.wikimedia.org/T392894) (owner: 10Hashar) [14:26:00] (03PS9) 10Hashar: jjb: create publisher for Prometheus metrics [integration/config] - 10https://gerrit.wikimedia.org/r/1163722 (https://phabricator.wikimedia.org/T392894) [14:26:58] (03CR) 10Hashar: "Instead of finding out how to deploy https://gitlab.wikimedia.org/phedenskog/push-to-pushgateway/-/blob/main/push.py , I went with a shel" [integration/config] - 10https://gerrit.wikimedia.org/r/1163722 (https://phabricator.wikimedia.org/T392894) (owner: 10Hashar) [14:28:19] (03PS2) 10Hashar: jjb: send seelenium-daily test results to Prometheus [integration/config] - 10https://gerrit.wikimedia.org/r/1163781 (https://phabricator.wikimedia.org/T392894) [14:28:27] (03PS3) 10Hashar: jjb: send selenium-daily test results to Prometheus [integration/config] - 10https://gerrit.wikimedia.org/r/1163781 (https://phabricator.wikimedia.org/T392894) [14:43:41] 10GitLab (CI & Job Runners), 06collaboration-services: Persistent puppet failure on gitlab-runner-1008.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T397641#10947571 (10Jelto) 05Open→03Resolved p:05Triage→03Medium a:03Jelto This should be fixed, the test instance is working... [14:44:18] 10GitLab (CI & Job Runners), 06collaboration-services: Persistent puppet failure on gitlab-runner-1008.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T397641#10947584 (10dancy) Thanks @Jelto ! [14:49:07] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deleteme is down - https://phabricator.wikimedia.org/T397777#10947601 (10dancy) 05Open→03Resolved a:03dancy There was a WMCS network outage while Andrew was working on T397517. [14:52:06] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS: ZuulDevOpsBot user can create but not delete a cluster template - https://phabricator.wikimedia.org/T396932#10947617 (10bd808) [14:52:09] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#10947616 (10bd808) [14:54:42] 06Release-Engineering-Team, 10Scap, 10Dumps-Generation: scap needs to be k8s-cluster aware - https://phabricator.wikimedia.org/T388761#10947625 (10BTullis) 05Open→03Resolved a:03BTullis I think that we can call this done. We're now automatically getting our updated mediawiki pod spec whenever scap... [15:09:49] 06Release-Engineering-Team, 06MediaWiki-Engineering, 06serviceops: Deprecate mwdebugXXXX hosts - https://phabricator.wikimedia.org/T397498#10947665 (10jijiki) 05Open→03In progress p:05Triage→03Medium [16:21:42] PHP Fatal error: Cannot declare class ComposerAutoloaderInit_mediawiki_vendor, because the name is already in use in /home/hashar/projects/mediawiki/core/vendor/composer/autoload_real.php on line 5 [16:21:45] when mediawiki makes me happy [16:22:12] it is $IP [16:22:57] err MW_INSTALL_PATH [16:34:21] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10947997 (10jijiki) >>! In T396767#10947228, @Tgr wrote: > The easy way to make private security patches was to make the edit on... [17:05:18] !log Upgrading scap to 4.182.0 in beta cluster [17:05:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:06:01] Project beta-scap-sync-world build #212494: 04FAILURE in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/212494/ [17:07:26] fun :) [17:09:55] Yippee, build fixed! [17:09:55] Project beta-scap-sync-world build #212495: 09FIXED in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/212495/ [17:10:28] pro tip: if you remove the code that runs a deployment during a deployment, that makes the deployment unhappy. In retrospect, maybe obvious! [17:19:45] 10Phabricator: Make it possible to change the 'default' column of a Phabricator workboard - https://phabricator.wikimedia.org/T397862 (10A_smart_kitten) 03NEW [17:23:02] (03update) 10thcipriani: Update scap beta release scripts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/827 [17:25:36] (03merge) 10thcipriani: Update scap beta release scripts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/827 [17:31:42] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10948139 (10Tgr) It would be great to find a solution for it. I don't do security changes often though, so maybe worth asking som... [17:45:57] 06Release-Engineering-Team, 07Code-Health: Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites - https://phabricator.wikimedia.org/T190891#10948160 (10SLong-WMF) [18:07:58] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T397717#10948219 (10bd808) p:05Triage→03Medium `COUNTEREXAMPLE CONFLICT (content): Merge conflict in modules/profil... [18:15:44] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T397717#10948241 (10bd808) Manually rebased as https://gerrit.wikimedia.org/r/c/operations/puppet/+/1137013/2 [18:19:09] !log Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1137013/2 to deployment-puppetserver-1 (T397717) [18:19:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:19:14] T397717: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T397717 [18:25:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:25:35] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T397717#10948252 (10wmcs-alerts) [18:26:30] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T397717#10948253 (10bd808) 05Open→03Resolved a:03bd808 [18:28:23] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-cxserver02 on project deployment-prep - https://phabricator.wikimedia.org/T397806#10948276 (10bd808) 05Open→03Invalid Fixed itself apparently [18:34:28] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10948310 (10sbassett) >>! In T396767#10948138, @Tgr wrote: > It would be great to find a solution for it. I don't do security cha... [19:02:26] 10Phabricator: Make it possible to change the 'default' column of a Phabricator workboard - https://phabricator.wikimedia.org/T397862#10948417 (10A_smart_kitten) [19:29:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:29:37] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T397872 (10wmcs-alerts) 03NEW [19:31:32] 10GitLab (Infrastructure), 10Cloud-VPS, 06collaboration-services, 10cloud-services-team (FY2024/2025-Q3-Q4): Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10948470 (10Andrew) 05Open→03Resolved I think that gitlab-prod-backup is now unstuck, so c... [19:46:27] 06Project-Admins: Create a tag for PagePermissions extension - https://phabricator.wikimedia.org/T397873 (10Techwizzie) 03NEW [19:47:25] (03CR) 10Ahmon Dancy: [C:03+1] "LGTM!" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163499 (https://phabricator.wikimedia.org/T289318) (owner: 10Krinkle) [19:57:27] (03CR) 10Ebomani: [C:03+1] "Looks good to me! Tested and verified that for the new (non-legacy) Patchdemo related changes we get redirect links in the 'Checks' tab to" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1163289 (https://phabricator.wikimedia.org/T391866) (owner: 10Jeena Huneidi) [20:05:40] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T397872#10948581 (10bd808) `lang=shell-session,counterexample bd808@deployment-puppetserver-1:~$ sudo -i puppet agent -tv Info: Using environment 'producti... [20:12:24] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T397872#10948602 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/c487a1b4328b87cd66fc8ceda446b1f5ce0f1c41%5E%21/#F0 ` diff... [20:12:54] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T397872#10948603 (10bd808) `lang=shell-session,counterexample bd808@deployment-puppetserver-1:~$ sudo -i puppet agent -tv Info: Using environment 'producti... [20:18:35] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T397872#10948608 (10bd808) More fallout from https://gerrit.wikimedia.org/r/c/operations/puppet/+/1161869. My manual rebase in https://gerrit.wikimedia.or... [20:23:44] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T397872#10948623 (10bd808) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1137013/2..3/modules/profile/manifests/puppetserver/volatile.pp [20:24:17] !log Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1137013/3 to deployment-puppetserver-1 (T397872) [20:24:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:24:23] T397872: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T397872 [20:25:58] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T397872#10948628 (10bd808) 05Open→03Resolved p:05Triage→03Medium a:03bd808 `lang=shell-session bd808@deployment-puppetserver-1:~$ sudo -i pup... [20:39:41] (03open) 10arlolra: Stop branching VipsScaler for Wikimedia production [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/185 (https://phabricator.wikimedia.org/T290759) [20:46:01] 10Beta-Cluster-Infrastructure, 07Puppet: /usr/local/bin/puppetserver-deploy-code emits scary looking error messages during a `git rebase` operation - https://phabricator.wikimedia.org/T397877 (10bd808) 03NEW [20:49:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:49:36] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T397879 (10wmcs-alerts) 03NEW [21:16:46] !log Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1163883/1 to deployment-puppetserver-1 (T397877) [21:16:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:16:48] T397877: /usr/local/bin/puppetserver-deploy-code emits scary looking error messages during a `git rebase` operation - https://phabricator.wikimedia.org/T397877 [21:21:28] (03open) 10dancy: scap php: New subcommand [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/859 [21:21:31] (03update) 10dancy: scap php: New subcommand [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/859 [21:21:33] 10Beta-Cluster-Infrastructure, 13Patch-For-Review, 07Puppet: /usr/local/bin/puppetserver-deploy-code emits scary looking error messages during a `git rebase` operation - https://phabricator.wikimedia.org/T397877#10948815 (10bd808) `lang=shell-session bd808@deployment-puppetserver-1:~$ sudo -i puppet agent -t... [21:26:59] 10Beta-Cluster-Infrastructure, 13Patch-For-Review, 07Puppet: /usr/local/bin/puppetserver-deploy-code emits scary looking error messages during a `git rebase` operation - https://phabricator.wikimedia.org/T397877#10948820 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [21:27:54] (03open) 10dancy: create-databases: Use scap to run php scripts [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/144 [21:27:57] (03update) 10dancy: create-databases: Use scap to run php scripts [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/144 [21:29:24] (03update) 10dancy: scap php: New subcommand [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/859 [21:38:08] (03update) 10thcipriani: spiderpig: Revise HTTP access logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/858 (owner: 10dancy) [21:40:10] (03merge) 10thcipriani: spiderpig: Revise HTTP access logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/858 (owner: 10dancy) [21:40:47] (03update) 10dancy: scap php: New subcommand [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/859 [21:41:08] (03update) 10dancy: create-databases: Use scap to run php scripts [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/144 [21:41:51] (03update) 10dancy: scap php: New subcommand [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/859 [21:43:13] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 06serviceops: helmfile/scap does not reliably bootstrap mediawiki - https://phabricator.wikimedia.org/T397685#10948866 (10Scott_French) I was chatting with @dancy earlier today about what might have caused this, and it's kind of a puzzling one. Super-naivel... [22:12:40] 10GitLab (Project Migration), 10Phabricator Antivandalism Extension: Move https://gerrit.wikimedia.org/r/phabricator/antivandalism to GitLab - https://phabricator.wikimedia.org/T349662#10948921 (10A_smart_kitten) [22:16:23] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10948922 (10bd808) `scap pull` is tied to the legacy bare metal PHP deployment process that is rapidly becoming obsolete. Improvi... [22:22:45] 06Release-Engineering-Team, 10Scap (SpiderPig 🕸️): SpiderPig session / login broken - https://phabricator.wikimedia.org/T397457#10948935 (10bd808) I have had CAS wedge itself into an infinite redirect loop with both spiderpig and horizon as clients. I don't have any insight into why this happens, but I have ha... [23:33:05] 10Beta-Cluster-Infrastructure, 10GitLab (CI & Job Runners), 06collaboration-services, 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#10949028 (10bd808) >>! In T397591#10945896, @Jelto wrote: > The current process of explicitly allow-listin... [23:49:00] (03CR) 10BryanDavis: [C:03+1] "Oops. Sorry my eye missed that on the earlier review." [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163056 (https://phabricator.wikimedia.org/T227626) (owner: 10Krinkle) [23:49:46] (03CR) 10BryanDavis: [C:03+1] Add support for Beta Cluster via "beta.wmcloud.org" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163057 (https://phabricator.wikimedia.org/T289318) (owner: 10Krinkle) [23:51:02] (03CR) 10BryanDavis: [C:03+1] Update logstash and xhgui links from wmflabs.org to wmcloud.org [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163499 (https://phabricator.wikimedia.org/T289318) (owner: 10Krinkle) [23:53:21] bd808: what might I do for a +2? [23:54:20] Krinkle: dance! I can totally +2 the if you are ready. I guess I was trying not to put myself on the hook for an immediate release. :) [23:54:39] I see. Happy to do the release myself this time. [23:55:09] I think it's ready to go. The logstash and perf links are canonical on the new name. For mw sites in beta, it's merely allowing it, not per-se assuming one over the other. [23:55:31] I'm sort of interested in finding a way to automate the releases. I'm not sure how hard that would be, but it seems like a thing other people have probably figured out already. [23:56:17] Krinkle: yeah, that all makes sense. +2s incoming :) [23:56:47] (03CR) 10BryanDavis: [C:03+2] Remove unused grant for tools.wmflabs.org [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163056 (https://phabricator.wikimedia.org/T227626) (owner: 10Krinkle) [23:56:52] (03CR) 10BryanDavis: [C:03+2] Add support for Beta Cluster via "beta.wmcloud.org" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163057 (https://phabricator.wikimedia.org/T289318) (owner: 10Krinkle) [23:56:56] (03CR) 10BryanDavis: [C:03+2] Update logstash and xhgui links from wmflabs.org to wmcloud.org [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163499 (https://phabricator.wikimedia.org/T289318) (owner: 10Krinkle) [23:57:11] (03Merged) 10jenkins-bot: Remove unused grant for tools.wmflabs.org [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163056 (https://phabricator.wikimedia.org/T227626) (owner: 10Krinkle) [23:57:15] (03Merged) 10jenkins-bot: Add support for Beta Cluster via "beta.wmcloud.org" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163057 (https://phabricator.wikimedia.org/T289318) (owner: 10Krinkle) [23:57:18] (03Merged) 10jenkins-bot: Update logstash and xhgui links from wmflabs.org to wmcloud.org [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1163499 (https://phabricator.wikimedia.org/T289318) (owner: 10Krinkle) [23:57:54] thx. pre-release testing does differ in that what's in the .zip isn't all that's in the repo, but if there's a step there for testing the zip file between making and publishing, that'd be nice to automate indeed. [23:58:20] maybe that's an edge case low enough that we can always fixup the release, which isn't to bad given auto updates. [23:59:54] In a gitlab CI pipeline we could do things like produce the zip artifact and then do some tests with it.