[00:01:08] 10Continuous-Integration-Infrastructure (Zuul upgrade): Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#10930261 (10bd808) Note to self: `lang=irc [23:59] < bd808> oh poop. now I have gitlab CI constraints to work around :/ masked and protected v... [00:09:27] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services: Investigate Zuul worker nodes for post-merge/trusted jobs - https://phabricator.wikimedia.org/T396655#10930278 (10thcipriani) 05Openβ†’03Resolved a:03thcipriani Per discussion in the... [00:32:51] 10Continuous-Integration-Infrastructure (Zuul upgrade): Artifact storage for updated Zuul CI - https://phabricator.wikimedia.org/T397385 (10thcipriani) 03NEW [00:41:21] 10Continuous-Integration-Infrastructure (Zuul upgrade): Artifact storage for updated Zuul CI - https://phabricator.wikimedia.org/T397385#10930309 (10thcipriani) One storage driver available for zuul s3 compatible object storage (either s3 or swift). This would be akin to the current artifact storage in GitLab.... [02:32:49] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#10930338 (10Dzahn) >>! In T395938#10929023, @Corvus wrote: > Executor: > * Bind mount a directory with the global nodepool ssh private key (if you want,... [05:18:44] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 10Data-Persistence-Backup: Backups for new m1 database: zuul - https://phabricator.wikimedia.org/T396322#10930439 (10jcrespo) 05Openβ†’03Resolved a:03jcrespo Backups were setup at T394844 with the same policy as the res... [05:30:10] 10Phabricator (Upstream), 07Upstream: Unclear how to update email address of a Phabricator bot account - https://phabricator.wikimedia.org/T396881#10930449 (10Aklapper) [05:30:14] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Update to Phorge upstream 202x.xx release (the one after 2025.18) - https://phabricator.wikimedia.org/T393840#10930450 (10Aklapper) [06:56:21] 10Phabricator (Search): Searching by Document Status "Open" includes resolved tasks - https://phabricator.wikimedia.org/T397392 (10kostajh) 03NEW [07:40:40] 10Continuous-Integration-Infrastructure (Zuul upgrade): Artifact storage for updated Zuul CI - https://phabricator.wikimedia.org/T397385#10930664 (10hashar) [07:51:09] 10Continuous-Integration-Infrastructure (Zuul upgrade): Artifact storage for updated Zuul CI - https://phabricator.wikimedia.org/T397385#10930721 (10MatthewVernon) I think the answer is "you need a proxy if you want to do it externally", but it's a little "It Depends". If you set a suitable ACL, you can download... [07:54:04] 10Continuous-Integration-Infrastructure (Zuul upgrade): Artifact storage for updated Zuul CI - https://phabricator.wikimedia.org/T397385#10930739 (10hashar) For the proof of concept on WMCS (`zuul3` tenant), I have created an object storage container via Horizon. There is support for both {nav Swift} and {nav S3... [08:52:56] 10GitLab (Infrastructure), 10Cloud-VPS, 06collaboration-services, 10cloud-services-team (FY2024/2025-Q3-Q4): Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10931174 (10dcaro) p:05Triageβ†’03High [08:53:02] 10GitLab (Infrastructure), 10Cloud-VPS, 06collaboration-services, 10cloud-services-team (FY2024/2025-Q3-Q4): Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10931176 (10dcaro) [08:53:13] 10GitLab (Infrastructure), 10Cloud-VPS, 06collaboration-services, 10cloud-services-team (FY2024/2025-Q3-Q4): Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10931177 (10dcaro) a:03Andrew [09:03:20] 10Phabricator (Search): Searching by Document Status "Open" includes resolved tasks - https://phabricator.wikimedia.org/T397392#10931222 (10Bugreporter) See: {T157783} [09:04:58] 10Phabricator (Search): Searching by Document Status "Open" includes resolved tasks - https://phabricator.wikimedia.org/T397392#10931229 (10kostajh) Hmm. So in theory mentioning {T363858} fixes the index? [09:06:08] 10Phabricator (Search): Searching by Document Status "Open" includes resolved tasks - https://phabricator.wikimedia.org/T397392#10931243 (10kostajh) >>! In T397392#10931229, @kostajh wrote: > Hmm. So in theory mentioning {T363858} fixes the index? (per T157783#5744922) If so, that didn't seem to work: T363858 s... [09:10:28] !log deployment-prep: Update changeprop config per https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1150699 using [[wikitech:Changeprop#To_deployment-prep]] (T394958) [09:10:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:10:31] T394958: Shorten the NotificationGetStartedJob delay on pilot wikis as an experiment - https://phabricator.wikimedia.org/T394958 [09:18:53] !log deployment-prep: Update changeprop config perhttps://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1161443 using [[wikitech:Changeprop#To_deployment-prep]] (T394958; this time actually changing the beta config) [09:18:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:18:55] T394958: Shorten the NotificationGetStartedJob delay on pilot wikis as an experiment - https://phabricator.wikimedia.org/T394958 [09:38:50] !log deployment-prep: GrowthExperiments config migration `foreachwiki extensions/CommunityConfiguration/maintenance/migrateConfig.php GrowthSuggestedEdits` β€” T393771 [09:38:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:38:53] T393771: Send a notification to newcomers when Add a link edit threshold is reached - https://phabricator.wikimedia.org/T393771 [09:56:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:56:37] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://phabricator.wikimedia.org/T397424 (10wmcs-alerts) 03NEW [10:03:05] 10Continuous-Integration-Infrastructure (Zuul upgrade): Artifact storage for updated Zuul CI - https://phabricator.wikimedia.org/T397385#10931412 (10hashar) >>! In T397385#10930721, @MatthewVernon wrote: > I think the answer is "you need a proxy if you want to do it externally", but it's a little "It Depends". I... [10:11:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:12:49] (03CR) 10Hashar: Add xdebug back to Quibble PHP images (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [10:12:56] (03PS5) 10Hashar: Add xdebug back to Quibble PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [10:15:46] 10Continuous-Integration-Infrastructure (Zuul upgrade): Artifact storage for updated Zuul CI - https://phabricator.wikimedia.org/T397385#10931448 (10MatthewVernon) Yes, I'd expect the AccessDenied when trying to list the mcv21test bucket - only the 1-month-old_kittens_32.jpg object within it is globally-readable... [10:19:32] (03PS6) 10Hashar: Add xdebug back to Quibble PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [10:19:55] (03CR) 10Hashar: [C:03+2] "Amended to also update the `quibble-fresnel` image." [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [10:21:11] (03Merged) 10jenkins-bot: Add xdebug back to Quibble PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [10:33:45] 10Continuous-Integration-Infrastructure: Drop xdebug from all RelEng images - https://phabricator.wikimedia.org/T319495#10931504 (10hashar) 05Openβ†’03Resolved == Step 2: publishing == Successfully published image docker-registry.wikimedia.org/releng/quibble-bullseye-php82:1.14.1-s6 Successfully published... [10:39:11] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10931513 (10jijiki) [10:43:03] (03PS1) 10Hashar: jjb: update Quibble jobs to add Xdebug [integration/config] - 10https://gerrit.wikimedia.org/r/1161466 (https://phabricator.wikimedia.org/T319495) [10:50:31] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10931528 (10jijiki) [10:52:06] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10931532 (10jijiki) [10:53:01] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade): Reduce the number of CI images - https://phabricator.wikimedia.org/T397429 (10hashar) 03NEW [11:05:15] (03CR) 10Daimona Eaytoy: "Shouldn't we also disable xdebug, like we're doing with pcov? I imagine xdebug is also going to have a greater impact than pcov." [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [11:28:49] (03CR) 10Hashar: [C:04-1] "Xdebug is not disabled :/" [integration/config] - 10https://gerrit.wikimedia.org/r/1161466 (https://phabricator.wikimedia.org/T319495) (owner: 10Hashar) [11:29:26] (03CR) 10Hashar: [C:03+2] "Indeed we should have `phpdismod xdebug` it :/" [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [11:42:24] (03PS1) 10Hashar: dockerfiles: disable XDebug by default [integration/config] - 10https://gerrit.wikimedia.org/r/1161481 (https://phabricator.wikimedia.org/T319495) [11:42:55] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Drop xdebug from all RelEng images - https://phabricator.wikimedia.org/T319495#10931697 (10hashar) 05Resolvedβ†’03Open [11:43:34] (03PS2) 10Hashar: jjb: update Quibble jobs to add Xdebug [integration/config] - 10https://gerrit.wikimedia.org/r/1161466 (https://phabricator.wikimedia.org/T319495) [11:47:21] (03CR) 10Hashar: [C:03+2] dockerfiles: disable XDebug by default [integration/config] - 10https://gerrit.wikimedia.org/r/1161481 (https://phabricator.wikimedia.org/T319495) (owner: 10Hashar) [11:48:40] (03Merged) 10jenkins-bot: dockerfiles: disable XDebug by default [integration/config] - 10https://gerrit.wikimedia.org/r/1161481 (https://phabricator.wikimedia.org/T319495) (owner: 10Hashar) [11:48:44] 10WikimediaDebug: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439 (10Tgr) 03NEW [11:49:59] 10WikimediaDebug, 06MediaWiki-Platform-Team: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#10931740 (10Tgr) [11:57:15] (03CR) 10Hashar: [C:03+2] "Done in Ie2a25f65a1f5ace49e43961a02bf254eeb4ea6a2" [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [11:58:17] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Drop xdebug from all RelEng images - https://phabricator.wikimedia.org/T319495#10931796 (10hashar) == Step 2: publishing == Successfully published image docker-registry.wikimedia.org/releng/quibble-bullseye-php83:1.14.1-s7 Successfully published imag... [12:19:04] (03CR) 10Daimona Eaytoy: "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [13:56:27] 06Release-Engineering-Team, 10Scap (SpiderPig πŸ•ΈοΈ): SpiderPig session / login broken - https://phabricator.wikimedia.org/T397457 (10Lucas_Werkmeister_WMDE) 03NEW [15:16:44] 06Release-Engineering-Team, 10Scap (SpiderPig πŸ•ΈοΈ): SpiderPig session / login broken - https://phabricator.wikimedia.org/T397457#10932531 (10Lucas_Werkmeister_WMDE) > I have a screen recording of the broken SpiderPig behavior, but I’m not 100% sure if it’s safe to share publicly or not, so contact me out-of-ban... [16:53:04] 06Release-Engineering-Team, 07Epic, 05MW-1.43-release, 05MW-1.44-release, and 2 others: [REST Sandbox] Remove SwaggerUI from MediaWiki Releases - https://phabricator.wikimedia.org/T397470 (10HCoplin-WMF) 03NEW [17:01:25] 06Release-Engineering-Team, 06MW-Interfaces-Team, 05MW-1.43-release, 05MW-1.44-release, 05MW-1.45-release: Remove SwaggerUI from MW-1.43 LTR - https://phabricator.wikimedia.org/T397471 (10HCoplin-WMF) 03NEW [17:01:37] 06Release-Engineering-Team, 06MW-Interfaces-Team, 05MW-1.43-release: Remove SwaggerUI from MW-1.43 LTR - https://phabricator.wikimedia.org/T397471#10932795 (10HCoplin-WMF) [17:03:50] 06Release-Engineering-Team, 06MW-Interfaces-Team, 05MW-1.44-release: Remove SwaggerUI from MW-1.44 release - https://phabricator.wikimedia.org/T397472 (10HCoplin-WMF) 03NEW [17:05:27] 06Release-Engineering-Team, 05MW-1.45-release: Remove SwaggerUI from MW-1.45 release - https://phabricator.wikimedia.org/T397473 (10HCoplin-WMF) 03NEW [17:05:37] 06Release-Engineering-Team, 05MW-1.45-release: Remove SwaggerUI from MW-1.45 release - https://phabricator.wikimedia.org/T397473#10932822 (10HCoplin-WMF) [17:05:54] 06Release-Engineering-Team, 06MW-Interfaces-Team, 05MW-1.44-release: Remove SwaggerUI from MW-1.44 release - https://phabricator.wikimedia.org/T397472#10932823 (10HCoplin-WMF) [17:14:51] 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T392176#10932845 (10Aklapper) {T397262} (has patch for review) is by far the noisiest issue (about 7500 in the last 7 hours) but apart from that thing... [17:16:19] 06Release-Engineering-Team, 07Epic, 05MW-1.43-release, 05MW-1.44-release, and 2 others: [REST Sandbox] Remove SwaggerUI from MediaWiki Releases - https://phabricator.wikimedia.org/T397470#10932851 (10HCoplin-WMF) [18:54:02] hashar: just an fyi that I'm looking at finally removing the last uses of polymer and to remove polymer alltogether from gerrit. I see some plugins you've build for the wmf use polymer. You'll need to migrate those to lit (as far as I'm aware you can't use lit in the uncompiled js file, you'll need to compile it). I've done https://gerrit-review.googlesource.com/c/plugins/zuul-results-summary/+/485281 for the zuul-results-summary [18:54:02] plugin. [18:54:35] I'm making progress on migrate elements away from paper-* and iron-*. [19:43:25] 10Continuous-Integration-Config, 07PHP 8.1 support: Make Phan on PHP 8.1 voting for wmf-deployed extensions/skins - https://phabricator.wikimedia.org/T353276#10933096 (10Umherirrender) 05Openβ†’03Resolved Already done via 97cbdc0ed3d5a54ab12578ef300744b6dab33892 [19:48:44] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06DBA: Prepare a database test for m3 - https://phabricator.wikimedia.org/T390034#10933100 (10brennen) 05Openβ†’03Resolved US holiday today, but just noting that test migration on phab1005 completed in about an hour yesterday, I thi... [20:58:22] 06Release-Engineering-Team, 06MW-Interfaces-Team, 05MW-1.43-release: Remove SwaggerUI from MW-1.43 LTR - https://phabricator.wikimedia.org/T397471#10933187 (10HCoplin-WMF) [20:59:16] 06Release-Engineering-Team, 06MW-Interfaces-Team, 05MW-1.43-release: Remove SwaggerUI from MW-1.43 LTR - https://phabricator.wikimedia.org/T397471#10933188 (10HCoplin-WMF) [20:59:33] 06Release-Engineering-Team, 06MW-Interfaces-Team, 05MW-1.44-release: Remove SwaggerUI from MW-1.44 release - https://phabricator.wikimedia.org/T397472#10933191 (10HCoplin-WMF) [22:16:02] 10Phabricator: Typos in Herald message for good first tasks tag - https://phabricator.wikimedia.org/T397487 (10TBurmeister) 03NEW [22:59:49] 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T392176#10933314 (10Aklapper) 05Openβ†’03Resolved