[00:01:05] 06Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 10ServiceOps-Mediawiki: Inform SREs and deployers of upcoming deployment and traffic routing to pretrain - https://phabricator.wikimedia.org/T436189#12267568 (10bd808) [00:20:32] 06Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 10ServiceOps-Mediawiki: Inform SREs and deployers of upcoming deployment and traffic routing to pretrain - https://phabricator.wikimedia.org/T436189#12267576 (10bd808) p:05Triage→03High [00:56:14] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12267623 (10bd808) 05Open→03In progress p:... [00:58:04] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [00:58:31] 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T436415 (10wmcs-alerts) 03NEW [01:00:49] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12267635 (10bd808) The tofu automation is curre... [04:15:29] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12267803 (10bd808) In a `make shell` session: `... [04:43:39] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12267828 (10bd808) To get things back on track... [06:56:49] 06Project-Admins, 10MediaWiki-User-Interface: Update #MediaWiki-User-Interface default filter - https://phabricator.wikimedia.org/T436411#12267920 (10Aklapper) 05Open→03Invalid "nonsense" is not an argument [08:42:16] Hi there, Lucas Werkmeister told me to bring this ticket to this channel's attention https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/1305800 [08:43:01] I thought the login screen of gerrit was so ugly that I re-designed it [08:43:39] https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/1306065/1 I also wrote a patch that gives its wallpaper a daily update [09:31:40] Hope somebody picks this up [10:23:16] (03CR) 10Aklapper: "Thanks for the patch. Is there a description of a problem somewhere, e.g. in a task? Apart from inserting custom background image attribut" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (owner: 101Veertje) [12:17:59] 10Phabricator: Add "thank you" to award token - https://phabricator.wikimedia.org/T434584#12268061 (10Aklapper) See T338133#10062974 for hints if someone would like to contribute a patch. [12:18:07] 10Phabricator: Add "thank you" to award token - https://phabricator.wikimedia.org/T434584#12268064 (10Aklapper) →14Duplicate dup:03T338133 [12:18:08] 10Phabricator: Add a "Thanks" token - https://phabricator.wikimedia.org/T338133#12268066 (10Aklapper) [12:18:10] 10Phabricator (Upstream), 07Upstream: Phabricator TOTP token should be evaluated against the currently valid one instead of expiring - https://phabricator.wikimedia.org/T434520#12268068 (10Aklapper) 05Stalled→03Invalid Closing for now until someone can reproduce and provide steps to reproduce, as I can... [12:19:21] 10Phabricator (Upstream), 07Upstream: Enhance tokens by adding rainbow-related ones - https://phabricator.wikimedia.org/T337986#12268070 (10Aklapper) 05Open→03Declined Cleaning the backlog a bit as I do not plan to work on this but please do reopen once someone contributed a patch! For previous exampl... [12:22:11] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work: Update to Phorge upstream 2025.18 release - https://phabricator.wikimedia.org/T386558#12268074 (10Aklapper) [12:22:14] 10Phabricator (Upstream), 07Accessibility, 07Upstream: When editing a task description in "full screen", the cloud upload button is hidden behind the dialog - https://phabricator.wikimedia.org/T88998#12268073 (10Aklapper) [12:22:42] 10Phabricator (Upstream), 07Accessibility, 07Upstream: When editing a task description in "full screen", the cloud upload button is hidden behind the dialog - https://phabricator.wikimedia.org/T88998#12268077 (10Aklapper) 05Stalled→03Resolved [12:25:41] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream 2026.xx - https://phabricator.wikimedia.org/T428547#12268079 (10Aklapper) [12:25:41] 10Diffusion, 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Title of commit message cut off in Diffusion after a full stop - https://phabricator.wikimedia.org/T130557#12268078 (10Aklapper) [12:26:28] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream 2026.xx - https://phabricator.wikimedia.org/T428547#12268081 (10Aklapper) [12:27:06] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream 2026.xx - https://phabricator.wikimedia.org/T428547#12268083 (10Aklapper) [12:27:07] 10Diffusion, 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Attempting to view a non-existent file through Diffusion causes an unhandled CommandException - https://phabricator.wikimedia.org/T435627#12268082 (10Aklapper) [12:30:10] (03update) 10aklapper: Hide Source Branch column in listed GitLab patches [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/61 (https://phabricator.wikimedia.org/T420059) [12:31:53] 10Phabricator: Allow Security Team members to see MFA status of a Phab user account - https://phabricator.wikimedia.org/T401106#12268086 (10Aklapper) 05Stalled→03Declined I believe that making folks an admin if requested is more sensible than maintaining additional custom downstream code for this [12:35:13] 10Phabricator, 07Epic: Virtual Offsite: Phabricator hacking session (May 2020) - https://phabricator.wikimedia.org/T253090#12268090 (10Aklapper) [12:35:54] 10Deployments, 10Gerrit, 10Phabricator, 06Release-Engineering-Team (Seen), and 2 others: Indicate deployed status (wmf branch) for Gerrit patches in Phabricator - https://phabricator.wikimedia.org/T88136#12268091 (10Aklapper) 05Open→03Declined I prefer not to pull this into Phabricator (and maintai... [13:51:05] 10Phabricator, 06cloud-services-team, 10Striker, 06tools-platform-team: Ensure that auto-created Phabricator projects for Toolforge tools are always editable by that tool's maintainers - https://phabricator.wikimedia.org/T412607#12268124 (10Aklapper) [14:24:46] (03open) 10phedenskog: Measure how long builds are blocked on the castor save [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/47 (https://phabricator.wikimedia.org/T436427) [14:55:35] 10MediaWiki-Releasing, 13Patch-For-Review, 07Security: Update HISTORY in master after 1.43.9/1.44.6/1.45.4 - https://phabricator.wikimedia.org/T421272#12268165 (10Zabe) a:03Zabe [14:56:37] 10MediaWiki-Releasing, 07Security: Obtain CVEs for 1.43.9/1.44.6/1.45.4 security releases - https://phabricator.wikimedia.org/T421265#12268168 (10Zabe) 05Open→03Resolved [15:12:05] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:56:26] (03merge) 10phedenskog: Measure how long builds are blocked on the castor save [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/47 (https://phabricator.wikimedia.org/T436427) [17:56:57] (03close) 10phedenskog: Show min and max in the core per-job stats tables [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/46 [18:29:16] (03open) 10phedenskog: Show castor queue wait and blocked time on the castor dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/48 (https://phabricator.wikimedia.org/T436427) [18:29:21] (03merge) 10phedenskog: Show castor queue wait and blocked time on the castor dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/48 (https://phabricator.wikimedia.org/T436427) [19:03:16] 10Gerrit, 07ci-test-error: quibble-vendor-mysql-php83 failing due to npm caching issues - https://phabricator.wikimedia.org/T436437 (10Monkeysmashingkeyboards) 03NEW [19:03:18] (03open) 10phedenskog: Tune the castor wait dashboards [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/49 [19:03:24] (03merge) 10phedenskog: Tune the castor wait dashboards [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/49 [19:18:13] 10Phabricator (Upstream), 07Upstream: Use pholio camera icon for "added a mock" events in task history - https://phabricator.wikimedia.org/T100538#12268276 (10Aklapper) 05Open→03Declined Closing per T100538#2976246 [19:21:31] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Castor, 07ci-test-error: quibble-vendor-mysql-php83 failing due to npm caching issues - https://phabricator.wikimedia.org/T436437#12268282 (10Monkeysmashingkeyboards) [19:23:12] 10Phabricator, 06Project-Admins, 06ESEAP-Hub: Create project tag and private Space for ESEAP Hub - https://phabricator.wikimedia.org/T403908#12268286 (10Aklapper) 05Open→03Declined No reply (and not much activity on https://phabricator.wikimedia.org/project/profile/8165/), thus I'm going to decline t... [19:26:15] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Castor, 07ci-test-error: quibble-vendor-mysql-php83 failing due to npm caching issues - https://phabricator.wikimedia.org/T436437#12268290 (10Monkeysmashingkeyboards) https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/ [19:39:06] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Castor, 07ci-test-error: quibble-vendor-mysql-php83 failing due to npm caching issues - https://phabricator.wikimedia.org/T436437#12268296 (10Monkeysmashingkeyboards) p:05Triage→03High Has been blocking merges for pretty much every extension on Gerr... [19:45:22] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Castor, 07ci-test-error: quibble-vendor-mysql-php83 failing due to npm caching issues - https://phabricator.wikimedia.org/T436437#12268303 (10SomeRandomDeveloper) →14Duplicate dup:03T436436 [19:49:13] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to "tarball data for ... seems to be corrupted." - https://phabricator.wikimedia.org/T436436#12268310 (10Monkeysmashingkeyboards) Has only been happening with the `quibble-vendor-... [19:49:35] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to "tarball data for ... seems to be corrupted." - https://phabricator.wikimedia.org/T436436#12268316 (10Servis.006) →14Duplicate dup:03T376276 [19:50:30] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to "tarball data for ... seems to be corrupted." - https://phabricator.wikimedia.org/T436436#12268317 (10SomeRandomDeveloper) 05Duplicate→03Open [19:50:58] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to "tarball data for ... seems to be corrupted." - https://phabricator.wikimedia.org/T436436#12268321 (10Monkeysmashingkeyboards) Why was this closed as a duplicate? [19:56:58] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to cache issues with quibble-vendor-mysql-php83 - https://phabricator.wikimedia.org/T436436#12268327 (10Monkeysmashingkeyboards) [20:18:33] 10Continuous-Integration-Infrastructure (Zuul upgrade): Update Magnum cluster config to use higher iops volume type and flavor - https://phabricator.wikimedia.org/T410290#12268332 (10bd808) @dduvall Did this functionally happen when you moved things to the CAPI backend in {T424061}? [20:20:37] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#12268338 (10bd808) 05In progress→03Resolved Cluster has existed for quite a while at this point. @dduvall did a lot... [21:51:08] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to cache issues with quibble-vendor-mysql-php83 - https://phabricator.wikimedia.org/T436436#12268376 (10Monkeysmashingkeyboards) [21:51:50] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor: npm cache saved by castor get corrupted for unknown reason - https://phabricator.wikimedia.org/T295351#12268377 (10Monkeysmashingkeyboards) [22:06:33] 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T436415#12268383 (10bd808) 05Open→03Resolved a:03bd808 `lang=shell-session bd808@deployment-puppetserver-1.deployment-prep.eqiad1:~$ sudo /usr/lo... [22:13:04] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [22:16:28] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to cache issues with quibble-vendor-mysql-php83 - https://phabricator.wikimedia.org/T436436#12268393 (10Ladsgroup) FWIW, it's not just one agent. All of them seem to have a corrup... [22:21:02] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to cache issues with quibble-vendor-mysql-php83 - https://phabricator.wikimedia.org/T436436#12268396 (10Ladsgroup) because that's being pulled from `integration-castor06`. I'm in... [22:24:42] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to cache issues with quibble-vendor-mysql-php83 - https://phabricator.wikimedia.org/T436436#12268397 (10Ladsgroup) I did this: ` ladsgroup@integration-castor06:/srv/castor/castor-... [22:31:48] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to cache issues with quibble-vendor-mysql-php83 - https://phabricator.wikimedia.org/T436436#12268398 (10siebrand) Success at https://integration.wikimedia.org/ci/job/quibble-v... [22:31:51] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to cache issues with quibble-vendor-mysql-php83 - https://phabricator.wikimedia.org/T436436#12268399 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup It's back to working.... [22:41:04] 10Continuous-Integration-Config, 10MediaWiki-extensions-Translate, 10TranslationNotifications: Add i18n related extensions to the CI gate - https://phabricator.wikimedia.org/T86930#12268404 (10siebrand) [22:51:25] (03PS1) 10Siebrand: gitignore: ignore JetBrains *.iml files [integration/config] - 10https://gerrit.wikimedia.org/r/1332246 [22:54:22] (03PS1) 10Siebrand: Add i18n related extension TranslationNotifications to the CI gate [integration/config] - 10https://gerrit.wikimedia.org/r/1332247 (https://phabricator.wikimedia.org/T86930) [22:56:00] (03CR) 10CI reject: [V:04-1] Add i18n related extension TranslationNotifications to the CI gate [integration/config] - 10https://gerrit.wikimedia.org/r/1332247 (https://phabricator.wikimedia.org/T86930) (owner: 10Siebrand) [23:00:34] (03PS2) 10Siebrand: Add i18n related extension TranslationNotifications to the CI gate [integration/config] - 10https://gerrit.wikimedia.org/r/1332247 (https://phabricator.wikimedia.org/T86930) [23:08:31] 06Release-Engineering-Team (Priority Backlog 📥): Talk with Security/Product Safety and Integrity foks about Pretrain workflow/timing implications - https://phabricator.wikimedia.org/T424107#12268415 (10bd808) I have added this as a dependency of {T436190}. The initial Pretrain automated deployments will run at 0... [23:22:11] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI in many extensions broken due to cache issues with quibble-vendor-mysql-php83 - https://phabricator.wikimedia.org/T436436#12268419 (10SomeRandomDeveloper) Thanks!