[00:02:48] <wikibugs>	 (03PS1) 10MusikAnimal: opensource: add PersonalDashboard docs to MW components [integration/docroot] - 10https://gerrit.wikimedia.org/r/1327223 (https://phabricator.wikimedia.org/T435392)
[00:13:03] <wmcs-alerts>	 FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:23:03] <wmcs-alerts>	 RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:23:32] <wikibugs>	 06Release-Engineering-Team (Doing 😎), 10SyntaxHighlight, 13Patch-For-Review: New 2.21.0 upstream release for Pygments - https://phabricator.wikimedia.org/T421653#12234956 (10bd808) Added new patch reviewers to see if I can get this moving.
[00:59:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-File-management, 10Wikimedia-SVG-rendering: Test resvg on Beta Cluster - https://phabricator.wikimedia.org/T243893#12235070 (10AntiCompositeNumber) 05Open→03Declined Decline per T40010
[05:07:03] <wmcs-alerts>	 FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[05:17:03] <wmcs-alerts>	 RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[06:08:05] <icinga-wm>	 PROBLEM - PHD should be running on phab1004 is CRITICAL: PROCS CRITICAL: 0 processes with regex args php ./phd-daemon, UID = 920 (phd) https://wikitech.wikimedia.org/wiki/Phabricator
[06:12:37] <andre>	 ^ Phabricator has a planned maintenance now, see https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/GHNYRAXG3WVZSAEB2FHWOT57K5356KES/
[06:17:15] <icinga-wm>	 PROBLEM - PHD should be running on phab1005 is CRITICAL: PROCS CRITICAL: 0 processes with regex args php ./phd-daemon, UID = 920 (phd) https://wikitech.wikimedia.org/wiki/Phabricator
[06:23:15] <icinga-wm>	 RECOVERY - PHD should be running on phab1005 is OK: PROCS OK: 1 process with regex args php ./phd-daemon, UID = 920 (phd) https://wikitech.wikimedia.org/wiki/Phabricator
[06:26:58] <wikibugs>	 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12235231 (10Arnoldokoth) Post!
[06:32:02] <wikibugs>	 06Release-Engineering-Team, 06Collaboration-Services: Deploying Phab/Phorge in devtools to new Trixie instance does not restart PHP 8.4 - https://phabricator.wikimedia.org/T433988#12235245 (10Aklapper) 05In progress→03Resolved Resolving as this got merged before T435087.
[06:33:03] <wikibugs>	 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12235252 (10Arnoldokoth)
[06:38:12] <wikibugs>	 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12235259 (10Aklapper) Lessons learned: Run a scap deploy on old phab1004 to apply the database settings in /etc/ppabricator/config.yaml before deplo...
[06:39:46] <wikibugs>	 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12235263 (10ABran-WMF)
[06:52:05] <wikibugs>	 (03PS1) 10Phedenskog: jjb: Only trigger castor-save for pipelines that save [integration/config] - 10https://gerrit.wikimedia.org/r/1327435 (https://phabricator.wikimedia.org/T434749)
[06:55:34] <wikibugs>	 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - from phab1004 (Bullseye, PHP 7.4) to phab1005 (Trixie, PHP 8.4) - https://phabricator.wikimedia.org/T435087#12235280 (10Aklapper)
[07:35:03] <wmcs-alerts>	 FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[07:49:03] <wikibugs>	 (03PS1) 10Hashar: jjb: Only trigger castor-save for pipelines that save [integration/config] - 10https://gerrit.wikimedia.org/r/1327435 (https://phabricator.wikimedia.org/T434749) (owner: 10Phedenskog)
[07:49:13] <wikibugs>	 10Phabricator Translations Extension, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S2 2026 August): Phabricator translations needs i18n check reviewed again - https://phabricator.wikimedia.org/T434026#12235359 (10Nikerabbit)
[07:50:03] <wmcs-alerts>	 RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[07:50:35] <wikibugs>	 (03CR) 10Hashar: [C:04-1] jjb: Only trigger castor-save for pipelines that save (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1327435 (https://phabricator.wikimedia.org/T434749) (owner: 10Phedenskog)
[07:53:39] <wikibugs>	 (03PS2) 10Phedenskog: jjb: Only trigger castor-save for pipelines that save [integration/config] - 10https://gerrit.wikimedia.org/r/1327435 (https://phabricator.wikimedia.org/T434749)
[07:53:54] <wikibugs>	 (03CR) 10Phedenskog: jjb: Only trigger castor-save for pipelines that save (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1327435 (https://phabricator.wikimedia.org/T434749) (owner: 10Phedenskog)
[07:55:21] <wikibugs>	 (03CR) 10Hashar: [C:03+2] jjb: Only trigger castor-save for pipelines that save [integration/config] - 10https://gerrit.wikimedia.org/r/1327435 (https://phabricator.wikimedia.org/T434749) (owner: 10Phedenskog)
[07:57:31] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Only trigger castor-save for pipelines that save [integration/config] - 10https://gerrit.wikimedia.org/r/1327435 (https://phabricator.wikimedia.org/T434749) (owner: 10Phedenskog)
[08:07:03] <wmcs-alerts>	 FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[08:16:26] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 06Test Platform, and 2 others: Offload queue wait time from the feedback loop time - https://phabricator.wikimedia.org/T434749#12235449 (10hashar) Peter and I have installed the two plugins `conditional-buildstep` and `run-condition`.  Tested th...
[08:27:03] <wmcs-alerts>	 RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[08:35:53] <wikibugs>	 (03open) 10phedenskog: Add daily castor run count to the castor dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/29 (https://phabricator.wikimedia.org/T434749)
[08:43:49] <wikibugs>	 10Phabricator: (DateInvalidTimeZoneException) DateTimeZone::__construct(): Unknown or bad timezone (Europe/Kiev) - https://phabricator.wikimedia.org/T435441 (10Aklapper) 03NEW
[08:44:03] <wmcs-alerts>	 FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[08:51:49] <wikibugs>	 10Phabricator, 06Collaboration-Services: phstats user lacks database permissions on phab1005 - https://phabricator.wikimedia.org/T435442 (10Aklapper) 03NEW
[08:52:42] <hashar>	 [2931262.724895] systemd-journald[1840774]: Failed to create new system journal: No space left on device
[08:52:42] <hashar>	 [2936204.283831] Out of memory: Killed process 2436986 (cache-main) total-vm:9575200kB, anon-rss:4334468kB, file-rss:0kB, shmem-rss:165808kB, UID:115 pgtables:16656kB oom_score_adj:0
[08:52:47] <hashar>	 poor deployment-cache-text08
[08:52:59] <wikibugs>	 10Phabricator, 06Collaboration-Services, 13Patch-For-Review: phstats user lacks database permissions on phab1005 - https://phabricator.wikimedia.org/T435442#12235653 (10Aklapper)
[08:54:03] <wmcs-alerts>	 RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[08:54:06] <hashar>	 !log deployment-prep: hard reboot deployment-cache-test-08 # T435421
[08:54:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:54:09] <stashbot>	 T435421: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435421
[08:58:16] <wikibugs>	 (03update) 10arnaudb: Add a dedicated debug-ci project runner for the CDN repro (T425441) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/626 (https://phabricator.wikimedia.org/T425441)
[08:59:47] <wikibugs>	 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435421#12235681 (10hashar) 05Open→03Resolved a:03hashar ` [2931262.724895] systemd-journald[1840774]: Failed to create new system journal: No space left on device [293620...
[09:05:46] <wikibugs>	 (03update) 10arnaudb: Draft: Add a dedicated debug-ci project runner for the CDN repro (T425441) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/626 (https://phabricator.wikimedia.org/T425441)
[09:10:14] <wikibugs>	 10Phabricator, 06Release-Engineering-Team (Doing 😎), 07Regression: (DateInvalidTimeZoneException) DateTimeZone::__construct(): Unknown or bad timezone (Europe/Kiev) - https://phabricator.wikimedia.org/T435441#12235749 (10Aklapper) p:05Triage→03Low a:03Aklapper User impact is low  - Phorge is intelligen...
[09:12:39] <wikibugs>	 10Phabricator, 06Collaboration-Services, 13Patch-For-Review: phstats user lacks database permissions on phab1005 - https://phabricator.wikimedia.org/T435442#12235764 (10ABran-WMF)
[09:20:39] <wikibugs>	 (03CR) 10Hashar: "Yesterday I changed the regex used by the collapsible section plugin, they expected `INFO:quibble.commands`, I have made that optional for" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1319382 (https://phabricator.wikimedia.org/T432966) (owner: 10Phedenskog)
[09:24:03] <wikibugs>	 (03CR) 10Hashar: "I have enabled the *Collapse only one level per ending string* option.  Looks like it solves that specific issue, but maybe it causes some" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1319382 (https://phabricator.wikimedia.org/T432966) (owner: 10Phedenskog)
[09:25:03] <wmcs-alerts>	 FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:25:13] <wikibugs>	 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435448 (10wmcs-alerts) 03NEW
[09:40:03] <wmcs-alerts>	 RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:42:07] <wikibugs>	 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435421#12235913 (10hashar) 05Resolved→03Open And it went down again almost immediately T435448
[09:42:22] <wikibugs>	 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435421#12235920 (10hashar)
[09:42:23] <wikibugs>	 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435448#12235922 (10hashar) →14Duplicate dup:03T435421
[09:46:29] <hashar>	 tldr: deployment-prep is being crawled to death
[10:09:03] <wmcs-alerts>	 FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:14:03] <wmcs-alerts>	 RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:01:36] <wikibugs>	 (03PS1) 10Phedenskog: castor: report a failed cache load as a Failed section [integration/config] - 10https://gerrit.wikimedia.org/r/1327512 (https://phabricator.wikimedia.org/T432685)
[11:04:20] <wikibugs>	 (03CR) 10Phedenskog: "This was a miss from me: I created the patch first without the Failed section (it wasn't implemented in Quibble). Then Peter W did another" [integration/config] - 10https://gerrit.wikimedia.org/r/1327512 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog)
[11:05:41] <wikibugs>	 (03merge) 10phedenskog: Add daily castor run count to the castor dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/29 (https://phabricator.wikimedia.org/T434749)
[11:08:15] <wikibugs>	 10Deployments, 06Release-Engineering-Team (Doing 😎): OpenSearch mediawiki new errors filters out some errors - https://phabricator.wikimedia.org/T431675#12236215 (10Aklapper) a:05Aklapper→03None
[11:08:28] <wikibugs>	 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T430835#12236216 (10Aklapper) 05Open→03Resolved [This train is definitely arriving!](https://www.youtube.com/watch?v=00XevbzQoI8)
[11:09:09] <wikibugs>	 (03update) 10phedenskog: Show castor restore times next to the save times [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/28 (https://phabricator.wikimedia.org/T432688)
[11:09:17] <wikibugs>	 (03merge) 10phedenskog: Show castor restore times next to the save times [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/28 (https://phabricator.wikimedia.org/T432688)
[11:14:03] <wmcs-alerts>	 FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:29:03] <wmcs-alerts>	 RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:38:52] <wikibugs>	 10Phabricator, 06Release-Engineering-Team (Doing 😎): PHP 8.1 "Implicit conversion from float 36.5 to int loses precision" in PhabricatorSearchResultView - https://phabricator.wikimedia.org/T435466 (10Aklapper) 03NEW p:05Triage→03Medium
[11:42:59] <wikibugs>	 10Phabricator, 06Release-Engineering-Team (Doing 😎): PHP 8.1 "Implicit conversion from float 36.5 to int loses precision" in PhabricatorSearchResultView - https://phabricator.wikimedia.org/T435466#12236411 (10Aklapper) See also {T404922}
[11:49:33] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Spike, 06Test Platform (Nairobi 29): Offload queue wait time from the feedback loop time - https://phabricator.wikimedia.org/T434749#12236426 (10Peter)
[11:50:38] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Spike, 06Test Platform (Nairobi 29): Offload queue wait time from the feedback loop time - https://phabricator.wikimedia.org/T434749#12236430 (10Peter) 05Open→03Resolved a:03Peter
[11:51:16] <wikibugs>	 (03PS2) 10Hashar: castor: report a failed cache load as a Failed section [integration/config] - 10https://gerrit.wikimedia.org/r/1327512 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog)
[11:52:18] <wikibugs>	 (03CR) 10Hashar: "I have:" [integration/config] - 10https://gerrit.wikimedia.org/r/1327512 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog)
[11:56:16] <wikibugs>	 (03open) 10aklapper: PHP 8.1 compat: Floor half length of highlighted search results [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/128 (https://phabricator.wikimedia.org/T435466)
[11:56:47] <wikibugs>	 (03merge) 10aklapper: PHP 8.1 compat: Floor half length of highlighted search results [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/128 (https://phabricator.wikimedia.org/T435466)
[11:57:28] <wikibugs>	 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): PHP 8.1 "Implicit conversion from float 36.5 to int loses precision" in PhabricatorSearchResultView - https://phabricator.wikimedia.org/T435466#12236473 (10Aklapper)
[12:05:03] <wmcs-alerts>	 FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:10:03] <wmcs-alerts>	 RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:29:35] <wikibugs>	 (03open) 10phedenskog: Make the castor Restore section mirror the Save section [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/30
[12:29:43] <wikibugs>	 (03merge) 10phedenskog: Make the castor Restore section mirror the Save section [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/30
[13:30:12] <wikibugs>	 (03CR) 10Hashar: "@jforrester@wikimedia.org feel free to roll that one. I am not familiar with the issue(s) that might have caused that job to be commented " [integration/config] - 10https://gerrit.wikimedia.org/r/1326397 (owner: 10Jforrester)
[13:33:56] <James_F>	 Ha, oops, thanks for reminding me hashar.
[13:33:59] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikiLambda] Re-enable Catalyst [integration/config] - 10https://gerrit.wikimedia.org/r/1326397 (owner: 10Jforrester)
[13:34:29] <James_F>	 hashar: https://gerrit.wikimedia.org/r/c/integration/config/+/1327120 would appreciate a review. :-)
[13:35:04] <hashar>	 James_F: that one is somewere at the tail of my long review queue :b
[13:35:11] <James_F>	 Psssssh.
[13:35:12] <James_F>	 Fair.
[13:35:31] <hashar>	 I think we had a similar issue with scap when upgraded to Bookworm
[13:35:43] <hashar>	 and using a venv would get rid of --break-system-package
[13:35:44] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Re-enable Catalyst [integration/config] - 10https://gerrit.wikimedia.org/r/1326397 (owner: 10Jforrester)
[13:35:44] <James_F>	 Yeah, this should avoid that in future.
[13:35:52] <hashar>	 but I gotta put a bit more thoughts in verifying it :b
[13:35:53] <James_F>	 !log Zuul: [mediawiki/extensions/WikiLambda] Re-enable Catalyst
[13:35:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:35:58] <hashar>	 at least we have a patch now!
[13:42:38] <James_F>	 It builds, it runs, and it works for Quibble’s self-build, FWIW.
[14:25:11] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T435500 (10Jatwa) 03NEW
[14:26:41] <wikibugs>	 (03CR) 10Phedenskog: [C:03+1] "Great thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/1327512 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog)
[14:28:11] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T435500#12237391 (10Pharos) I vouch for this user.
[14:34:12] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503 (10hashar) 03NEW
[14:36:47] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T435500#12237455 (10Aklapper) Please fill in data. The ticket is completely empty currently. Thanks.
[14:37:00] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Jatwa - https://phabricator.wikimedia.org/T435500#12237456 (10Jatwa)
[14:37:57] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Jatwa - https://phabricator.wikimedia.org/T435500#12237461 (10Jatwa) a:03Jatwa Added it in @Aklapper
[14:46:00] <wikibugs>	 (03CR) 10Hashar: [C:03+2] castor: report a failed cache load as a Failed section [integration/config] - 10https://gerrit.wikimedia.org/r/1327512 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog)
[14:48:07] <wikibugs>	 (03PS1) 10Jforrester: Zuul: Temporarily make wikilambda-catalyst-end-to-end non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/1327558
[14:48:27] <wikibugs>	 (03CR) 10Phedenskog: "Looks good, the important thing for me is that we get the data into the database. Then we can make list of slowest browser tests etc." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1319382 (https://phabricator.wikimedia.org/T432966) (owner: 10Phedenskog)
[14:48:43] <wikibugs>	 (03Merged) 10jenkins-bot: castor: report a failed cache load as a Failed section [integration/config] - 10https://gerrit.wikimedia.org/r/1327512 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog)
[14:51:48] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "Done" [integration/config] - 10https://gerrit.wikimedia.org/r/1327512 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog)
[14:51:57] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "Done" [integration/config] - 10https://gerrit.wikimedia.org/r/1327512 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog)
[14:52:02] <wikibugs>	 (03PS1) 10Hashar: castor: update jobs to releng/castor:0.4.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1327559 (https://phabricator.wikimedia.org/T432685)
[14:52:52] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "Jobs are being updated!" [integration/config] - 10https://gerrit.wikimedia.org/r/1327559 (https://phabricator.wikimedia.org/T432685) (owner: 10Hashar)
[14:54:39] <wikibugs>	 (03Merged) 10jenkins-bot: castor: update jobs to releng/castor:0.4.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1327559 (https://phabricator.wikimedia.org/T432685) (owner: 10Hashar)
[14:55:05] <James_F>	 !log jforrester@integration-castor06:$ sudo rm -rf /srv/castor/castor-mw-ext-and-skins/master/wikilambda-catalyst-end-to-end # Clear stale Catalyst castor npm downloads.
[14:55:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:13:18] <paladox>	 https://usercontent.irccloud-cdn.com/file/EAtic1Vr/Screenshot%202026-08-20%20at%2016.11.46.png
[15:13:20] <paladox>	 Adding better line ending support to codemirror. Shows the line ending of the document, can paste \r\n content and it'll be preserved, converts mixed line endings to \r\n if you open the file (with an alert).
[15:15:23] <James_F>	 paladox: Nice.
[15:16:56] <paladox>	 CM doesn't support dual line endings hence the "convert it". It'll do it on the initial load of the document, you can discard the change when you close it (if you don't want it saved). 
[15:19:42] <paladox>	 FYI Gerrit is getting taken over by a new team in San Fransisco, with the team in Germany transitioning.
[15:20:21] <paladox>	 One old face is on the new team but otherwise all newbies in taking over gerrit maintenance.
[15:22:53] <brennen>	 well that's good to know
[15:26:35] <James_F>	 I’m sure we can rely on newbies and LLMs to maintain gerrit forever.
[15:26:39] <wikibugs>	 (03open) 10phedenskog: Add metadata about Castor failures [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/31
[15:27:12] <wikibugs>	 (03approved) 10dancy: Draft: Add a dedicated debug-ci project runner for the CDN repro (T425441) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/626 (https://phabricator.wikimedia.org/T425441) (owner: 10arnaudb)
[15:35:41] <andre>	 brennen: [FYI, nothing to do here, just feeling like oversharing]: Morn, Phab post-mortem. https://phabricator.wikimedia.org/config/issue/deprecations/ turns out to be quite useful bc Phab Logstash is noisy for a while
[15:35:42] <andre>	 I've started handling all known issues in some way, https://phabricator.wikimedia.org/maniphest/?ids=435466,435441,435442#R for downstream ones, https://we.phorge.it/differential/?ids=27210,27209,27208,27207,27206#R for upstream ones
[15:35:50] <andre>	 That's all to share. :)
[15:42:19] <paladox>	 it's newbies in terms of contributing to gerrit. They've been with google for a while/have delt with git.
[15:43:17] <wikibugs>	 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - from phab1004 (Bullseye, PHP 7.4) to phab1005 (Trixie, PHP 8.4) - https://phabricator.wikimedia.org/T435087#12237848 (10Arnoldokoth)
[15:43:56] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Jatwa - https://phabricator.wikimedia.org/T435500#12237866 (10Pppery) a:05Jatwa→03None
[15:44:26] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12237867 (10Lars) @BLiviero-WMF or @hashar Who would be able to...
[15:45:13] <wikibugs>	 (03update) 10pppery: Use PhutilNumber for $weeks [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/127
[15:46:01] <wikibugs>	 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - from phab1004 (Bullseye, PHP 7.4) to phab1005 (Trixie, PHP 8.4) - https://phabricator.wikimedia.org/T435087#12237877 (10Arnoldokoth) I'll resolve this... In case of anything, we can use the...
[15:46:19] <wikibugs>	 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - from phab1004 (Bullseye, PHP 7.4) to phab1005 (Trixie, PHP 8.4) - https://phabricator.wikimedia.org/T435087#12237879 (10Arnoldokoth) 05Open→03Resolved
[15:50:26] <wikibugs>	 (03merge) 10aklapper: Use PhutilNumber for $weeks [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/127 (owner: 10pppery)
[15:53:06] <wikibugs>	 10Phabricator, 06Release-Engineering-Team (Doing 😎), 06ServiceOps, 07Essential-Work: Rule to tag new bugs with `Serviceops new` instead of `serviceops` tag - https://phabricator.wikimedia.org/T414777#12237925 (10MLechvien-WMF) Thanks Andre!
[15:59:05] <wikibugs>	 10Phabricator, 06Collaboration-Services, 13Patch-For-Review: phstats user lacks database permissions on phab1005 - https://phabricator.wikimedia.org/T435442#12237984 (10Arnoldokoth) Merged the fix. Thanks @Aklapper
[16:00:50] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12237995 (10bd808) >>! In T434024#12214917, @BLiviero-WMF wrote:...
[16:06:39] <wikibugs>	 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12238031 (10pmiazga) Maybe it's time to put a honeypot  - add a hidden link on Beta Cluster output (somwhere on top, maybe banner ? ) - link should be h...
[16:14:19] <wikibugs>	 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12238067 (10bd808) >>! In T420833#12238031, @pmiazga wrote: > Maybe it's time to put a honeypot  - add a hidden link on Beta Cluster output (somwhere on...
[16:39:28] <wikibugs>	 (03merge) 10phedenskog: Add metadata about Castor failures [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/31
[16:40:43] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Jatwa - https://phabricator.wikimedia.org/T435500#12238209 (10Urbanecm) Hi @Jatwa, thanks for the request. Please connect your Phabricator account with your developer account. You can do so under Settings -> Exter...
[17:04:49] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12238362 (10Andrew) It looks to me like quite a lot of our cloud...
[17:18:41] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Jatwa - https://phabricator.wikimedia.org/T435500#12238436 (10Jatwa) done!
[17:23:12] <wikibugs>	 10Phabricator: False-positive for "File not attached" - https://phabricator.wikimedia.org/T435532 (10Quiddity) 03NEW
[17:27:36] <wikibugs>	 (03open) 10phedenskog: Show which branches failed restores happen on for Castor [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/32
[17:27:48] <wikibugs>	 (03merge) 10phedenskog: Show which branches failed restores happen on for Castor [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/32
[17:47:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 (10BLiviero-WMF) 03NEW
[17:49:43] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12238655 (10BLiviero-WMF) Added phab subtask for DC Ops to look...
[17:52:54] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12238677 (10wiki_willy) a:03VRiley-WMF Adding ops-eqiad tag and assigning to Valerie.  ++ @RobH - can you confirm...
[17:59:20] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12238697 (10RobH) >>! In T435537#12238677, @wiki_willy wrote: > Adding ops-eqiad tag and assigning to Valerie. >  >...
[18:46:31] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] "Acknowledged" [integration/config] - 10https://gerrit.wikimedia.org/r/1321619 (owner: 10Jforrester)
[23:20:59] <bd808>	 *sigh* Beta Cluster is functionally down because of crawlers. They have used 51,606 distinct /24 networks today to make 1,431,324 requests. so. tiring.
[23:23:04] <bd808>	 that "51,606 distinct /24 networks" stat is a undercount. Those are the unique /24 networks that have each been used to make *2* requests. There are other /24s carrying more traffic.
[23:23:54] <bd808>	 151,920 is the total number of distinct /24s seen today
[23:26:06] <bd808>	 !log deployment-mediawiki14: `systemctl stop php8.3-fpm; sleep 5m; systemctl start php8.3-fpm` -- maybe the bot storm will break if we give fast 500 responses for 5 minutes.
[23:26:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[23:30:17] <wikibugs>	 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12239739 (10bd808) `lang=irc [23:20]  <    bd808> *sigh* Beta Cluster is functionally down because of crawlers. They have used 51,606 distinct /24 netwo...
[23:33:09] <wikibugs>	 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12239747 (10bd808) > maybe the bot storm will break if we give fast 500 responses for 5 minutes. Spoiler, it did not.
[23:34:42] <wikibugs>	 (03PS1) 10Dreamy Jazz: Zuul: [mediawiki/extensions/ConfirmEdit] Run extension jsdoc jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1327660 (https://phabricator.wikimedia.org/T434585)
[23:43:38] <wikibugs>	 06Release-Engineering-Team (Doing 😎), 10SyntaxHighlight, 05MW-1.47-notes (1.47.0-wmf.17; 2026-08-25): New 2.21.0 upstream release for Pygments - https://phabricator.wikimedia.org/T421653#12239763 (10bd808) [[https://wikitech.wikimedia.org/wiki/Shellbox#Deploying_a_new_version|Deployment plan]] for 2026-09-24...