[00:14:56] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: WMF-NDA group request for gturkington - https://phabricator.wikimedia.org/T435095#12239840 (10Urbanecm) 05Resolvedβ†’03Open a:05Aklapperβ†’03None @Aklapper: Per https://www.mediawiki.org/wiki/Phabricator/Help#Multi-factor_authentication and #wmf-nda... [00:15:35] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: WMF-NDA group request for gturkington - https://phabricator.wikimedia.org/T435095#12239843 (10Urbanecm) a:03GTurkington-WMF @GTurkington-WMF Please unassign yourself once the MFA has been setup on your account. [00:25:56] (03PS1) 10Ahmon Dancy: README: Update plugin testing instructions [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327666 (https://phabricator.wikimedia.org/T434726) [00:32:01] (03PS1) 10Ahmon Dancy: plugins-router: set the MIME type of a served plugin [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327668 (https://phabricator.wikimedia.org/T434726) [00:33:08] (03CR) 10Ahmon Dancy: [C:03+2] README: Update plugin testing instructions [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327666 (https://phabricator.wikimedia.org/T434726) (owner: 10Ahmon Dancy) [00:33:50] (03Merged) 10jenkins-bot: README: Update plugin testing instructions [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327666 (https://phabricator.wikimedia.org/T434726) (owner: 10Ahmon Dancy) [00:35:14] (03PS1) 10Ahmon Dancy: plugins: Add wm-wheres-my-code-running plugin [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327669 (https://phabricator.wikimedia.org/T434726) [00:38:29] 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Where is my change running tool - https://phabricator.wikimedia.org/T434726#12239916 (10dancy) Looks like this: {F99311598 size=full} [00:50:22] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Jatwa - https://phabricator.wikimedia.org/T435500#12239947 (10Urbanecm) 05Openβ†’03Resolved a:03Urbanecm And I see https://gitlab.wikimedia.org/jatwa is now approved! [00:58:55] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12239970 (10Hakimi97) I asked ChatGPT 5.6 Sol Extra High, and it seems to recommend adding admission challenge (CAPTCHA, Wikimedia/OAuth-based verificat... [01:04:44] 10Continuous-Integration-Infrastructure, 10Castor, 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571 (10Mhurd) 03NEW [01:25:16] 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Configure additional httpbb checks to perform during a deployment - https://phabricator.wikimedia.org/T428972#12240001 (10Scott_French) Thanks, Bryan! I made some minor formatting / wording tweaks, but that looks solid. [02:41:01] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12240092 (10Novem_Linguae) It's probably harder than it looks to solve this due to wanting to keep beta cluster pretty similar to prod, and due to bots... [03:06:26] (03PS1) 10Vaughn Walters: jjb: [selenium-daily-beta-*] Stop e-mailing beta Selenium failures [integration/config] - 10https://gerrit.wikimedia.org/r/1327655 (https://phabricator.wikimedia.org/T435552) [03:09:19] (03CR) 10Vaughn Walters: "@hashar@free.fr or @jforrester@wikimedia.org I will +2 and update this if one of yall could give me a +1 to make sure I'm not missing anyt" [integration/config] - 10https://gerrit.wikimedia.org/r/1327655 (https://phabricator.wikimedia.org/T435552) (owner: 10Vaughn Walters) [03:30:13] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12240138 (10jeremyb) >>! In T420833#12240092, @Novem_Linguae wrote: > It's probably harder than it looks to solve this due to wanting to keep beta clust... [06:31:01] 10Diffusion, 10Phabricator: Diffusion uses insufficient color coding for Less files (fixed in Pygments 2.8.0 version in Debian Bookworm) - https://phabricator.wikimedia.org/T147980#12240233 (10Aklapper) 05Stalledβ†’03Resolved We are using [pygmentize 2.18.0](https://packages.debian.org/trixie/python3-pyg... [06:34:26] 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#12240240 (10Aklapper) @Arnoldokoth Can this task be resolved? [06:35:04] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Collaboration-Services: Upgrade phab (phorge) hosts to bookworm - https://phabricator.wikimedia.org/T372619#12240242 (10Aklapper) @Arnoldokoth Can this task be resolved? [06:47:20] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work: Phabricator Switchover - from phab1004 (Bullseye, PHP 7.4) to phab1005 (Trixie, PHP 8.4) - https://phabricator.wikimedia.org/T435087#12240244 (10Aklapper) [07:09:14] 10Phabricator-Bot-Requests, 10Tool-techcontribs: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582 (10Chlod) 03NEW [07:10:00] 10Phabricator-Bot-Requests, 10Tool-techcontribs: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240293 (10Chlod) [07:15:27] 10Phabricator-Bot-Requests, 10Tool-techcontribs: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240294 (10Chlod) [07:15:40] 10Phabricator-Bot-Requests, 10Tool-techcontribs: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240295 (10Chlod) [07:17:15] 10Phabricator (Upstream), 07Upstream: Some task notifications include a @mention entry in the X-Phabricator-Stamps mail header for no obvious reason - https://phabricator.wikimedia.org/T266328#12240296 (10Aklapper) [08:18:30] 10Phabricator-Bot-Requests, 10Tool-techcontribs: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240405 (10taavi) a:03taavi [08:21:42] 10Phabricator-Bot-Requests, 10Tool-techcontribs: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240410 (10taavi) 05Openβ†’03Resolved Created @TechContribsBot. See P96218 for the API token. [08:28:17] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: WMF-NDA group request for gturkington - https://phabricator.wikimedia.org/T435095#12240427 (10Aklapper) 05Openβ†’03Resolved a:05GTurkington-WMFβ†’03Aklapper Hi, I re-added the user to #WMF-NDA as the user will not be able to log into Phabricator... [08:50:02] 10Phabricator-Bot-Requests, 10Tool-techcontribs: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240470 (10Chlod) Thank you! :D [08:54:18] (03CR) 10Hashar: [C:03+1] "Looks good Vaughn, well done :-]" [integration/config] - 10https://gerrit.wikimedia.org/r/1327655 (https://phabricator.wikimedia.org/T435552) (owner: 10Vaughn Walters) [09:51:19] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Delete page is broken - https://phabricator.wikimedia.org/T435589 (10jnuche) 03NEW [09:52:13] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Delete page is broken - https://phabricator.wikimedia.org/T435589#12240638 (10jnuche) [10:10:16] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Delete page is broken - https://phabricator.wikimedia.org/T435589#12240713 (10jnuche) 05Openβ†’03Resolved Fix is in prod [10:37:06] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services, 13Patch-For-Review: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12240766 (10hashar) a:03hashar [10:45:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:00:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:24:27] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12240963 (10LucasWerkmeister) >>! In T420833#12239970, @Hakimi97 wrote: > I asked ChatGPT 5.6 Sol Extra High This is //basically// the same as going β€œI... [12:14:40] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12241167 (10Hakimi97) >>! In T420833#12240963, @LucasWerkmeister wrote: >>>! In T420833#12239970, @Hakimi97 wrote: >> I asked ChatGPT 5.6 Sol Extra High... [12:27:46] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/ParserMigration] Add ConfirmEdit Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1328180 [12:39:36] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for CoderAnimeshPathak - https://phabricator.wikimedia.org/T435611 (10CoderAnimeshPathak) 03NEW [13:00:12] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/ParserMigration] Add ConfirmEdit Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1328180 (owner: 10Fomafix) [13:00:51] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/ConfirmEdit] Run extension jsdoc jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1327660 (https://phabricator.wikimedia.org/T434585) (owner: 10Dreamy Jazz) [13:01:43] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/PersonalDashboard] Publish doxygen docs [integration/config] - 10https://gerrit.wikimedia.org/r/1327222 (https://phabricator.wikimedia.org/T435392) (owner: 10MusikAnimal) [13:02:04] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ParserMigration] Add ConfirmEdit Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1328180 (owner: 10Fomafix) [13:02:39] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ConfirmEdit] Run extension jsdoc jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1327660 (https://phabricator.wikimedia.org/T434585) (owner: 10Dreamy Jazz) [13:03:39] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PersonalDashboard] Publish doxygen docs [integration/config] - 10https://gerrit.wikimedia.org/r/1327222 (https://phabricator.wikimedia.org/T435392) (owner: 10MusikAnimal) [13:07:56] !log integration: triggered Doxygen doc for PersonalDashboard extension using: `zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/PersonalDashboard --change 1328044,1` # T435392 [13:07:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:07:59] T435392: Publish PersonalDashboard documentation to doc.wikimedia.org - https://phabricator.wikimedia.org/T435392 [13:08:47] (03CR) 10Hashar: [C:03+2] "I have deployed the change and manually triggered a postmerge run for the latest merged change ( I811d9196a795bdb5e47204d8fbff1ca16fd5a670" [integration/config] - 10https://gerrit.wikimedia.org/r/1327222 (https://phabricator.wikimedia.org/T435392) (owner: 10MusikAnimal) [13:10:45] (03CR) 10Hashar: [C:03+2] "I have deployed the CI config change which enables the Doxygen publishing and manually ran it. https://doc.wikimedia.org/PersonalDashboar" [integration/docroot] - 10https://gerrit.wikimedia.org/r/1327223 (https://phabricator.wikimedia.org/T435392) (owner: 10MusikAnimal) [13:12:12] (03Merged) 10jenkins-bot: opensource: add PersonalDashboard docs to MW components [integration/docroot] - 10https://gerrit.wikimedia.org/r/1327223 (https://phabricator.wikimedia.org/T435392) (owner: 10MusikAnimal) [13:17:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:27:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:36:21] (03CR) 10Hashar: "Thank you so much for the documentation update! Replacing an existing plugin is what upstream recommends (after they have moved Gerrit FE" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327666 (https://phabricator.wikimedia.org/T434726) (owner: 10Ahmon Dancy) [13:37:06] (03PS2) 10Hashar: plugins-router: set the MIME type of a served plugin [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327668 (https://phabricator.wikimedia.org/T434726) (owner: 10Ahmon Dancy) [13:38:19] (03CR) 10Hashar: [C:03+2] "Nit: I have amended the commit message to rewrap it at 78 characters" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327668 (https://phabricator.wikimedia.org/T434726) (owner: 10Ahmon Dancy) [13:39:02] (03Merged) 10jenkins-bot: plugins-router: set the MIME type of a served plugin [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327668 (https://phabricator.wikimedia.org/T434726) (owner: 10Ahmon Dancy) [13:54:42] (03open) 10phedenskog: Add a Performance section to the browser tests dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/33 [13:56:14] (03CR) 10Hashar: [C:03+1] "That is quite impressive and I am pretty sure that feature has been a very long ask from our developers." [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327669 (https://phabricator.wikimedia.org/T434726) (owner: 10Ahmon Dancy) [13:58:17] (03merge) 10phedenskog: Add a Performance section to the browser tests dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/33 [14:00:19] (03open) 10jnuche: jenkins-rel: use new LVS URL downloaders [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/123 (https://phabricator.wikimedia.org/T429175) [14:00:34] (03merge) 10jnuche: jenkins-rel: use new LVS URL downloaders [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/123 (https://phabricator.wikimedia.org/T429175) [14:01:37] (03CR) 10Hashar: [C:03+2] Measure specific test commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1319382 (https://phabricator.wikimedia.org/T432966) (owner: 10Phedenskog) [14:21:32] (03Merged) 10jenkins-bot: Measure specific test commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1319382 (https://phabricator.wikimedia.org/T432966) (owner: 10Phedenskog) [14:35:20] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12241539 (10jeremyb) >>! In T420833#12240963, @LucasWerkmeister wrote: > Also, for the HTTP basic auth idea, see T420833#12082393. I'm sure I saw that... [14:38:17] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12241545 (10jeremyb) >>! In T420833#12241167, @Hakimi97 wrote: > have been using LLM to identify security vulnerabilities this isn't a security vulnera... [14:59:06] 10Diffusion: Attempting to view a non-existent file through Diffusion causes an unhandled CommandException - https://phabricator.wikimedia.org/T435627 (10SomeRandomDeveloper) 03NEW [14:59:36] 10Diffusion, 10Phabricator: Attempting to view a non-existent file through Diffusion causes an unhandled CommandException - https://phabricator.wikimedia.org/T435627#12241696 (10SomeRandomDeveloper) [15:00:26] (03open) 10phedenskog: Show min:sec and a per-extension trend on the browser tests page [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/34 [15:07:52] (03merge) 10phedenskog: Show min:sec and a per-extension trend on the browser tests page [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/34 [15:11:28] (03open) 10phedenskog: Make the window 14 days for browser test [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/35 [15:11:35] (03merge) 10phedenskog: Make the window 14 days for browser test [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/35 [15:12:27] (03CR) 10Ahmon Dancy: [C:03+2] plugins: Add wm-wheres-my-code-running plugin [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327669 (https://phabricator.wikimedia.org/T434726) (owner: 10Ahmon Dancy) [15:13:12] (03Merged) 10jenkins-bot: plugins: Add wm-wheres-my-code-running plugin [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1327669 (https://phabricator.wikimedia.org/T434726) (owner: 10Ahmon Dancy) [15:18:25] Project mwcore-phpunit-coverage-master build #5484: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/5484/ [15:18:26] (03open) 10phedenskog: Add click to legends to show only that metric [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/36 [15:18:30] (03merge) 10phedenskog: Add click to legends to show only that metric [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/36 [15:19:46] 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Where is my change running tool - https://phabricator.wikimedia.org/T434726#12241769 (10dancy) 05Openβ†’03Resolved This has been deployed. [15:24:25] Project mediawiki-core-doxygen build #22328: 04FAILURE in 6 min 22 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22328/ [15:40:50] 10Diffusion, 10Phabricator: Attempting to view a non-existent file through Diffusion causes an unhandled CommandException - https://phabricator.wikimedia.org/T435627#12241872 (10Les4353) Can you report it in https://we.phorge.it ? [15:41:43] 10Diffusion, 10Phabricator (Upstream), 07Upstream: Attempting to view a non-existent file through Diffusion causes an unhandled CommandException - https://phabricator.wikimedia.org/T435627#12241875 (10Aklapper) p:05Triageβ†’03Low [15:43:52] 10Diffusion, 10Phabricator (Upstream), 07Upstream: Attempting to view a non-existent file through Diffusion causes an unhandled CommandException - https://phabricator.wikimedia.org/T435627#12241889 (10SomeRandomDeveloper) >>! In T435627#12241872, @Les4353 wrote: > Can you report it in https://we.phorge.it ?... [15:45:17] 10Diffusion, 10Phabricator: Attempting to view a non-existent file through Diffusion causes an unhandled CommandException - https://phabricator.wikimedia.org/T435627#12241893 (10Les4353) >>! 在T435627#12241889中，@SomeRandomDeveloperε†™ι“οΌš >>>! In T435627#12241872, @Les4353 wrote: >> Can you report it in https://we.... [16:02:25] (03open) 10dancy: tests: Drop the grace period from the monitor_release comments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1274 (https://phabricator.wikimedia.org/T375514) [16:03:40] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for MediaWiki-extensions-WandaVoice - https://phabricator.wikimedia.org/T434418#12241962 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Requested public project tag #MediaWiki-extensions-WandaVoice has been created: https://phabr... [16:05:17] (03merge) 10dancy: tests: Drop the grace period from the monitor_release comments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1274 (https://phabricator.wikimedia.org/T375514) [16:15:05] (03open) 10dancy: tests: Correct comments that no longer match the code [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1275 (https://phabricator.wikimedia.org/T375514) [16:17:49] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12242055 (10VRiley-WMF) After looking into this a bit, it seems we can set these to performance mode, but... [16:19:03] (03update) 10dancy: tests: Do not describe helm as holding a release [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1275 (https://phabricator.wikimedia.org/T412941) [16:19:03] (03update) 10dancy: tests: Do not describe helm as holding a release [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1275 (https://phabricator.wikimedia.org/T375514 https://phabricator.wikimedia.org/T412941) [16:19:39] (03merge) 10dancy: tests: Do not describe helm as holding a release [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1275 (https://phabricator.wikimedia.org/T412941) [16:20:00] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [16:20:08] (03update) 10dancy: deploy-service: Run helmfile commands, add --confirm-diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1273 (https://phabricator.wikimedia.org/T412941) [16:20:14] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [16:24:09] 06Release-Engineering-Team (Doing 😎): Where is my change running tool - https://phabricator.wikimedia.org/T434726#12242070 (10brennen) This is //sick//. [16:24:46] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [16:24:58] (03update) 10dancy: deploy-service: Run helmfile commands, add --confirm-diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1273 (https://phabricator.wikimedia.org/T412941) [16:24:59] (03CR) 10Vaughn Walters: [C:03+2] "jobs updated, so now merging" [integration/config] - 10https://gerrit.wikimedia.org/r/1327655 (https://phabricator.wikimedia.org/T435552) (owner: 10Vaughn Walters) [16:25:00] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [16:27:05] (03Merged) 10jenkins-bot: jjb: [selenium-daily-beta-*] Stop e-mailing beta Selenium failures [integration/config] - 10https://gerrit.wikimedia.org/r/1327655 (https://phabricator.wikimedia.org/T435552) (owner: 10Vaughn Walters) [16:35:45] Yippee, build fixed! [16:35:45] Project mediawiki-core-doxygen build #22329: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22329/ [16:43:26] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [16:44:25] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12242141 (10bd808) >>! In T420833#12241167, @Hakimi97 wrote: > Okay here is another question: Can someone here guarantee that the beta cluster to have t... [17:00:52] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T432292#12242171 (10bd808) 05In progressβ†’03Resolved [17:02:03] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435421#12242174 (10bd808) There have been a lot of these in the last couple of days. >>! In T435408#12234663, @bd808 wrote: > I think the crawler pressure is so high that the CDN e... [17:06:53] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435421#12242182 (10bd808) One of the things I tried to do in {T435193} was turn down the varnish log output because `varnish-frontend-slowlog` seems to have very recently become a... [18:05:10] 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#12242438 (10Arnoldokoth) @Aklapper I was hoping to use this to track decommissioning `phab1004`. I think once we've done that we can consider this complete. [18:32:52] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12242483 (10BLiviero-WMF) HI! will let @Andrew weigh in on whether we can depool and make this change to... [18:39:48] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12242494 (10Andrew) Thanks for the followup @VRiley-WMF -- indeed these will need to be drained before we... [20:18:50] 06Release-Engineering-Team (Doing 😎): Where is my change running tool - https://phabricator.wikimedia.org/T434726#12242704 (10bd808) [20:18:51] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Epic, 06Quality-and-Test-Engineering-Team (Test Infrastructure): Pretrain (nΓ©e Group -1) QTE validation environment - https://phabricator.wikimedia.org/T369112#12242705 (10bd808) [20:39:27] 06Release-Engineering-Team (Doing 😎), 10Tool-wheres-my-code-running: Where is my change running tool - https://phabricator.wikimedia.org/T434726#12242739 (10bd808) I made a #tool-wheres-my-code-running project with #striker and a stub developer docs page at https://wikitech.wikimedia.org/wiki/Tool:Wheres-m... [21:00:54] 06Release-Engineering-Team (Doing 😎), 10Tool-wheres-my-code-running, 13Patch-For-Review: wheres-my-code-running.git missing license information - https://phabricator.wikimedia.org/T435661#12242786 (10bd808) 05Openβ†’03In progress a:03bd808 [21:24:16] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12242859 (10LucasWerkmeister) >>! In T420833#12241539, @jeremyb wrote: > how important can an automation be if no one noticed it stopped working? I ima... [21:31:57] 06Release-Engineering-Team (Doing 😎), 10Tool-wheres-my-code-running, 13Patch-For-Review: Convert deb install spec from Aptfile to project.toml - https://phabricator.wikimedia.org/T435662#12242900 (10bd808) 05Openβ†’03In progress [21:36:08] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12242907 (10AlexisJazz) >>! In T420833#12239970, @Hakimi97 wrote: > I asked ChatGPT 5.6 Sol Extra High, and it seems to recommend adding admission chall... [21:42:22] Yippee, build fixed! [21:42:22] Project mwcore-phpunit-coverage-master build #5485: 09FIXED in 42 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/5485/ [21:43:43] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12242922 (10jeremyb) >>! In T420833#12242859, @LucasWerkmeister wrote: > I imagine plenty of people have noticed, but what are they supposed to do about... [22:27:24] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [22:27:29] (03update) 10dancy: deploy-service: Run helmfile commands, add --confirm-diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1273 (https://phabricator.wikimedia.org/T412941) [22:27:33] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [23:10:05] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12243156 (10bd808) >>! In T420833#12242907, @AlexisJazz wrote: > Would it be okay to create, say, a few dozen or hundred articles on beta cluster with L...