[05:47:39] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11188066 (... [07:32:28] (03update) 10aklapper: Merge Phorge upstream master as of 2025-09-08 into wmf/stable [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/99 (https://phabricator.wikimedia.org/T404134) (owner: 10brennen) [09:44:51] 06Project-Admins, 10MediaWiki-extensions-OATHAuth: Create only phab project for WebAuthn? - https://phabricator.wikimedia.org/T404818#11188541 (10Aklapper) [10:01:43] 06Project-Admins, 10MediaWiki-extensions-OATHAuth: Create dedicated phab project for WebAuthn? - https://phabricator.wikimedia.org/T404818#11188610 (10Reedy) [10:31:16] 10Phabricator: Change Maniphest AdvancedSearch defaults for "group by" and "order by" - https://phabricator.wikimedia.org/T404843 (10hashar) 03NEW [10:32:25] 00:02:19.447 1) MediaWiki\Tests\Maintenance\Includes\SchemaGeneratorTest::testNormalizePath with data set "Extension directory under $IP then path is relative to the extension root. " ('db_patches/patch.json', '/workspace/src/tests/phpunit/...h.json', '/workspace/src/tests/phpunit/...enance', '/workspace/src/tests/phpunit/...nsions') [10:32:34] thank you PHPUnit for stripping the strings .. [10:50:14] 10Phabricator: Change Maniphest AdvancedSearch defaults for "group by" and "order by" - https://phabricator.wikimedia.org/T404843#11188870 (10A_smart_kitten) FWIW, my personal opinion on this is something like: - Group By: Yeah, I can't recall a time where I've wanted to group the results by task priority (or b... [12:41:43] 06Project-Admins: Create project tag for WikiIndaba-Hackathon-2025 - https://phabricator.wikimedia.org/T404853 (10Eugene233) 03NEW [13:09:41] (03PS2) 10Hashar: Zuul: explicitly add CodeEditor with AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/1188880 (https://phabricator.wikimedia.org/T389998) [13:25:06] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), 10Wikidata-Omega (The Board): Zuul does not support Gerrit rebase submit strategy correctly, may merge deterministically broken code (breaking all ... - https://phabricator.wikimedia.org/T400918#11189388 [15:01:14] 10Beta-Cluster-Infrastructure: Unable to connect to deployment-mwmaint03.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T404873 (10Etonkovidova) 03NEW [15:19:34] 06Project-Admins, 10Release-Engineering-Team (Doing 😎): Create project tag for WikiIndaba-Hackathon-2025 - https://phabricator.wikimedia.org/T404853#11190169 (10Aklapper) 05Openβ†’03Resolved Requested public project #wikiindaba-hackathon-2025 has been created: https://phabricator.wikimedia.org/project/vi... [15:25:56] (03CR) 10Krinkle: [C:03+2] Replace utils/jenkins-jobs-list.py by ./jjb-list [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 (owner: 10Hashar) [15:27:34] (03Merged) 10jenkins-bot: Replace utils/jenkins-jobs-list.py by ./jjb-list [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 (owner: 10Hashar) [15:50:18] 10Beta-Cluster-Infrastructure: Unable to connect to deployment-mwmaint03.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T404873#11190375 (10bd808) The root problem here looks to be: > Unable to negotiate with 185.15.56.16 port 22: no matching key exchange method found. Their offer: ml... [15:53:32] 10Beta-Cluster-Infrastructure: No acceptable KexAlgorithm found between "OpenSSH_9.9p2, LibreSSL 3.3.6" and "OpenSSH_10.0p2 Debian-7, OpenSSL 3.5.1 1 Jul 2025" (Debian Trixie, bastion.wmcloud.org) - https://phabricator.wikimedia.org/T404873#11190406 (10bd808) [15:53:48] 10Continuous-Integration-Config: Reconcile dependencies.yaml and phan_dependencies.yaml - https://phabricator.wikimedia.org/T391008#11190408 (10Dreamy_Jazz) I would argue this isn't as simple as just combining the lists. For example: # Take Extension:A and Extension:B # A depends on B, with associated tests and... [15:57:05] 10GitLab: GitLab Private Repository Request for: Client-side Bot Detection logic - https://phabricator.wikimedia.org/T404885 (10Milimetric) 03NEW [16:00:54] 10Beta-Cluster-Infrastructure: No acceptable KexAlgorithm found between "OpenSSH_9.9p2, LibreSSL 3.3.6" and "OpenSSH_10.0p2 Debian-7, OpenSSL 3.5.1 1 Jul 2025" (Debian Trixie, bastion.wmcloud.org) - https://phabricator.wikimedia.org/T404873#11190460 (10Lucas_Werkmeister_WMDE) FWIW, I have no special `KexAlgorith... [16:09:43] 10Scap: Allow using path/file wildcards with scap remove-patch - https://phabricator.wikimedia.org/T404887 (10Reedy) 03NEW [16:11:27] 10Scap: Allow using path/file wildcards with scap remove-patch - https://phabricator.wikimedia.org/T404887#11190573 (10taavi) β†’14Duplicate dup:03T404593 [16:11:28] 10Scap: scap remove-patch: support removing multiple patches - https://phabricator.wikimedia.org/T404593#11190575 (10taavi) [16:12:15] lol [16:16:42] sorry [16:19:15] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11190630 (... [16:20:15] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11190631 (... [16:22:05] taavi: have you made a task anywhere about the opentofu provider download issues with github? [16:23:09] bd808: not other than T403028 [16:23:10] T403028: toolforge tofu-provisioning: Cache terraform-provider-openstack binary somewhere - https://phabricator.wikimedia.org/T403028 [16:27:32] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11190675 (... [16:48:35] 10Continuous-Integration-Config: Reconcile dependencies.yaml and phan_dependencies.yaml - https://phabricator.wikimedia.org/T391008#11190736 (10Daimona) I think that scenario is worth considering, but I don't think it would prevent this work entirely. For example, we could keep phan_dependencies but only for ove... [16:55:24] 06Release-Engineering-Team, 10Catalyst (PatchDemo): Enable creation of a patchdemo with all repos rolled back to a specific custom moment in time - https://phabricator.wikimedia.org/T403854#11190746 (10hashar) [17:00:48] 06Release-Engineering-Team, 10Catalyst (PatchDemo): Enable creation of a patchdemo with all repos rolled back to a specific custom moment in time - https://phabricator.wikimedia.org/T403854#11190752 (10hashar) //Copy pasting my comment from Slack// This use case would be solved by crafting a mono repository t... [17:01:02] (03approved) 10jforrester: make-release: Start branching ReaderExperiments [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/214 (https://phabricator.wikimedia.org/T404398) (owner: 10egardner) [17:02:23] (03merge) 10jforrester: make-release: Start branching ReaderExperiments [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/214 (https://phabricator.wikimedia.org/T404398) (owner: 10egardner) [17:04:14] 06Release-Engineering-Team, 10Catalyst (PatchDemo): Enable creation of a patchdemo with all repos rolled back to a specific custom moment in time - https://phabricator.wikimedia.org/T403854#11190769 (10hashar) Regarding the old `wmf/` branches, we went to delete them since T244368 and I have fixed the deletion... [17:15:58] 06Release-Engineering-Team, 10Catalyst (PatchDemo): Enable creation of a patchdemo with all repos rolled back to a specific custom moment in time - https://phabricator.wikimedia.org/T403854#11190799 (10thcipriani) >>! In T403854#11164150, @thcipriani wrote: > I think this might "just work" if we convert branch... [17:32:06] 06Release-Engineering-Team, 10Catalyst (PatchDemo): Enable creation of a patchdemo with all repos rolled back to a specific custom moment in time - https://phabricator.wikimedia.org/T403854#11190866 (10A_smart_kitten) >>! In T403854#11190769, @hashar wrote: > […] Then I do not know how to retrieve the original... [18:28:36] 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.45.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T396380#11191092 (10dduvall) [18:31:40] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11191106 (10BTracy-WMF) Hello, The Wikidata Platform team was just staffed in August and is taking over ownership of the [[ https://phabricator.wikimedia.org/... [18:31:41] (03open) 10dancy: app.py: Improve REGGIE_DATA_DIR error message [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/100 [18:31:45] (03update) 10dancy: app.py: Improve REGGIE_DATA_DIR error message [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/100 [18:35:52] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11191124 (10Jdforrester-WMF) >>! In T706#11191105, @BTracy-WMF wrote: > Hello, > > The Wikidata Platform team was just staffed in August and is taking over ow... [18:36:09] (03open) 10dancy: .pipeline/blubber.yaml: Use Debian Bookworm base images [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/101 [18:36:12] (03update) 10dancy: .pipeline/blubber.yaml: Use Debian Bookworm base images [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/101 [19:04:43] 06Project-Admins, 10MediaWiki-extensions-OATHAuth: Create dedicated phab project for WebAuthn? - https://phabricator.wikimedia.org/T404818#11191226 (10taavi) I haven't felt the need for a separate project so far.. but if one is created, it definitely should not be a subproject as we want to tag some tasks with... [19:11:08] (03PS1) 10Andrew Bogott: Review access change [openstack/horizon/horizon] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1189292 [19:11:25] (03Abandoned) 10Andrew Bogott: Review access change [openstack/horizon/horizon] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1189292 (owner: 10Andrew Bogott) [19:23:57] (03open) 10dancy: .pipeline/blubber.yaml: Remove old tag from bullseye repo reference [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/102 [19:24:00] (03update) 10dancy: .pipeline/blubber.yaml: Remove old tag from bullseye repo reference [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/102 [19:33:25] (03PS1) 10Andrew Bogott: Review access change [openstack/horizon/horizon] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1189297 [19:41:54] (03merge) 10dancy: .pipeline/blubber.yaml: Remove old tag from bullseye repo reference [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/102 [19:42:06] (03open) 10dancy: blubber.yaml: Run under tini [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/103 [19:42:13] (03update) 10dancy: blubber.yaml: Run under tini [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/103 [19:43:00] (03close) 10dancy: .pipeline/blubber.yaml: Use Debian Bookworm base images [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/101 [19:43:46] (03update) 10dancy: app.py: Improve REGGIE_DATA_DIR error message [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/100 [19:45:15] (03merge) 10dancy: app.py: Improve REGGIE_DATA_DIR error message [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/100 [19:45:42] (03update) 10dancy: blubber.yaml: Run under tini [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/103 [19:46:39] (03merge) 10dancy: blubber.yaml: Run under tini [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/103 [19:50:27] 10Beta-Cluster-Infrastructure: No acceptable KexAlgorithm found between "OpenSSH_9.9p2, LibreSSL 3.3.6" and "OpenSSH_10.0p2 Debian-7, OpenSSL 3.5.1 1 Jul 2025" (Debian Trixie, bastion.wmcloud.org) - https://phabricator.wikimedia.org/T404873#11191414 (10bd808) I don't think there is anything especially Beta Clust... [19:50:49] 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 10ChangeProp, 06Data-Engineering, and 2 others: Run EventBus tests in MediaWiki core CI - https://phabricator.wikimedia.org/T257583#11191416 (10BPirkle) Untagging MWI, and also removing a bunch of obsolete tags. [19:52:43] (03open) 10dancy: Chart.yaml: appVersion: "v1.16.0" [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/104 [19:52:46] (03update) 10dancy: Chart.yaml: appVersion: "v1.16.0" [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/104 [19:53:30] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11191431 (10BTracy-WMF) [19:53:45] (03merge) 10dancy: Chart.yaml: appVersion: "v1.16.0" [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/104 [19:58:52] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11191440 (10Aklapper) [20:09:19] 10Beta-Cluster-Infrastructure: No acceptable KexAlgorithm found between "OpenSSH_9.9p2, LibreSSL 3.3.6" and "OpenSSH_10.0p2 Debian-7, OpenSSL 3.5.1 1 Jul 2025" (Debian Trixie, bastion.wmcloud.org) - https://phabricator.wikimedia.org/T404873#11191457 (10bd808) @Etonkovidova based on the information you provided i... [20:09:49] 10Beta-Cluster-Infrastructure: No acceptable KexAlgorithm found between "OpenSSH_9.9p2, LibreSSL 3.3.6" and "OpenSSH_10.0p2 Debian-7, OpenSSL 3.5.1 1 Jul 2025" (Debian Trixie, bastion.wmcloud.org) - https://phabricator.wikimedia.org/T404873#11191459 (10bd808) p:05Triageβ†’03Medium [20:14:03] 10Beta-Cluster-Infrastructure, 10observability: Bring beta-logs.wmcloud.org to a readable state - https://phabricator.wikimedia.org/T389239#11191479 (10bd808) [20:15:15] (03open) 10dancy: manifest.py: Add support for application/vnd.cncf.helm.config.v1+json [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/105 [20:15:19] (03update) 10dancy: manifest.py: Add support for application/vnd.cncf.helm.config.v1+json [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/105 [20:16:28] (03merge) 10dancy: manifest.py: Add support for application/vnd.cncf.helm.config.v1+json [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/105 [20:19:38] (03open) 10dancy: Chart.yaml: appVersion: "v1.17.0" [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/106 [20:19:42] (03update) 10dancy: Chart.yaml: appVersion: "v1.17.0" [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/106 [20:20:57] (03merge) 10dancy: Chart.yaml: appVersion: "v1.17.0" [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/106 [20:20:59] 10Beta-Cluster-Infrastructure, 10observability: Bring beta-logs.wmcloud.org to a readable state - https://phabricator.wikimedia.org/T389239#11191528 (10bd808) The biggest normalized bucket right now is "Pool key 'CirrusSearch-Search:_elasticsearch' (CirrusSearch-Search): β§Όpoolcounter-connection-error⧽" which i... [20:23:19] 10Beta-Cluster-Infrastructure, 10observability: Bring beta-logs.wmcloud.org to a readable state - https://phabricator.wikimedia.org/T389239#11191538 (10bd808) 05Openβ†’03Declined I am going to close this as declined. I do think fixing broken things in Beta Cluster is worthwhile, but I do not feel this ta... [20:29:57] 10Gerrit (Gerrit 3.12), 06collaboration-services: Upgrade Gerrit hosts from Bullseye to Bookworm - https://phabricator.wikimedia.org/T392464#11191550 (10hashar) Paladox has let me know Gerrit maintainers have announced support for Gerrit 3.10 will stop on November 10th. https://gerrit-review.googlesource.com/... [20:34:41] 10Beta-Cluster-Infrastructure, 10Scap: Automate beta scap3/keyholder setup - https://phabricator.wikimedia.org/T144647#11191565 (10bd808) 05Openβ†’03Declined If this was an active problem someone would have bumped the ticket in the last **9 years**. [20:35:16] 10Beta-Cluster-Infrastructure: Create automation service to reduce toil and increase visibility of local Puppetserver cherry-picks - https://phabricator.wikimedia.org/T381948#11191570 (10bd808) [20:35:27] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Seen), 13Patch-For-Review: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#11191569 (10bd808) [20:36:40] 10Beta-Cluster-Infrastructure: No acceptable KexAlgorithm found between "OpenSSH_9.9p2, LibreSSL 3.3.6" and "OpenSSH_10.0p2 Debian-7, OpenSSL 3.5.1 1 Jul 2025" (Debian Trixie, bastion.wmcloud.org) - https://phabricator.wikimedia.org/T404873#11191581 (10Etonkovidova) 05Openβ†’03Resolved a:03Etonkovidova T... [20:37:42] (03CR) 10Hashar: "Finally! Thank you for you patience." [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 (owner: 10Hashar) [20:54:34] 10Continuous-Integration-Config, 10Math, 10MediaWiki-Platform-Team (Radar), 10MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout) - https://phabricator.wikimedia.org/T389998#11191649 (10hashar) a:... [21:09:56] 10Beta-Cluster-Infrastructure, 10Scap, 07ci-test-error: beta-code-update-eqiad: fatal: No url found for submodule path {repo} in .gitmodules - https://phabricator.wikimedia.org/T340030#11191680 (10bd808) 05Openβ†’03Resolved a:03hashar [21:14:44] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Seen), 07Epic, 07Performance Issue: [EPIC] Performance testing environment - https://phabricator.wikimedia.org/T67394#11191716 (10bd808) 05Openβ†’03Declined [21:15:42] (03CR) 10Hashar: [C:03+1] "Looks good thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/1188391 (owner: 10Jforrester) [21:16:07] 10Beta-Cluster-Infrastructure: Puppet agent is failing on restbase beta node - https://phabricator.wikimedia.org/T333438#11191725 (10bd808) 05Openβ†’03Resolved Resolved some time in the last 2.5 years. [21:20:12] 10Beta-Cluster-Infrastructure, 06Data-Persistence, 06SRE, 06Traffic: ATS isn't caching documents in deployment-cache-upload07 - https://phabricator.wikimedia.org/T322575#11191738 (10bd808) 05Openβ†’03Declined deployment-cache-text07 was replaced by deployment-cache-text08. `lang=shell-session,lines=1... [21:20:22] (03PS14) 10Hashar: Script to sync dependencies with extension registry [integration/config] - 10https://gerrit.wikimedia.org/r/1182115 (https://phabricator.wikimedia.org/T389998) [21:22:53] 10Beta-Cluster-Infrastructure: Setup monitoring for database servers in beta cluster - https://phabricator.wikimedia.org/T87093#11191764 (10bd808) 05Stalledβ†’03Open [21:23:25] (03open) 10aklapper: .arclint: Revert to standard phutil.xhpast [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/100 [21:23:41] 10Beta-Cluster-Infrastructure, 10CXServer: CXServer doesn't work on Beta Cluster - https://phabricator.wikimedia.org/T323417#11191771 (10bd808) 05Openβ†’03Invalid >>! In T323417#8534720, @YedidyaPopper wrote: > @Nikerabbit The issue resolved itself after waiting a few days, thanks anyway [21:24:04] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Remove custom Wikimedia xhpast Linter - https://phabricator.wikimedia.org/T404760#11191773 (10Aklapper) p:05Triageβ†’03Low a:03Aklapper [21:25:34] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-SettingsBuilder, 07ci-test-error: beta-update-databases-eqiad failing due to invalid MediaWiki configuration parameters - https://phabricator.wikimedia.org/T313128#11191782 (10bd808) 05Openβ†’03Resolved This job certa... [21:26:04] 10Beta-Cluster-Infrastructure: Setup monitoring for database servers in beta cluster - https://phabricator.wikimedia.org/T87093#11191787 (10Dzahn) > a check to ensure the mysql process is running. Doesn't puppet already ensure that the mysql process is running? And if puppet has a failure there are already aut... [21:26:34] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Remove custom Wikimedia xhpast Linter - https://phabricator.wikimedia.org/T404760#11191788 (10Aklapper) * First part: Phab: https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/100 * Second part: Arcanist: Remove... [21:31:28] 10Gerrit (Gerrit 3.12), 06collaboration-services: Upgrade Gerrit hosts from Bullseye to Bookworm - https://phabricator.wikimedia.org/T392464#11191804 (10Dzahn) That's because the actual ticket is T372804 which was created 10 months earlier. Most of this was already done back in September 2024 and has been wait... [21:31:29] 10Beta-Cluster-Infrastructure: Setup monitoring for database servers in beta cluster - https://phabricator.wikimedia.org/T87093#11191806 (10bd808) >>! In T87093#11191787, @Dzahn wrote: >> a check to ensure the mysql process is running. > > Doesn't puppet already ensure that the mysql process is running? No, an... [21:32:17] !log Started running MediaWiki jobs with non recursive dependencies injected, excluding selenium jobs and using 24 workers. From contint1002 in a screen. Log written to /home/hashar/run.20250917.log # T389998 [21:32:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:32:20] T389998: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout) - https://phabricator.wikimedia.org/T389998 [21:32:37] turns out we have plenty of executors nowadays https://integration.wikimedia.org/ci/label/Docker/load-statistics [21:33:02] I ll analyse the logs later and update https://docs.google.com/spreadsheets/d/1JA1AsCxcDN76WQOhhpxsgiQgKxJNzyjw2Rsl4xnXsyE/edit?gid=0#gid=0 [21:33:33] 10Beta-Cluster-Infrastructure: Setup monitoring for database servers in beta cluster - https://phabricator.wikimedia.org/T87093#11191820 (10Dzahn) Ah, right, I remember something like this now (puppet not supposed to auto-restart mysqld process). Is this blocked on "existence of monitoring system" (thinking of... [21:34:29] 10Beta-Cluster-Infrastructure: Betacommons is slowing down (5 seconds for a 6KB JS file) - https://phabricator.wikimedia.org/T301841#11191822 (10bd808) 05Openβ†’03Declined It got better at some point, probably years ago. [21:35:42] Some days I wonder if folks think the #Beta-Cluster-Infrastructure tag is just for filing bugs and never following up on them... [21:38:23] 10Beta-Cluster-Infrastructure: Set up an alerting server - https://phabricator.wikimedia.org/T318247#11191842 (10bd808) 05Openβ†’03Declined We are using [[https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring#Monitoring_for_Cloud_VPS|metricsinfra]] for this these days. [21:39:54] 10Beta-Cluster-Infrastructure, 10MediaWiki-Language-converter, 13Patch-Needs-Improvement: (Develop spec for) Test wiki for English language variants - https://phabricator.wikimedia.org/T143628#11191863 (10bd808) [21:40:03] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Wikimedia-Site-requests, 10Sustainability (Incident Followup): Enable pig latin variant on English Wikipedia beta cluster - https://phabricator.wikimedia.org/T299975#11191866 (10bd808) β†’14Duplicate dup:03T143628 [21:42:22] 10Continuous-Integration-Config, 10Math, 10MediaWiki-Platform-Team (Radar), 10MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout) - https://phabricator.wikimedia.org/T389998#11191876 (10hashar) I h... [21:51:33] 10Beta-Cluster-Infrastructure, 10Scap, 06Infrastructure-Foundations, 10Puppet-Core, and 2 others: Define scap::sources in a way that is shared between prod and beta - https://phabricator.wikimedia.org/T196034#11191921 (10bd808) 05Openβ†’03Declined Making the custom `role` function used in production... [22:00:49] 10Phabricator (Search), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Upstream (or revert) custom fulltext search result match preview - https://phabricator.wikimedia.org/T404922 (10Aklapper) 03NEW p:05Triageβ†’03Low [22:06:46] 10Beta-Cluster-Infrastructure, 10Scap, 06Infrastructure-Foundations, 10Puppet-Core, and 2 others: Define scap::sources in a way that is shared between prod and beta - https://phabricator.wikimedia.org/T196034#11191978 (10Dzahn) I believe the following 2 steps should result in the same thing we would ge... [22:46:21] 10Beta-Cluster-Infrastructure, 06SRE, 06Traffic: Rename deployment-cache-(text|upload)0x to deployment-cp0x - https://phabricator.wikimedia.org/T280393#11192124 (10bd808) >>! In T280393#7163610, @taavi wrote: > One more issue: given cloud vps does not have per-role hiera keys, we need to rely on instance pre... [22:49:34] 10Beta-Cluster-Infrastructure, 06SRE, 06Traffic, 13Patch-For-Review: Incorrect X-Cache-Status reported by deployment-prep caches - https://phabricator.wikimedia.org/T269825#11192132 (10bd808) [22:58:47] 10Beta-Cluster-Infrastructure, 06SRE, 07Technical-Debt, 07Tracking-Neverending: Minimize infrastructure differences between Beta Cluster and production - https://phabricator.wikimedia.org/T87220#11192146 (10bd808) 05Openβ†’03Declined Yuvi had a very strong opinion here, but `*All* other differences h... [23:08:02] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Seen), 10MediaWiki-Configuration: Fatal error Cannot redeclare wmfLabsSettings() on Beta cluster wikis - https://phabricator.wikimedia.org/T224899#11192156 (10bd808) 05Openβ†’03Resolved a:03Ladsgroup The whole setup of how operations/mediaw... [23:10:25] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11192160 (... [23:20:57] 10Beta-Cluster-Infrastructure: Investigate cloud VM creation failures for instances with deployment-mediawiki prefix - https://phabricator.wikimedia.org/T232901#11192176 (10bd808) 05Openβ†’03Declined [23:22:00] 10Beta-Cluster-Infrastructure, 07Wikimedia-production-error: Fatal error: Invalid operand type was used: cannot perform this operation with arrays in /srv/mediawiki/php-master/languages/Language.php on line 514 - https://phabricator.wikimedia.org/T233155#11192177 (10bd808) 05Openβ†’03Declined [23:24:24] 10Beta-Cluster-Infrastructure: Beta is serving CSP headers allowing prod rather than beta - https://phabricator.wikimedia.org/T233533#11192196 (10bd808) 05Openβ†’03Resolved ` $ curl -v https://meta.wikimedia.beta.wmcloud.org/wiki/Main_Page 2>&1 | grep content-security-policy: < content-security-policy: scr... [23:29:06] 10Beta-Cluster-Infrastructure, 06Growth-Team, 06Growth-Team-Filtering: cswiki beta has incorrect wgGENewcomerTasksRemoteApiUrl - https://phabricator.wikimedia.org/T238321#11192214 (10bd808) @Urbanecm_WMF is this still something that needs fixing? [23:39:34] 10Beta-Cluster-Infrastructure: Setup monitoring for database servers in beta cluster - https://phabricator.wikimedia.org/T87093#11192220 (10bd808) >>! In T87093#11191820, @Dzahn wrote: > Is this blocked on "existence of monitoring system" (thinking of back when icinga existed in cloud VPS) or is there one that j... [23:42:39] 10Beta-Cluster-Infrastructure, 10observability, 07Tracking-Neverending: Setup monitoring for Beta Cluster (tracking) - https://phabricator.wikimedia.org/T53497#11192225 (10bd808) See {T315695} for some discussion of how we might use https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_managed_monitoring more e... [23:44:41] 10Beta-Cluster-Infrastructure, 07Performance Issue: BackendResponseTime on beta cluster is high when logged in - https://phabricator.wikimedia.org/T323655#11192233 (10bd808) 05Openβ†’03Declined [23:52:23] 10Beta-Cluster-Infrastructure: Setup monitoring for database servers in beta cluster - https://phabricator.wikimedia.org/T87093#11192251 (10Dzahn) I see! thank you. ftr, I was actually thinking that getting the old icinga system back might not be the hardest of all the options. It was already there once, the pu...