[00:41:44] 10Beta-Cluster-Infrastructure, 10Acme-chief, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11183331 (10BCornwall) Hm, it sounded like the directory should have bee... [07:23:13] Expected: -1 [07:23:13] Actual: -1 [07:23:19] Thanks JavaScript! [07:36:04] :cry: [08:55:08] (03CR) 10Hashar: [C:03+2] Zuul: CirrusSearch transitively depends on EventStreamConfig [integration/config] - 10https://gerrit.wikimedia.org/r/1188368 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [08:55:22] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/DrawioEditor] Add Forms dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1188726 [08:56:40] (03Merged) 10jenkins-bot: Zuul: CirrusSearch transitively depends on EventStreamConfig [integration/config] - 10https://gerrit.wikimedia.org/r/1188368 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [08:59:03] 06Project-Admins: Create #mediawiki-actions component - https://phabricator.wikimedia.org/T268386#11183854 (10A_smart_kitten) >>! In T268386#6638452, @Aklapper wrote: > But preferably not under a `#MediaWiki-Actions` umbrella? (I know #MediaWiki-Special-pages exists, we shouldn't repeat that mistake.) I rea... [09:02:09] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1188368 "CirrusSearch transitively depends on EventStreamConfig" [09:02:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:06:29] 06Project-Admins: Create #mediawiki-actions component - https://phabricator.wikimedia.org/T268386#11183878 (10Aklapper) > I realise that I'm replying to a 5-year-old comment, but @aklapper for my own understanding, please could you expand on why you consider it to be a mistake? Too broad/generic and/or not... [09:21:37] 06Project-Admins: Create #mediawiki-actions component - https://phabricator.wikimedia.org/T268386#11183939 (10taavi) In general, developers/teams work (or don't work) on features things like "blocks" or "user preferences", which involve changes across the related special pages, tests, APIs, database schema,... [09:56:28] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#11184115 (10Jelto) My proposal to move forward is to sync the files from object storage to a local folder... [10:33:39] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#11184312 (10jcrespo) If that unblocks you, I am ok with that- sadly because other priorities keep enterin... [12:33:48] (03open) 10jakob: Add missing wikidata RDF dumps dependency "serdi" [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/213 (https://phabricator.wikimedia.org/T403882) [12:43:23] 10Phabricator, 10Release-Engineering-Team (Doing 😎): Phabricator weekly changes email: List active personal projects of disabled user accounts - https://phabricator.wikimedia.org/T404411#11184802 (10Aklapper) 05Open→03Resolved Merged. [13:07:56] (03PS1) 10Hashar: Zuul: [mediawiki/extensions/WikibaseMediaInfo] depends on CirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/1188773 (https://phabricator.wikimedia.org/T389998) [13:11:51] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikibaseMediaInfo] depends on CirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/1188773 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [13:13:17] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikibaseMediaInfo] depends on CirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/1188773 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [13:13:36] thx :) [13:14:56] !log Zuul: [mediawiki/extensions/WikibaseMediaInfo] Add dependencies for CirrusSearch and Elastica, for T389998 [13:14:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:14:59] T389998: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout) - https://phabricator.wikimedia.org/T389998 [13:15:50] hashar: A quick +1/-1 on https://gerrit.wikimedia.org/r/c/integration/config/+/1188388 would be appreciated if you have 5 mins. [13:17:13] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/DrawioEditor] Add Forms dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1188726 (owner: 10Hslater) [13:18:29] (03CR) 10Hashar: [C:03+1] "Yes please go for it. In the child change for Zuul ( I1e4107ecd04089d46fe20b02dba9f0c794b9f6c7 ), there is no jobs removed/added in the d" [integration/config] - 10https://gerrit.wikimedia.org/r/1188388 (owner: 10Jforrester) [13:18:45] <3 [13:18:52] the only limit is that.. [13:18:56] the jobs are going to be too long [13:19:01] the jobs names [13:19:08] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/DrawioEditor] Add Forms dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1188726 (owner: 10Hslater) [13:19:21] Eh, they're mostly ignored by humans anyway, and the tech doesn't care. [13:19:32] !log Zuul: [mediawiki/extensions/DrawioEditor] Add Forms dependency [13:19:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:21:13] (03PS5) 10Jforrester: jjb: Rename the thicket of mediawiki-quibble… jobs to make sense [integration/config] - 10https://gerrit.wikimedia.org/r/1188388 [13:23:18] (03CR) 10Jforrester: [C:03+2] jjb: Rename the thicket of mediawiki-quibble… jobs to make sense [integration/config] - 10https://gerrit.wikimedia.org/r/1188388 (owner: 10Jforrester) [13:24:43] (03Merged) 10jenkins-bot: jjb: Rename the thicket of mediawiki-quibble… jobs to make sense [integration/config] - 10https://gerrit.wikimedia.org/r/1188388 (owner: 10Jforrester) [13:24:54] (03PS3) 10Jforrester: Zuul: Switch confusing mediawiki-quibble… jobs to clearer names [integration/config] - 10https://gerrit.wikimedia.org/r/1188392 [13:29:50] (03CR) 10Jforrester: "(Will deploy when G&S isn't full of lots of commits, as this will reset the CI progress on renamed jobs.)" [integration/config] - 10https://gerrit.wikimedia.org/r/1188392 (owner: 10Jforrester) [13:41:49] !log Manually aborting quibble-vendor-mysql-php83/12997, stalled for > 20 minutes and blocking G&S. [13:41:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:42:46] (03CR) 10Jforrester: [C:03+2] Zuul: Switch confusing mediawiki-quibble… jobs to clearer names [integration/config] - 10https://gerrit.wikimedia.org/r/1188392 (owner: 10Jforrester) [13:44:57] (03Merged) 10jenkins-bot: Zuul: Switch confusing mediawiki-quibble… jobs to clearer names [integration/config] - 10https://gerrit.wikimedia.org/r/1188392 (owner: 10Jforrester) [13:45:13] !log Zuul: Switch confusing mediawiki-quibble… jobs to clearer names [13:45:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:46:01] James_F: I suspect that job you killed was still making progress :/ [13:46:10] it just wasn’t printing it (due to the way the split groups are implemented) [13:46:26] “gzip: log/mw-debug-cli.split-group-4.log: file size changed while zipping” sounds like something was still going on [13:46:41] (though the file contents aren’t timestamped so I can’t say for sure) [13:46:59] Lucas_WMDE: Yeah, there's seemingly a memory leak from the PHPUnit splitting. [13:47:07] There's a couple of tasks about it. [13:47:24] yes, T401204 has some builds that still finished successfully after >40 minutes [13:47:24] T401204: PHPUnit tests randomly time out in CI - https://phabricator.wikimedia.org/T401204 [13:47:40] (03PS3) 10Jforrester: jjb: Drop mediawiki-quibble… jobs, now renamed [integration/config] - 10https://gerrit.wikimedia.org/r/1188393 [13:47:45] (03PS4) 10Jforrester: jjb: Drop mediawiki-quibble… jobs, now renamed [integration/config] - 10https://gerrit.wikimedia.org/r/1188393 [13:47:47] (03CR) 10CI reject: [V:04-1] jjb: Drop mediawiki-quibble… jobs, now renamed [integration/config] - 10https://gerrit.wikimedia.org/r/1188393 (owner: 10Jforrester) [13:47:51] (03PS2) 10Jforrester: jjb: Move quibble-apitests-only-… up to the rest [integration/config] - 10https://gerrit.wikimedia.org/r/1188394 [13:49:00] Anything that causes PHPUnit to not emit a single character for > 20 minutes isn't just "slow", it's broken, even if will eventually complete. [13:49:17] CI isn't intended to get single patches over the line, it's an orchestration system. [13:49:29] 🙄 [13:50:08] and all the *other* patches will now also have to re-run their CI [13:50:11] I'm still not sure I like the hacky PHPUnit suite splitting. [13:50:18] when with 20 more minutes of waiting they may well have all gone through directly [13:50:38] Or with 20 minutes more waiting it'll have timed out, like they normally do in this situation. [13:50:54] Just because sometimes it looks stuck but eventually *does* complete doesn't mean that's the normal pattern. [13:51:04] But it's very irritating. [13:51:22] Hopefully the PHPUnit 10 upgrade can finally be done, and we'll use the proper, supported system. [13:52:20] I’m still curious where the proper, supported system is expected to get a good test distribution for [13:52:28] which seems to be the hardest part about parallelizing the tests [13:52:35] s/for/from/ [13:53:01] I understood that it was a dynamic set of pools that tests would be assigned to as they because free, so slow-running tests would balance out. [13:53:05] But maybe I mis-read? [13:53:26] Rather than static assignment at start-up, which is subject to slow pools. [13:55:32] as long as it still runs tests in order, which IIUC Is one of the most limiting constraints put on the current split implementation [13:55:38] * James_F nods. [13:55:50] Yeah, async running of tests would break a few things. [13:56:00] I'm sure in upgrading we'll find a number of things that need tweaking. [13:58:25] (03PS1) 10Hashar: Zuul: [mediawiki/extensions/WikibaseLexeme] add cldr [integration/config] - 10https://gerrit.wikimedia.org/r/1188787 (https://phabricator.wikimedia.org/T389998) [14:48:54] (03merge) 10dancy: Add missing wikidata RDF dumps dependency "serdi" [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/213 (https://phabricator.wikimedia.org/T403882) (owner: 10jakob) [15:22:19] (03approved) 10jnuche: kubernetes.py: Define DepConfig and use it [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1000 (owner: 10dancy) [16:13:01] 10Release-Engineering-Team (Doing 😎), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on the test Phabricator instance (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11185883 (10Dzahn) reverts do n... [16:15:03] maintenance-disconnect-full-disks build 737597 integration-agent-docker-1061 (/: 24%, /srv: 95%, /var/lib/docker: 28%): OFFLINE due to disk space [16:20:03] maintenance-disconnect-full-disks build 737598 integration-agent-docker-1061 (/: 24%, /srv: 35%, /var/lib/docker: 26%): RECOVERY disk space OK [16:53:53] (03CR) 10Jforrester: [C:03+2] jjb: Drop mediawiki-quibble… jobs, now renamed [integration/config] - 10https://gerrit.wikimedia.org/r/1188393 (owner: 10Jforrester) [16:53:58] (03CR) 10Jforrester: [C:03+2] jjb: Move quibble-apitests-only-… up to the rest [integration/config] - 10https://gerrit.wikimedia.org/r/1188394 (owner: 10Jforrester) [16:54:38] (03PS3) 10Jforrester: Zuul: Adjust Quibble mediawiki mutex to explicitly apply to extensions/skins [integration/config] - 10https://gerrit.wikimedia.org/r/1188391 [16:55:44] (03Merged) 10jenkins-bot: jjb: Drop mediawiki-quibble… jobs, now renamed [integration/config] - 10https://gerrit.wikimedia.org/r/1188393 (owner: 10Jforrester) [16:55:53] (03Merged) 10jenkins-bot: jjb: Move quibble-apitests-only-… up to the rest [integration/config] - 10https://gerrit.wikimedia.org/r/1188394 (owner: 10Jforrester) [17:42:51] (03update) 10dancy: kubernetes.py: Define DepConfig and use it [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1000 [17:43:04] (03merge) 10dancy: kubernetes.py: Define DepConfig and use it [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1000 [18:24:32] (03update) 10dancy: kubernetes.py: DeploymentsConfig.parse(): Handle multiple clusters [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/999 (https://phabricator.wikimedia.org/T402350) [18:28:45] (03update) 10dancy: kubernetes.py: DeploymentsConfig.parse(): Handle multiple clusters [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/999 (https://phabricator.wikimedia.org/T402350) [18:31:44] (03update) 10dancy: kubernetes.py: DeploymentsConfig.parse(): Handle multiple clusters [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/999 (https://phabricator.wikimedia.org/T402350) [18:33:36] (03update) 10dancy: Set up traindev-staging cluster [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/160 (https://phabricator.wikimedia.org/T402350) [18:34:37] (03merge) 10dancy: Set up traindev-staging cluster [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/160 (https://phabricator.wikimedia.org/T402350) [19:15:56] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Remove custom Wikimedia xhpast Linter? - https://phabricator.wikimedia.org/T404760 (10Aklapper) 03NEW [19:37:44] 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Merge Phorge's upstream master (2025-09-08) into our wmf/stable - https://phabricator.wikimedia.org/T404134#11186734 (10Aklapper) The search failure on the test instance might be related to accidentally removing a custom downstream change in `src... [19:38:19] brennen: Hey. Does https://phabricator.wikimedia.org/T404134#11186734 make sense to you? [19:38:28] * brennen looks [19:38:54] andre: yeah, seems totally possible [19:39:18] i wouldn't be real surprised if i botched something in the merge [19:39:22] I still need to dig into the rest of rebase conflicts but the search currently worries me most [19:39:43] hehe, happens. We should just strip more downstream customizations I guess [19:39:51] it occurs to me we should maintain some kind of meta list of our divergences from upstream [19:40:18] just something we can reference when we do a merge, that kind of thing [19:40:23] that's... still gonna be a long list :( [19:40:34] yeha [19:40:36] 'yeah [19:40:55] (03CR) 10Hashar: [C:03+2] "cldr is already injected, this is just making it explicit." [integration/config] - 10https://gerrit.wikimedia.org/r/1188787 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [19:41:24] now I need to find out how to update your GitLab patch. Probably check out that branch and push additional commits. Sigh, dealing with three different code review systems... [19:42:26] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikibaseLexeme] add cldr [integration/config] - 10https://gerrit.wikimedia.org/r/1188787 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [19:43:20] (03PS1) 10Hashar: Zuul: explicitly add MobileFrontend with Echo [integration/config] - 10https://gerrit.wikimedia.org/r/1188874 (https://phabricator.wikimedia.org/T389998) [19:43:47] i think i can probably re-do the merge with that change [19:46:22] brennen: if you have some spare brain capacity (mine is somehow limited today)... If not I'll probably take a shot myself tomorrow when I'm more fresh [19:57:54] 10Phabricator, 10SRE-Access-Requests: Phabricator admin access for new team members - https://phabricator.wikimedia.org/T404768 (10BTracy-WMF) 03NEW [20:12:03] 10Phabricator, 10SRE-Access-Requests: Phabricator admin access for new team members - https://phabricator.wikimedia.org/T404768#11186979 (10Aklapper) 05Open→03Declined Hi, this does not require admin access. Please see https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects and bring... [20:12:29] 10Phabricator: Phabricator admin access for new team members - https://phabricator.wikimedia.org/T404768#11186986 (10Aklapper) [20:43:32] (03PS1) 10Hashar: Zuul: explicitly add CodeEditor with AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/1188880 (https://phabricator.wikimedia.org/T389998) [20:52:03] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Phab Scap template vs actual value out of sync for maniphest.custom-field-definitions - https://phabricator.wikimedia.org/T398460#11187247 (10A_smart_kitten) [21:02:21] 10Continuous-Integration-Config, 10Math, 10MediaWiki-Platform-Team (Radar), 10MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout) - https://phabricator.wikimedia.org/T389998#11187340 (10hashar) [21:14:34] 10Release-Engineering-Team (Doing 😎), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11187376... [21:17:39] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11187381 (... [21:19:16] 10Phabricator (Upstream), 10Release-Engineering-Team (Doing 😎), 07Upstream: Phabricator manifest edit API failing with a 200 response - https://phabricator.wikimedia.org/T404214#11187392 (10Aklapper) [21:19:27] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Merge Phorge's upstream master (2025-09-08) into our wmf/stable - https://phabricator.wikimedia.org/T404134#11187394 (10Aklapper) [21:19:57] 10Phabricator, 10Release-Engineering-Team (Radar), 06Traffic: Phabricator videos fail in Firefox ("Range" request gets 503 from Varnish) - https://phabricator.wikimedia.org/T397661#11187396 (10Aklapper) [21:20:36] 06Release-Engineering-Team, 10MediaWiki-extensions-PlaceNewSection, 10Wikimedia-Extension-setup, 07Russian-Sites, 07Wikimedia-extension-review-queue: Add Extension:PlaceNewSection to Russian Wikipedia - https://phabricator.wikimedia.org/T344501#11187397 (10Ernstkm) [21:20:37] 10Phabricator (Search), 10Release-Engineering-Team (Seen): Autocomplete project proposals in Maniphest search form don't always offer "any" and "not" options - https://phabricator.wikimedia.org/T360701#11187399 (10Aklapper) [21:29:15] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Ruvchoudhary - https://phabricator.wikimedia.org/T404484#11187475 (10Aklapper) @Ruvchoudhary: Hi, what's your involvement with the Wikimedia movement so far? And which specific projects in Wikimedia GitLab do you... [21:39:11] (03open) 10egardner: make-release: Start branching ReaderExperiments [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/214 (https://phabricator.wikimedia.org/T404398) [23:09:36] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11187713 (... [23:14:08] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11187729 (... [23:18:28] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11187733 (... [23:26:58] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11187746 (... [23:31:12] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11187755 (... [23:32:27] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11187773 (... [23:34:45] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11187774 (... [23:38:33] 10Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11187791 (...