[07:45:00] (03PS2) 10Hashar: jjb: [quibble*] Drop worst-case timeout to 30 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/1130326 (https://phabricator.wikimedia.org/T389536) (owner: 10Jforrester) [08:00:17] (03CR) 10Hashar: [C:04-1] "I have looked at the `*quibble*` jobs builds duration and lot of them took more than 30 minutes. I guess that should be investigated a bit" [integration/config] - 10https://gerrit.wikimedia.org/r/1130326 (https://phabricator.wikimedia.org/T389536) (owner: 10Jforrester) [08:01:13] (03PS2) 10Hashar: jjb: [mwcore-phpunit-coverage-patch] lower timeout from 60 to 15 mins [integration/config] - 10https://gerrit.wikimedia.org/r/1130325 (owner: 10Jforrester) [08:08:53] (03CR) 10Hashar: "There are some builds that took longer than 15 minutes / 900 seconds or nearing it. In seconds:" [integration/config] - 10https://gerrit.wikimedia.org/r/1130325 (owner: 10Jforrester) [08:09:41] (03PS3) 10Hashar: jjb: [quibble*] lower timeout from 60 to 30 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/1130326 (https://phabricator.wikimedia.org/T389536) (owner: 10Jforrester) [08:26:32] hashar: seeing some flakiness last 24 hours, not sure what to do about it [08:26:52] https://gerrit.wikimedia.org/r/1130284 ran out of memory, which suggests maybe there is an inbalance between what memory we give to workers and how many builds we do on the same VM. [08:28:03] and phpunit on Math extension seems to time out 2-3 times across 10-15 patches. each time it was not stuck but the time spent very spread out, suggesting it was very slow in general, which perhaps means too many executors on the same host, insufficnet cpu? e.g. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/1130284/1 [08:29:18] note that this is quibble with phpunit timout out after 60 minutes: [08:29:20] https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php82-noselenium/7602/console : SUCCESS in 43m 05s [08:29:20] https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php83-noselenium/5902/console : SUCCESS in 59m 51s [08:29:32] https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php81-noselenium/9914/console : FAILURE in 1h 00m 25s [08:30:14] I've been trying to land this stack for 3 days... [09:26:45] and again... [09:26:46] https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php81-noselenium/9918/console : FAILURE in 1h 00m 30s [10:14:14] These patches are on REL1_43 and I wonder if for some reason the parallel runs for PHPUnit tests are not running there? The examples you've linked all run the non-DB integration tests and DB integration tests without splitting them. [10:15:55] The parallel splitting code wasn't backported to REL1_43 [10:16:04] Yeah, good point. [10:16:27] Perhaps a combination of slow tests and no parallel runs is causing this? [10:21:22] Also there were some additional Zuul dependencies added on Sunday that could mean more tests are running in Math builds? [10:22:51] Krinkle: tests flappiness has been talked about last week. That involved Selenium tests though [10:24:12] for an out of memory issues I'would go to the Grafana cloud dashboard https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?orgId=1&var-project=integration&var-instance=All [10:24:49] but if they take 59mn/1h00 that is definitely unreasonable [10:25:39] Project beta-scap-sync-world build #199217: 04FAILURE in 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/199217/ [10:28:50] for https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php81-noselenium/9914/consoleFull [10:28:54] it ran on the agent 1051 [10:29:00] the grafana view is https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?orgId=1&var-project=integration&var-instance=integration-agent-docker-1051&from=1742800800000&to=1742804520000 [10:31:49] there is pressure on the CPU for the first few minutes. Quibble runs bunch of steps until 7:31 when it runs `phpunit '--testsuite' 'extensions' '--group' 'Database'` [10:32:05] and get aborted at 8:20 [10:33:03] during all that time the CPU usage is at ~ 24% (two cpus at 100%)? [10:34:38] anyway, it somehow depends on GrowthExperiments or Wikibase [10:34:56] and our phpunit extensions test suite blindly runs every single tests [10:35:51] Project beta-scap-sync-world build #199218: 04STILL FAILING in 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/199218/ [10:41:00] Amir1: deployment-prep broke: [10:41:00] 10:35:51 mergeMessageFileList.php generated PHP notices/warnings: [10:41:00] 10:35:51 Notice: Undefined variable: wmgMainStashServers in /srv/mediawiki-staging/wmf-config/CommonSettings.php on line 679 [10:41:12] ah, shoot [10:41:16] let me fix it [10:41:25] :) [10:41:58] (btw check main page of English wikipedia and the images there, sizes and the widths) [10:42:33] have you finally standardized the thumbnails sizes and let the browser do the resize? :) [10:43:30] 35% deployed [10:43:37] (5% every day) [10:43:49] \o/ [10:43:52] Also bumping the default size: 220px -> 250 [10:45:28] 06Release-Engineering-Team, 10MW-on-K8s, 06serviceops: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10667103 (10BTullis) I'd just like to pick up on something that was mentioned in the description: > cli (used by mw-cron and mw-s... [10:45:48] Project beta-scap-sync-world build #199219: 04STILL FAILING in 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/199219/ [10:53:43] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1130545 [10:53:45] This should fix it [10:54:50] will it be merged in time? let's see 🍿 [10:55:47] Project beta-scap-sync-world build #199220: 04STILL FAILING in 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/199220/ [10:55:57] :( [10:56:37] I forced a rebuild [10:57:07] Project beta-scap-sync-world build #199221: 04STILL FAILING in 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/199221/ [10:57:51] sigh. Maybe it's the file load order [11:03:52] it was the file load order but I'm not sure IS-labs.php will be before it [11:18:13] Yippee, build fixed! [11:18:14] Project beta-scap-sync-world build #199222: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/199222/ [11:19:07] awesome [11:49:55] Amir1: thank you for the fix! [11:50:09] it's a bit ugly [11:56:10] 10Deployments, 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, and 5 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10667501 (10ArthurTaylor) [11:56:18] 10Deployments, 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, and 5 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10667503 (10ArthurTaylor) a:03ArthurTaylor [12:13:57] 06Project-Admins, 03WMA-Hackathon-2025: Create project tag for Wiki-LeaderBoard - https://phabricator.wikimedia.org/T389714#10667543 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Requested public project #LeaderBoard has been created: https://phabricator.wikimedia.org/project/view/7804/ (In case you n... [12:13:59] 06Project-Admins, 03WMA-Hackathon-2025: Create project tag for Wdqsai - https://phabricator.wikimedia.org/T389715#10667546 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Requested public project #wdqsai has been created: https://phabricator.wikimedia.org/project/view/7805/ (In case you need to edit the... [12:26:14] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Wikimedia-Phabricator-Extensions: Remove custom "Expert Mode" - https://phabricator.wikimedia.org/T351289#10667625 (10Jdforrester-WMF) I'm really sad to lose the controls this gives me via Parent ID searches. Can this proposal be discussed fir... [12:28:19] (03CR) 10Jforrester: "Yes, I think it's reasonable for these lower-value jobs to not be allowed to run to completion in pathalogical cases like that." [integration/config] - 10https://gerrit.wikimedia.org/r/1130325 (owner: 10Jforrester) [12:31:28] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, 06Traffic: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786#10667658 (10Clement_Goubert) 05Openβ†’03In progress [13:04:34] 10Deployments, 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, and 5 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10667800 (10ArthurTaylor) a:05ArthurTaylorβ†’03None [14:01:09] 06Release-Engineering-Team, 10Projects-Cleanup, 10Wikimedia-GitHub: Archive unused limn-*-data Gerrit repositories and delete their Github mirrors - https://phabricator.wikimedia.org/T372412#10668174 (10Milimetric) @Aklapper: I'm 100% ok with deleting these. Nothing of value will be found in them, and they'... [14:05:03] maintenance-disconnect-full-disks build 686892 integration-agent-docker-1056 (/: 25%, /srv: 98%, /var/lib/docker: 66%): OFFLINE due to disk space [14:10:03] maintenance-disconnect-full-disks build 686893 integration-agent-docker-1056 (/: 25%, /srv: 36%, /var/lib/docker: 66%): RECOVERY disk space OK [14:28:12] 10Gerrit, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 13Patch-For-Review: Remove explicit enablement of G1 garbage collector for Gerrit - https://phabricator.wikimedia.org/T387223#10668442 (10hashar) We set `G1NewSizePercent` to `15` (from a default of `5`) which, with 32G of... [14:37:40] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Consider disabling personal access token forced expiration - https://phabricator.wikimedia.org/T385930#10668501 (10Jelto) >>! In T385930#10661714, @LSobanski wrote: > Same thing for gi... [14:47:46] (03PS1) 10Kevin Bazira: inference-services: trigger article-country CI on python dir change [integration/config] - 10https://gerrit.wikimedia.org/r/1130627 (https://phabricator.wikimedia.org/T382295) [15:23:23] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10WMF-JobQueue, 05Account-Vanishing, and 4 others: "JobQueueError: Could not enqueue jobs" when trying to request a global vanish on Beta Cluster - https://phabricator.wikimedia.org/T389585#10668898 (10pmiazga) [15:40:12] 14RelEng-Archive-FY201718-Q1, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-Campaigns, 10MediaWiki-extensions-General: Update Campaigns to use AuthManager - https://phabricator.wikimedia.org/T135043#10669078 (10Reedy) [16:14:48] (03CR) 10Hashar: [C:03+1] "Looks good CI wise but I imagine @achou@wikimedia.org might want to have a look at it as well." [integration/config] - 10https://gerrit.wikimedia.org/r/1130627 (https://phabricator.wikimedia.org/T382295) (owner: 10Kevin Bazira) [16:27:23] 10GitLab (CI & Job Runners), 13Patch-For-Review: Reggie sometimes gets evicted and restarted - https://phabricator.wikimedia.org/T386115#10669512 (10dancy) [16:46:01] 10GitLab (CI & Job Runners), 13Patch-For-Review: Reggie sometimes gets evicted and restarted - https://phabricator.wikimedia.org/T386115#10669672 (10dancy) 05Openβ†’03Resolved [16:51:36] 10Scap: Scap seemingly doesn't fully/properly clean backup deployment server - https://phabricator.wikimedia.org/T389830 (10Reedy) 03NEW [16:52:33] (03CR) 10AikoChou: [C:03+1] "LGTM!" [integration/config] - 10https://gerrit.wikimedia.org/r/1130627 (https://phabricator.wikimedia.org/T382295) (owner: 10Kevin Bazira) [16:53:53] 10Scap: Scap seemingly doesn't fully/properly clean backup deployment server - https://phabricator.wikimedia.org/T389830#10669821 (10hashar) I am pretty sure that is because when syncing the spare deployment servers, we exclude the CDB files and thus they are left behind when we rsync mediawiki-staging. Scap ha... [16:54:44] (03CR) 10Hashar: [C:03+2] inference-services: trigger article-country CI on python dir change [integration/config] - 10https://gerrit.wikimedia.org/r/1130627 (https://phabricator.wikimedia.org/T382295) (owner: 10Kevin Bazira) [16:56:00] (03Merged) 10jenkins-bot: inference-services: trigger article-country CI on python dir change [integration/config] - 10https://gerrit.wikimedia.org/r/1130627 (https://phabricator.wikimedia.org/T382295) (owner: 10Kevin Bazira) [16:58:29] 06Release-Engineering-Team: Clean up old php- dirs on deploy1003 - https://phabricator.wikimedia.org/T389833 (10Reedy) 03NEW [16:58:42] 06Release-Engineering-Team: Clean up old php- dirs on deploy1003 - https://phabricator.wikimedia.org/T389833#10669869 (10Reedy) [16:58:43] 10Scap: Scap seemingly doesn't fully/properly clean backup deployment server - https://phabricator.wikimedia.org/T389830#10669870 (10Reedy) [16:58:48] 06Release-Engineering-Team: Clean up old php- dirs on deploy1003 - https://phabricator.wikimedia.org/T389833#10669875 (10Reedy) p:05Triageβ†’03High [17:00:21] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Enable browser notifications system in Gerrit - https://phabricator.wikimedia.org/T389327#10669879 (10hashar) a:03hashar [17:00:25] 06Release-Engineering-Team: Clean up old php- dirs on deploy1003 - https://phabricator.wikimedia.org/T389833#10669881 (10Reedy) [17:00:35] 06Release-Engineering-Team: Clean up old php- dirs on deploy1003 - https://phabricator.wikimedia.org/T389833#10669887 (10Clement_Goubert) deploy1003 is the current deployment server fwiw. [17:01:13] 06Release-Engineering-Team: Clean up old php- dirs on deploy1003 - https://phabricator.wikimedia.org/T389833#10669893 (10Reedy) >>! In T389833#10669887, @Clement_Goubert wrote: > deploy1003 is the current deployment server fwiw. Yeah, {T389830} was meaning when it *was* the backup server etc. [17:01:33] (03CR) 10Hashar: [C:03+2] "I have deployed the config change!" [integration/config] - 10https://gerrit.wikimedia.org/r/1130627 (https://phabricator.wikimedia.org/T382295) (owner: 10Kevin Bazira) [17:13:33] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 07Browser-Tests, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10669969 (10Daimona) [17:16:27] 10Scap: Scap seemingly doesn't fully/properly clean backup deployment server - https://phabricator.wikimedia.org/T389830#10669983 (10dancy) `scap clean` has a `self._clean_remote_caches(target_hosts)` step which performs the required action (against `/srv/mediawiki`), but it only applies to hosts in the `mediawi... [17:29:44] (03CR) 10Hashar: [C:03+2] Fix logging exception when using --resolve-requires (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1129251 (owner: 10Hashar) [17:36:50] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Enable browser notifications system in Gerrit - https://phabricator.wikimedia.org/T389327#10670189 (10Dzahn) If Gerrit was to move behind the CDN in the future this would be something to keep in mind because presumably this... [17:39:53] hashar: Regarding https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php81-noselenium/9914/consoleFull - I'm not sure we're seeing the same thing. It sounds like you're seeing it run "fine" until 7:31 then run phpunit and then nothing and then killed at 8:20? [17:40:01] https://usercontent.irccloud-cdn.com/file/B0KaUgdG/Screenshot%202025-03-24%20at%2010.38.36.png https://usercontent.irccloud-cdn.com/file/tXll9lys/Screenshot%202025-03-24%20at%2010.38.47.png [17:40:10] I'm seeing it run contininously with output every few seconds. [17:40:26] It just takes over 60 minutes total. No obvious bottleneck within the job. [17:40:42] But it passes is much less than that most of the time on the master branch [17:40:49] but when I was trying to land 15 patches at once it all timed out [17:40:55] which suggests some kind of scheduling problem. [17:41:03] it shouldn't run jobs if it doesn't have cpus available. [17:41:12] that is for Math isn't it? [17:41:28] yep [17:41:41] the build I looked at this morning had plenty of CPU time available [17:41:58] so maybe it is because Math has some dependencies that have been added ot it [17:43:01] hmm Well it only directly depends on VisualEditor and Wikibase [17:43:25] master branch: [17:43:27] https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php81-noselenium/26297/console : SUCCESS in 12m 42s [17:43:27] https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php82-noselenium/26069/console : SUCCESS in 12m 14s [17:43:37] REL1_43: [17:43:40] AH [17:43:41] https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php81-noselenium/9904/console : SUCCESS in 58m 34s [17:43:41] https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php82-noselenium/7594/console : SUCCESS in 1m 07s [17:43:45] same patch, same repo [17:44:30] other REL1_43 patch: [17:44:31] https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php81-noselenium/9882/console : SUCCESS in 39m 17s [17:44:33] half as much [17:44:55] that is because the PHPUnit parallelism hasn't been backported to REL1_43 [17:45:23] another REL1_43 patch: [17:45:24] https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php81-noselenium/9883/console : SUCCESS in 58s [17:45:29] less than a minute [17:45:37] some very strange things [17:45:51] and since Math depends on Wikibase and VisualEditor, it ends up bringing in a looooonnnngg list of dependencies [17:45:53] namely: [17:45:53] mediawiki/extensions/AbuseFilter\nmediawiki/extensions/AntiSpoof\nmediawiki/extensions/ArticlePlaceholder\nmediawiki/extensions/BetaFeatures\nmediawiki/extensions/CentralAuth\nmediawiki/extensions/CheckUser\nmediawiki/extensions/CirrusSearch\nmediawiki/extensions/Cite\nmediawiki/extensions/CodeEditor\nmediawiki/extensions/CommunityConfiguration\nmediawiki/extensions/CommunityConfigurationExample\nmediawiki/extensions/ConfirmEdit\nmediawi [17:45:53] ki/extensions/DiscussionTools\nmediawiki/extensions/Echo\nmediawiki/extensions/Elastica\nmediawiki/extensions/EventBus\nmediawiki/extensions/EventLogging\nmediawiki/extensions/EventStreamConfig\nmediawiki/extensions/FlaggedRevs\nmediawiki/extensions/Flow\nmediawiki/extensions/Gadgets\nmediawiki/extensions/GeoData\nmediawiki/extensions/GlobalBlocking\nmediawiki/extensions/GlobalPreferences\nmediawiki/extensions/Graph\nmediawiki/extensions [17:45:53] /GrowthExperiments\nmediawiki/extensions/GuidedTour\nmediawiki/extensions/IPInfo\nmediawiki/extensions/JsonConfig\nmediawiki/extensions/Kartographer\nmediawiki/extensions/Linter\nmediawiki/extensions/MobileApp\nmediawiki/extensions/MobileFrontend\nmediawiki/extensions/PageImages\nmediawiki/extensions/PageViewInfo\nmediawiki/extensions/ParserFunctions\nmediawiki/extensions/PdfHandler\nmediawiki/extensions/Popups\nmediawiki/extensions/Prop [17:45:53] ertySuggester\nmediawiki/extensions/Renameuser\nmediawiki/extensions/Scribunto\nmediawiki/extensions/SecurePoll\nmediawiki/extensions/SiteMatrix\nmediawiki/extensions/SpamBlacklist\nmediawiki/extensions/SyntaxHighlight_GeSHi\nmediawiki/extensions/TemplateData\nmediawiki/extensions/TextExtracts\nmediawiki/extensions/Thanks\nmediawiki/extensions/TimedMediaHandler\nmediawiki/extensions/TorBlock\nmediawiki/extensions/UniversalLanguageSelecto [17:45:54] r\nmediawiki/extensions/VisualEditor\nmediawiki/extensions/WikiEditor\nmediawiki/extensions/Wikibase\nmediawiki/extensions/WikibaseCirrusSearch\nmediawiki/extensions/WikibaseLexeme\nmediawiki/extensions/WikibaseLexemeCirrusSearch\nmediawiki/extensions/WikibaseMediaInfo\nmediawiki/extensions/WikibaseQualityConstraints\nmediawiki/extensions/WikimediaBadges\nmediawiki/extensions/WikimediaEvents\nmediawiki/extensions/WikimediaMessages\nmedia [17:45:54] wiki/extensions/cldr [17:45:59] (03Merged) 10jenkins-bot: Fix logging exception when using --resolve-requires [integration/quibble] - 10https://gerrit.wikimedia.org/r/1129251 (owner: 10Hashar) [17:46:01] oh f*** that was long [17:46:01] > 00:00:51.563 INFO:quibble.commands:Success cache: HIT [17:46:10] OK, that's makes sense at least, the result cache [17:46:30] sure, but that would happen on master too, and any patch in core with wmf gate has all that [17:46:32] Math is part of tarball [17:46:37] this isn't about extension dependencies [17:47:09] or do we exclude Wikibase/VE outside Math as some kind of optional dependency? [17:47:10] the result cache is an optimiazation that got discovered/invented by dduvall . So that the same set of commits + docker image do not need to be retested . That speeds up backports to the wmf branches [17:47:40] OK, so we can ignore the <1min runs [17:47:58] still strang that it varies between 12min, 30min and >60min for the same code more or less [17:48:05] cause it runs differently [17:48:12] on REL1_43 the phpunit tests are run serially [17:48:29] and given that runs the test of dozens and dozen of extensions, for sure it ends up taking an hour /timing out [17:49:08] our ci did not run that long before we enabled parallel, though. [17:49:18] that has been THE major issue for slowness but I don't quite know how to fix it or at least all those years I had not found a compelling idea to fix that [17:49:39] running tests in parallel improves things, but hides the fact we simply dumbly run everything [17:50:14] we solved thsi with group standalone though, right? [17:50:26] for anything slow that doesn't need downstream integration. [17:50:30] for SOME cases such as the Scribunto integration tests [17:50:53] but by default tests are not flagged as such and are thus included in the UnitTestList hook [17:50:54] Oct 2024: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/1083865 [17:50:57] It was 48 minutes there [17:51:07] Okay, so it has definitely been fairly high for a while in REL1_43. [17:52:20] yup [17:52:24] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Testing Support, 07Browser-Tests, and 3 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10670267 (10zeljkofilipin) [17:52:27] and maybe some more dependencies got added since that time [17:53:18] what we would need is a mechanism to filter the list of tests in UnitTestList [17:53:42] so that if one sends a patch to Math, we would need to extract the set of tests that got registered by Math and run those [17:53:51] if any fails, stop there [17:54:30] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Testing Support, 07Browser-Tests, and 4 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10670274 (10zeljkofilipin) [17:55:21] then for the other repos, find whether they have an integration tests for `Math` and only run those [17:56:23] Daniel also filed a task saying the dependencies should be tested both way [17:56:51] I see a different issue as well, we're mixing up integration targets and dependencies [17:57:07] so that a patch to `Wikibase` should also run the `Math` tests marked as integration tests with Wikibsae [17:57:09] yeah [17:57:12] DiscussionTools and Cite provide plugins to VisualEditor so VisualEditor "depends" on them in CI to include those. [17:57:32] But when Math runs, it depends on "VE" to run those, but we do not need DiscussionTools/Cite as well. [17:57:42] They are not tarnsitive, these are repo-specific instructions. [17:57:58] VE isn't that big in terms of CI [17:58:08] Math is spending most of its time running DiscussionTools tests [17:58:44] I thought the yaml dependencies in CI declared what extensions patches to a given repo should install [17:58:49] ie. flat, not recursive. [17:58:54] it is flat for phan [17:59:08] but for wfLoadExtension() the yaml entries are processed recursively [17:59:53] else the MediaWiki installer ends up bailing out cause of a missing requirement (which might be needed at installation time) [18:00:53] and since all of the extension are loaded, they inject the tests in UnitTestsList (or our PHPUnit suite discovers the tests in `*/tests/phpunit`, I can't remember) and every run [18:03:49] the installer won't bail because they are not actually dependencies [18:03:59] of course if the yaml is missing hard dependencies, that should be fixed. [18:04:31] Math/extension.json: "requires": { [18:04:31] "MediaWiki": ">= 1.43" [18:04:31] }, [18:05:15] Math pathces should integrate with Wikibase and VE, that's a good thing. It doees not need to integrate with things that VE patches or Wikibase patches want to integrate with. [18:05:20] ah yeah correct, I stand corrected [18:06:06] maybe if we change this we don[t need separate phan list [18:07:02] there are some side effects though such as WikibaseCirrusSearch that uses CirrusSearch which itselfs requires Elastica [18:07:13] but I suppose it might be useful to have even more stuff in Phan, so that it can statically analyze code that depends on an optional dependency. However, i'm not sure. Becuase if that extension is not listed in th regular one, it won't even be executed in phpunit, not even within the same repo. [18:07:28] yeah, we list that out already for WikibaseCirrusSearch [18:07:51] extensions generally have very few deps, they are manually installed after all, so the wiki docs also have a flat list of how to install them [18:12:09] hashar: hm.. so as a first step, what if we substitute in a patch the current expansion of recusion and then remov ethe runtime recursion. so a no-op today. Then we can start trimming it down after that. [18:13:06] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10Quibble: "should mark the revert as a bot edit" test flaky with Apache + php-fpm - https://phabricator.wikimedia.org/T298514#10670398 (10zeljkofilipin) [18:13:07] for Math I need to remove DiscussionTools (via VE) and remove a dozen unrelated extensions (via Wikibase). [18:13:21] othewise MW 1.43 is broken/unmergable [18:14:27] We've had to do this many times in the past [18:14:50] we can then also later make it so that phan_deps is optional, i.e. if it has no key there, use the default. In most cases it won't need an override. [18:15:13] Reedy: yeah, we usually fix it by disabling the whole jobs [18:15:58] I'd like to avoid that and instead improve things more generally by not doing this recursion, I don't think this was really thought through. It came about as a way to more efficiency encode dependencies, but then we started using the gate for more downstream integraitno which is usefu, but that has a clear uninteded effect that we can actually fix. [18:16:03] I guess that will work if the `extension.json` do not have a requires [18:16:33] https://github.com/wikimedia/integration-config/commit/d5d6578123cada4acddbe95ff089b9776f369c17#diff-33fb327b586d32b6d8adb6c99318b9378c621f0663a81ce24d66f3e2005bdc9bR656-R667 [18:17:07] ah I see, I hadn't seen those kind of exclusions before [18:17:48] hashar: My proposal is to substitute it once locally today, so remove runtime recursion. Eveyrthing behaves the same as yesterday. Then after that we can remove stuff we don't want since the list will be flat. [18:18:26] the recursive processing comes from 2015 and the introduction of the dependencies: https://gerrit.wikimedia.org/r/c/integration/config/+/207132/3/zuul/ext_dependencies.py [18:18:31] I've had to do it many times, especially for LTS [18:19:54] Krinkle: I don't know what you mean by "substitute it once locally today" [18:20:22] hashar: I will edit config.git to flatten the list (one-off script) and remove runtime recursion. [18:20:41] so I turn A:-B -C, B:-X into A:-B-C-X, B:-X [18:21:20] I don't know [18:21:36] hashar: Thanks/exension.json depends only on Echo not Flow. Echo depends on nothing. I think that was wrong from the beginning. ext-Thanks wants to run with Echo and Flow installed for integration. Flow wants various optional features enabled. I don't think those desires are transitive at all. [18:21:38] I see what you mean, but that is going to be a pain to manage [18:21:48] and I'd rather not rush it [18:22:26] 06Release-Engineering-Team, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10670449 (10BTullis) As mentioned, this is our effort to integrate scap with our cluster/namespace {T389786} [18:22:50] Dependencies don't change much at all. Adding a dependency is not trivial. Thes are not npm packages. Doing so is a breaking change for an extension. It's very rare. 99% of the informatin here is local, it is about extension X wanting to test with Y and Z. Every single time someone edits this list, theyh are not thinking about recursion. [18:23:39] when Wikibase devs add something to this list, I can guruantee you they are not saying "other people integraint with us should also integrate with these". [18:24:11] Yes definitely [18:24:30] I love our bi-direction integration and gates. This is just about transitive applying of "extra" integrations on a per-repo basis. [18:25:02] I don't think those are meant to be transitive, that was a mistake. That mistake was very small when we just used it to micro optimize the CI config the first time. [18:25:55] well maybe we can "just" disable the recursive processing [18:26:34] then list each of the extensions keys in zuul/dependencies.yaml and see whether their extension.json has any extra requires [18:26:49] potentially start with skins? That is a smaller set to deal with [18:27:26] I am off it is dinner time here [18:27:43] but looks like at least the recursive processing of dependencies can be removed [18:27:49] I see, so you want to make a bigger change than me. I was suggesting we disable recursion and start by listing the indirect ones directly so that nothing changes. Low risk change. [18:28:14] yeah there are multiple ways to break things :b [18:28:38] If I understand you, you want to start by taking the list as written to be the flat list, and then add anything critical. That would win more indeed. [18:29:03] 10Continuous-Integration-Infrastructure, 10Testing Support, 07Browser-Tests, 10ci-test-error (WMF-deployed Build Failure), and 3 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10670514 (10zeljkofilipin) [18:29:12] but might break a few things which I dont' want to deal with. I'd rather start as today, and then widdle down what I know I can remove, and others who care can reduce their own. [18:29:44] I can write a script that yaml dumps the expanded version today, check that in, and remove recursion. That's what I meant by substituting. [18:30:23] yeah I understood that [18:30:46] I guess you can have the result substituted / expanded list to keep at the top the current direct dependencies [18:30:57] 10Beta-Cluster-Infrastructure, 10Testing Support: Run Selenium tests targeting Beta cluster - https://phabricator.wikimedia.org/T373680#10670537 (10zeljkofilipin) [18:31:07] ah yeah, I can add a comment or line break to make it easier [18:31:10] then follow with a comment "# below are what used to be transitive dependencies" [18:31:12] and the rest [18:31:14] sure [18:31:32] I'll try to do that by end of today, and if it takes more than 1h, I will add "if REL1_43" hacks instead [18:31:45] Math: [18:31:45] - VisualEditor [18:31:45] - Wikibase [18:31:45] # Rest of stuff: [18:31:45] - Flow [18:31:45] ... [18:32:30] 10Continuous-Integration-Config, 10Testing Support: Jenkins selenium job should fail when all tests are skipped - https://phabricator.wikimedia.org/T324480#10670572 (10zeljkofilipin) [18:32:34] and if the transitive dependencies are not in the `requires` we most certainly don't need them [18:32:38] anyway dinner [18:32:58] also James_F might be interested [18:32:58] 10Continuous-Integration-Config, 10Quibble: "should mark the revert as a bot edit" test flaky with Apache + php-fpm - https://phabricator.wikimedia.org/T298514#10670577 (10zeljkofilipin) [18:33:03] * hashar away [18:34:14] In meeting now, but will look later. [18:34:15] 10Continuous-Integration-Config, 10Testing Support, 13Patch-For-Review: Make MediaWiki Wdio tests less slow (Sept 2019) - https://phabricator.wikimedia.org/T234002#10670600 (10zeljkofilipin) [18:35:19] 10Continuous-Integration-Config, 06Machine-Learning-Team, 10ORES, 10Testing Support, 10VisualEditor: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939#10670611 (10zeljkofilipin) [18:39:58] hashar: James_F well, we often add more here because teams actively want and benefit from deeper integration. There is a reason we don't just automate this with extension.json, because we want soft and optional dependencies as well as bi-directional coverage of things we hook into and such. There is a lot of useful infromation here. [18:40:34] But yeah, those decisions are local to the repo, not transitive. so we need to decouple the part that is transitive (i.e. hard deps A>B>C) from the "I want extra integration with X Y Z". [18:40:36] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: CI jobs failing with various timeouts (March 2025) - https://phabricator.wikimedia.org/T388416#10670691 (10Daimona) [18:46:30] 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.22; 2025-03-25), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10670770 (10zeljkofil... [18:49:16] Krinkle: "There is a reason we don't just automate this with extension.json" is mostly "because we haven't built it yet", with the separation in mind. An alternative would be to have a `suggests` key in extension.json for CI 'bonus' extensions, but this needs properly writing out in terms of what we want and how we might do it. [18:49:39] ack, I wouldn't mind moving this data in-repo at some point, for sure. [18:51:58] suggest is a bit different though, that's conveying that the host extension has more to offer if you install X. E.g. akin to how MW might do more if you have monolog or have memcached. This CI bonus is somewhat akin to "is suggested by others", i.e. VE might not want to list every extension that has a VE plugin as suggested, but the other way around is likely. There may also be cases where it isn't suggested in either direction. But [18:51:58] where it's more a situation of "well, shit something goes wrong between X and Y. If you install both we need to handle it special. Let's write a test for it and integrate in both directions" which isn't per-se cross-promoted to end users. [18:52:15] (assuming you mean "suggest" semantics of composer/npm.) [18:52:16] Yes, indeed. There's "need this or can't operate", " [18:52:20] Bah. [18:52:44] Yes, indeed. There's "need this or can't operate", "need this or can't usefully operate", "need this or can't use important features", and then "need this or can't be tested", etc. [18:53:18] for now we only have the latter as this yaml file and it includes all the others unmarked. [18:53:26] which is not optimal indeed :) [18:54:07] Well, we have requires in extension.json which is somewhere between the first and second. [18:54:24] And then there's the disaster of a different file for phan dependencies compared to PHPUnit ones. [19:08:53] (03PS1) 10Zfilipin: jjb: stop removing log/rawSeleniumVideoGrabs [integration/config] - 10https://gerrit.wikimedia.org/r/1130700 (https://phabricator.wikimedia.org/T294341) [19:10:32] (03CR) 10Zfilipin: "We agreed in 2021 to do this in 2022 but I forgot. 😐" [integration/config] - 10https://gerrit.wikimedia.org/r/1130700 (https://phabricator.wikimedia.org/T294341) (owner: 10Zfilipin) [19:11:10] 10Continuous-Integration-Infrastructure, 10CampaignEvents, 07ci-test-error: api-testing CI failure: Timeout of 5000ms exceeded in "before all" hook in "POST /campaignevents/v0/event_registration" - https://phabricator.wikimedia.org/T389863 (10Daimona) 03NEW [19:11:17] (03CR) 10Zfilipin: "New version in https://gerrit.wikimedia.org/r/c/integration/config/+/1130700" [integration/config] - 10https://gerrit.wikimedia.org/r/734639 (https://phabricator.wikimedia.org/T294341) (owner: 10Zfilipin) [19:11:35] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: CI jobs failing with various timeouts (March 2025) - https://phabricator.wikimedia.org/T388416#10670945 (10Daimona) [19:11:36] 10Continuous-Integration-Infrastructure, 10CampaignEvents, 07ci-test-error: api-testing CI failure: Timeout of 5000ms exceeded in "before all" hook in "POST /campaignevents/v0/event_registration" - https://phabricator.wikimedia.org/T389863#10670946 (10Daimona) [19:11:51] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: CI jobs failing with various timeouts (March 2025) - https://phabricator.wikimedia.org/T388416#10670949 (10Daimona) [19:18:37] 10Continuous-Integration-Config, 10Testing Support, 13Patch-For-Review, 10Test-Platform (The Next One): Remove wdio-video-reporter from all repositories - https://phabricator.wikimedia.org/T294341#10671030 (10zeljkofilipin) [19:21:54] 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.22; 2025-03-25), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10671043 (10Daimona)... [19:25:13] 10Continuous-Integration-Config, 10Quibble: "should mark the revert as a bot edit" test flaky with Apache + php-fpm - https://phabricator.wikimedia.org/T298514#10671060 (10hashar) 05Openβ†’03Resolved a:03kostajh @kostajh fixed it , we had to run the jobs iirc (see https://gerrit.wikimedia.org/r/c/media... [19:35:17] (03CR) 10Hashar: [C:03+2] "From the task, wdio-video-reporter were last in CodeMirror and TwoColConflict and both had it removed with REL1_38 which is no more suppor" [integration/config] - 10https://gerrit.wikimedia.org/r/1130700 (https://phabricator.wikimedia.org/T294341) (owner: 10Zfilipin) [19:35:34] !log Updating Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/1130700 [19:35:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:36:10] 10Continuous-Integration-Config, 10Testing Support, 13Patch-For-Review, 10Test-Platform (The Next One): Remove wdio-video-reporter from all repositories - https://phabricator.wikimedia.org/T294341#10671107 (10hashar) [19:36:32] 10Continuous-Integration-Config, 10Testing Support, 13Patch-For-Review, 10Test-Platform (The Next One): Remove wdio-video-reporter from all repositories - https://phabricator.wikimedia.org/T294341#10671109 (10hashar) 05In progressβ†’03Resolved Thank you for the cleanup! [19:36:34] 10Gerrit, 10Wikimedia-GitHub: Certain Gerrit repositories are missing GitHub mirrors - https://phabricator.wikimedia.org/T389868 (10Pppery) 03NEW [19:36:38] (03Merged) 10jenkins-bot: jjb: stop removing log/rawSeleniumVideoGrabs [integration/config] - 10https://gerrit.wikimedia.org/r/1130700 (https://phabricator.wikimedia.org/T294341) (owner: 10Zfilipin) [19:38:35] 10Gerrit, 10Wikimedia-GitHub: Certain Gerrit repositories are missing GitHub mirrors - https://phabricator.wikimedia.org/T389868#10671126 (10hashar) a:03hashar The repositories need to be **manually** created on GitHub :-/ [19:48:21] 10Scap (SpiderPig πŸ•ΈοΈ): User uid=bd808 unable to finish authentication with spiderpig - https://phabricator.wikimedia.org/T389869 (10bd808) 03NEW [19:53:24] 10Gerrit, 10Wikimedia-GitHub: Certain Gerrit repositories are missing GitHub mirrors - https://phabricator.wikimedia.org/T389868#10671177 (10hashar) 05Openβ†’03Resolved I have created the repositories: https://github.com/wikimedia/mediawiki-extensions-NoCat https://github.com/wikimedia/mediawiki-extensi... [19:55:11] 10Gerrit, 10Wikimedia-GitHub: Certain Gerrit repositories are missing GitHub mirrors - https://phabricator.wikimedia.org/T389868#10671180 (10Pppery) 05Resolvedβ†’03Open Looks like the default branch is set wrong for many of them. Please check that. [19:58:41] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review, 07User-notice: Enable browser notifications system in Gerrit - https://phabricator.wikimedia.org/T389327#10671193 (10Nemoralis) [20:08:43] 10Gerrit, 10Wikimedia-GitHub: Certain Gerrit repositories are missing GitHub mirrors - https://phabricator.wikimedia.org/T389868#10671223 (10hashar) NoCat is made to point to REL1_31: ` $ git ls-remote https://github.com/wikimedia/mediawiki-extensions-NoCat 462fbaa277d14f38b98621cd4a906214fc551ee0 HEAD 462fbaa... [20:09:53] 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.22; 2025-03-25), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10671241 (10Umherirre... [20:23:53] 10Gerrit, 10Wikimedia-GitHub: Certain Gerrit repositories are missing GitHub mirrors - https://phabricator.wikimedia.org/T389868#10671305 (10Pppery) 05Openβ†’03Resolved Thanks. [20:34:04] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T389875 (10Paul26) 03NEW [20:48:57] 10Gerrit: Gerrit's "indent with tabs" option is seemingly ignored - https://phabricator.wikimedia.org/T389878 (10Daimona) 03NEW [20:55:03] 10Continuous-Integration-Infrastructure, 10CampaignEvents, 07ci-test-error: api-testing CI failure: Timeout of 5000ms exceeded in "before all" hook in "POST /campaignevents/v0/event_registration" - https://phabricator.wikimedia.org/T389863#10671424 (10Daimona) [21:03:21] 10Gerrit: Gerrit's "indent with tabs" option is seemingly ignored - https://phabricator.wikimedia.org/T389878#10671461 (10Jdforrester-WMF) I've noticed this; I just copy a tab character from other lines. It's most irritating. [21:03:57] Ohhhhh I'm so happy that it wasn't just me :D [21:07:59] (03PS1) 10Jeena Huneidi: WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) [21:09:20] (03CR) 10CI reject: [V:04-1] WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) (owner: 10Jeena Huneidi) [21:12:24] 10Scap, 13Patch-For-Review: Scap seemingly doesn't fully/properly clean backup deployment server - https://phabricator.wikimedia.org/T389830#10671483 (10dancy) p:05Triageβ†’03Low a:03dancy [21:39:20] (03PS2) 10Jeena Huneidi: WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) [21:41:14] (03CR) 10CI reject: [V:04-1] WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) (owner: 10Jeena Huneidi) [21:49:48] (03PS3) 10Jeena Huneidi: WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) [21:50:58] (03CR) 10CI reject: [V:04-1] WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) (owner: 10Jeena Huneidi) [21:55:04] (03PS4) 10Jeena Huneidi: WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) [21:56:21] (03CR) 10CI reject: [V:04-1] WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) (owner: 10Jeena Huneidi) [22:01:22] (03PS5) 10Jeena Huneidi: WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) [22:02:29] (03CR) 10CI reject: [V:04-1] WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) (owner: 10Jeena Huneidi) [22:05:11] (03PS6) 10Jeena Huneidi: WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) [22:06:42] (03CR) 10CI reject: [V:04-1] WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) (owner: 10Jeena Huneidi) [22:08:18] (03PS7) 10Jeena Huneidi: WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) [22:09:21] (03CR) 10CI reject: [V:04-1] WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) (owner: 10Jeena Huneidi) [22:14:11] (03PS1) 10Jeena Huneidi: Add catalyst dockerfile [integration/config] - 10https://gerrit.wikimedia.org/r/1130729 [22:15:22] (03PS8) 10Jeena Huneidi: WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) [22:15:24] (03CR) 10CI reject: [V:04-1] Add catalyst dockerfile [integration/config] - 10https://gerrit.wikimedia.org/r/1130729 (owner: 10Jeena Huneidi) [22:16:36] (03CR) 10CI reject: [V:04-1] WIP: Add wikilamba end-to-end selenium testing [integration/config] - 10https://gerrit.wikimedia.org/r/1130722 (https://phabricator.wikimedia.org/T383065) (owner: 10Jeena Huneidi) [22:25:43] 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.22; 2025-03-25), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10671757 (10Jdlrobson... [22:27:07] 10Continuous-Integration-Infrastructure, 06Test-Platform, 07ci-test-error, 10MinervaNeue (Tracking): Minerva test builds are failing after recent Selenium upgrades - https://phabricator.wikimedia.org/T389889#10671760 (10Jdlrobson-WMF) [22:59:38] 06Release-Engineering-Team, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10671841 (10Scott_French) Thanks, @BTullis - I was aware of Joe's recent scap change to support `cluster_di... [23:08:07] 10Continuous-Integration-Infrastructure, 07Jenkins: Manage provisioning and configuration of Jenkins agents using OpenTofu - https://phabricator.wikimedia.org/T389891 (10bd808) 03NEW [23:24:59] 10Continuous-Integration-Infrastructure, 10CampaignEvents, 10Campaigns-Product-Team (Campaign-Tools-Current-Sprint), 07ci-test-error: api-testing CI failure: Timeout of 5000ms exceeded in "before all" hook in "POST /campaignevents/v0/event_registration" - https://phabricator.wikimedia.org/T389863#10671902 (... [23:55:10] 10Gerrit: Gerrit's "indent with tabs" option is seemingly ignored - https://phabricator.wikimedia.org/T389878#10672035 (10Paladox) You can use the tab key. Although it should probably default to tab when you press enter to go onto a new line.