[00:33:19] [1/5] I'd be curious to know the overhead per query and the overhead of storing hundreds of rows on a single page since these are known areas where Cargo performs poorly. [00:33:19] [2/5] E.g. https://stellasora.miraheze.org/wiki/Amber takes a few seconds to parse mainly because it makes hundreds of cargo_query calls. [00:33:19] [3/5] https://bluearchive.wiki/wiki/Banner_List got split off into subpages by year because it was timing out the server on page save. [00:33:20] [4/5] These are arguably abuses of Cargo in a way it was not designed to handle, but it would be nice if Bucket performs better than Cargo. My local benchmarks worked in Bucket's favor but MH's setup might be different. [00:33:20] [5/5] With that said I think you've done more than enough testing to make sure Bucket works. If we run into performance problems later we can always restrict it like Cargo. With how good WG is when it comes to performance, I doubt many wikis will run into perf issues. [00:38:07] [1/34] https://buckettest.mirabeta.org/wiki/User:TheWWRNerdGuy/sandbox it didn't update properly when I stored 5000 rows on the page but it manages 2500 perfectly fine [00:38:07] [2/34] ``` [00:38:07] [3/34] NewPP limit report [00:38:08] [4/34] Parsed by test151 [00:38:08] [5/34] Cached time: 20260306222600 [00:38:08] [6/34] Cache expiry: 1296000 [00:38:09] [7/34] Reduced expiry: false [00:38:09] [8/34] Complications: [] [00:38:09] [9/34] CPU time usage: 4.106 seconds [00:38:09] [10/34] Real time usage: 4.083 seconds [00:38:10] [11/34] Preprocessor visited node count: 52480/1000000 [00:38:10] [12/34] Revision size: 522300/2097152 bytes [00:38:11] [13/34] Post‐expand include size: 0/2097152 bytes [00:38:11] [14/34] Template argument size: 0/2097152 bytes [00:38:12] [15/34] Highest expansion depth: 2/100 [00:38:12] [16/34] Expensive parser function count: 0/99 [00:38:13] [17/34] Unstrip recursion depth: 0/20 [00:38:13] [18/34] Unstrip post‐expand size: 0/5000000 bytes [00:38:14] [19/34] Lua time usage: 2.704/10.000 seconds [00:38:14] [20/34] Lua memory usage: 949689/52428800 bytes [00:38:15] [21/34] Lua Profile: [00:38:15] [22/34] recursiveClone 960 ms 34.5% [00:38:16] [23/34] MediaWiki\Extension\Scribunto\Engines\LuaSandbox\LuaSandboxCallback::put 540 ms 19.4% [00:38:16] [24/34] MediaWiki\Extension\Scribunto\Engines\LuaSandbox\LuaSandboxCallback::getAllExpandedArguments 320 ms 11.5% [00:38:17] [25/34] (for generator) 160 ms 5.8% [00:38:17] [26/34] type 140 ms 5.0% [00:38:18] [27/34] MediaWiki\Extension\Scribunto\Engines\LuaSandbox\LuaSandboxCallback::getExpandedArgument 120 ms 4.3% [00:38:18] [28/34] MediaWiki\Extension\Scribunto\Engines\LuaSandbox\LuaSandboxCallback::find 80 ms 2.9% [00:38:19] [29/34] pairs 80 ms 2.9% [00:38:19] [30/34] 60 ms 2.2% [00:38:20] [31/34] gsplit 40 ms 1.4% [00:38:20] [32/34] [others] 280 ms 10.1% [00:38:21] [33/34] Bucket time usage: 0.000/10 seconds [00:38:21] [34/34] ``` [00:39:36] iirc the recursiveClone is the overhead of an `invoke` statement [00:48:31] [1/28] https://buckettest.mirabeta.org/wiki/User:TheWWRNerdGuy/sandbox2 [00:48:31] [2/28] This is for 10,000 queries (all queries are the same) [00:48:31] [3/28] ``` [00:48:32] [4/28] NewPP limit report [00:48:32] [5/28] Parsed by test151 [00:48:32] [6/28] Cached time: 20260307004720 [00:48:32] [7/28] Cache expiry: 1296000 [00:48:33] [8/28] Reduced expiry: false [00:48:33] [9/28] Complications: [] [00:48:33] [10/28] CPU time usage: 4.689 seconds [00:48:34] [11/28] Real time usage: 7.388 seconds [00:48:34] [12/28] Preprocessor visited node count: 4/1000000 [00:48:35] [13/28] Revision size: 35/2097152 bytes [00:48:35] [14/28] Post‐expand include size: 79996/2097152 bytes [00:48:36] [15/28] Template argument size: 0/2097152 bytes [00:48:36] [16/28] Highest expansion depth: 2/100 [00:48:37] [17/28] Expensive parser function count: 0/99 [00:48:37] [18/28] Unstrip recursion depth: 0/20 [00:48:38] [19/28] Unstrip post‐expand size: 0/5000000 bytes [00:48:38] [20/28] Lua time usage: 4.320/10.000 seconds [00:48:39] [21/28] Lua memory usage: 1326273/52428800 bytes [00:48:39] [22/28] Lua Profile: [00:48:40] [23/28] MediaWiki\Extension\Scribunto\Engines\LuaSandbox\LuaSandboxCallback::run 4180 ms 96.8% [00:48:40] [24/28] MediaWiki\Extension\Scribunto\Engines\LuaSandbox\LuaSandboxCallback::jsonEncode 80 ms 1.9% [00:48:41] [25/28] 20 ms 0.5% [00:48:41] [26/28] [others] 40 ms 0.9% [00:48:42] [27/28] Bucket time usage: 0.013/10 seconds [00:48:42] [28/28] ``` [00:50:45] These benchmarks are obviously very flawed since it's doing the same thing thousands of times rather than what you'd actually have in production but it seems to be holding up quite well [01:19:02] hell even if it was ass I'd still like it more than the ones that turn your wiki into a bomb [01:27:38] That's amazing considering how Cargo cannot handle 400 stores on a page. [01:35:29] Good job bucket [01:53:39] I'll work a bit more on formatting the output of Cargo queries in Module:Bucket, but it's not a blocker. @thewwrnerdguy if you feel like there are no longer any blockers we can probably deploy it? Some issues were raised on MM but I think Random addressed it. [07:02:14] [1/4] I added `format=template` and `format=userformat` which should cover most use cases for Bucket formatting. [07:02:15] [2/4] There's also a simple expression parser for the `where` clause. Unfortunately Bucket limits the types of queries you can make by design, so we might have to create a separate argument (e.g. `filter`) which filters results after they are retrieved from the database with a few more supported features. [07:02:15] [3/4] Support for joins is still missing and the fact that we allow people to give fields aliases will make this a lot more interesting. [07:02:15] [4/4] I think WWR has some ideas about link formatting so I'll leave it to him. I believe only `format=table` needs this since for the other formats the user can manually specify the linking syntax. [08:40:50] The bucket issues migration still needs to be converted to a maintenance script, I'm almost done doing that but there is still an issue with autoloading, but I can look into that today or tomorrow [08:42:05] is public now and the fix for that is still deployed as a patch, but it can be replaced with the latest commit later [09:51:20] ah, i saw that, but i assumed that $value had to be a valid title [09:51:36] or did i just misunderstand the code [10:17:06] update: that was an ouchie in hindsight [10:47:46] ah [10:48:19] The commit causing it was pushed literally 3 days after I reviewed Bucket lol [10:48:29] in october [12:01:39] I don't actually have any ideas about link formatting, the only special case I coded was that `page_name` and `page_name_sub` would get formatted as links [12:03:54] I think there was something I had thought maybe to do but I can't remember if I had anything other than find the bucket schema and find out which fields are PAGEs based on that (but I'm not sure that's a good idea) [17:36:42] [1/4] https://cdn.discordapp.com/attachments/1006789349498699827/1479895579969454183/image_9s4ctd.jpg?ex=69adb3a9&is=69ac6229&hm=d74c6dff9c0be00c44f01b1d6f37a862c0aa00ec33742cd99e0fbbc7e1127aa6& [17:36:42] [2/4] https://cdn.discordapp.com/attachments/1006789349498699827/1479895580846326002/1771854798364.jpg?ex=69adb3a9&is=69ac6229&hm=9c3ce707a3b9f9b143b8856dbae7207ad19f1c8844fac51e424f3768031ba2bb& [17:36:42] [3/4] https://cdn.discordapp.com/attachments/1006789349498699827/1479895581450309824/20260213_134054.jpg?ex=69adb3a9&is=69ac6229&hm=f20a132163cfacf7413d1fdc37081fa8bc0c48bcfd824c48c7f4bf73eb1aafb3& [17:36:43] [4/4] https://cdn.discordapp.com/attachments/1006789349498699827/1479895582154817658/image_ksuynz.jpg?ex=69adb3a9&is=69ac6229&hm=698948ed5ce76fc09a7c00c1e48c9ffc912e349d877dce5694427f2192d5febc& [20:17:46] https://issue-tracker.miraheze.org/T15069 just wanna check, hyphens in wiki urls isn't allowed right [20:19:23] when in doubt try requesting a wiki on mirabeta [20:31:42] Yeah doing a page read is expensive, though code-wise it wouldn't add too much complexity I think. Perhaps best hidden behind an option. [21:33:50] Wait, you just need `mw.loadJsonData`. Not sure if it works for Bucket NS but if it's json then there's a fair amount of chance it can work. [21:36:08] 🤦‍♂️ how did I forget about `mw.loadJsonData` lol [21:37:42] [1/2] looks fine [21:37:42] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1479956231299797133/image.png?ex=69adec25&is=69ac9aa5&hm=2a2dd514121547315d526ad5f16f3dfc5cb1701bd773d5053b82a62ab28af822& [21:38:20] I made a small update to Module:Bucket. The next ones are join and group by which will introduce some complications. [21:47:41] It can be if use server url like a custom domain but it is not for databases. [23:51:43] [1/2] @posix_memalign btw I made Module:Bucket/put in a way that you can only store data from an invoke directly [23:51:44] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1479989959237697536/image.png?ex=69ae0b8f&is=69acba0f&hm=48e579b6b9fa167d91694be720def6e2d3c50a98d2d7ebda23bf1120fcb3e6cb& [23:52:04] I didn't want a thing where you'd have arguments to the template also getting inserted into the bucket