[18:04:38] [[Tech]]; 41.113.137.32; /* Car prices */ new section; https://meta.wikimedia.org/w/index.php?diff=26201375&oldid=26189789&rcid=29980669 [18:04:55] [[Tech]]; Tegel; Reverted changes by [[Special:Contributions/41.113.137.32|41.113.137.32]] ([[User talk:41.113.137.32|talk]]) to last version by ArchiverBot; https://meta.wikimedia.org/w/index.php?diff=26201376&oldid=26201375&rcid=29980670 [22:55:05] bvibber: Not sure if you're the best person to ask - do you happen to know if increasing $wgTranscodeBackgroundSizeLimit to 5GB is feasible (T357184)? Right now its 3GB, which doesn't entirely make sense when you can upload videos quite a bit larger than that, and it seems like 3GB was chosen way back in the day. I'm pretty sure all the swift stuff would be fine with the bigger limit, but i don't [22:55:06] T357184: Consider increasing $wgTranscodeBackgroundSizeLimit to 5GB - https://phabricator.wikimedia.org/T357184 [22:55:11] really know how the shellbox part works [23:11:35] I think it should be good as long as shellbox doesn't impose a limit [23:12:11] Yeah I noticed that longer high res films were bumping up on the limit [23:12:27] I'll assign that to me to peek at real quick [23:13:03] _joe_ would know for sure about the size limit for shellbox output [23:17:45] <_joe_> bawolff, bvibber the problem is not shellbox per se, it just makes things slightly slower [23:18:19] <_joe_> the biggest issue is the time it takes to verify the signature of the file that is posted [23:18:29] <_joe_> which for a 5 gb file could be... minutes? [23:18:36] <_joe_> but a transcode is gonna be hours [23:19:06] <_joe_> so, maybe it doesn't work because of some limits I'm not thinking about, but we can at least definitely test it [23:19:20] yeah, i think we already have transcodes that take like 12 hours [23:19:38] <_joe_> we have transcodes that never finish because they last more than 24 hours [23:19:53] <_joe_> (a problem I want to fix with the transition to kubernetes, btw) [23:20:23] <_joe_> but OTOH, I have to say, let's first see how things go when we move to a remote shellbox :) [23:21:04] I'm working on a different bug about 5GB uploads. We're at the testing stage for that one [23:21:44] <_joe_> if you're curious about the details, https://phabricator.wikimedia.org/T292322 is probably kind-of a blocker [23:22:30] <_joe_> tldr: I've been waiting for 2.5 years now? for a decision on wether we're ok lowering the limit for upload-by-url, or if someone will work on make it asynchrnous [23:22:42] <_joe_> so I'm doing the latter wokr myself [23:23:06] <_joe_> but some details on the time cost of shipping multi-gb files to shellbox is measured there :) [23:24:01] upload by url is synchronous ! [23:24:04] I did not know that [23:25:05] I feel like not too many people use upload by url. Requiring you host to get on a whitelist is a bit of a barrier towards people just using it [23:25:56] it's synch-y and also flakey [23:26:29] so there are a lot of people who want to use it, but can't because it breaks and resort to downloading and chunked uploading or server-side uploads [23:27:36] chunked upload of course is also super flakey [23:29:23] <_joe_> https://gerrit.wikimedia.org/r/c/mediawiki/core/+/982755 this set of patchsets introduces async upload-by-url via the api [23:29:47] <_joe_> now that I've done the work on tmh, I will go back and to the same in special:upload [23:30:06] <_joe_> because ofc the api endpoint and the special page do not share business logic :) [23:31:01] <_joe_> so finally we'll complete the migration to shellbox. In hindsight, I should just have done this two years ago, I would've saved everyone time [23:31:28] The upload code in mediawiki is sketch. It is my least favourite part of mediawiki [23:31:44] <_joe_> bawolff: some of the code I touched could drive in the US :D [23:31:53] i wish i could commit to doing a full rewrite of that code ;) [23:31:54] lol [23:32:19] I sort of want to rewrite chunked upload, but it looks like a lot of work [23:32:21] just gonna have to keep refactoring as we go [23:32:26] <_joe_> bawolff: yeah, it's not like you're paid to modify it. It's not my job, but at least I get paid for the pain :P [23:32:49] lol [23:33:05] <_joe_> now, I'd argue it's problematic that it's not anyone's job, but I sound like a broken record :) [23:33:48] <_joe_> bvibber: I've been tempted to fix so many bugs [23:34:11] <_joe_> but I'm restraining myself or I'll get lost, especially as someone who's hardly ever done feature development in mediawiki [23:34:20] Well, doing stuff in mediawiki core gives me a reputation as someone who knows about mediawiki, which i have leveraged into jobs in the past :) [23:34:29] hehe [23:34:41] software development is more fun when you don't get paid for it [23:34:48] truuuuuuuue [23:34:59] <_joe_> bawolff: yeah and that's why you don't go refactor that monster [23:35:01] <_joe_> :P [23:35:09] <_joe_> you do the stuff that is fun and rewarding [23:35:27] truee [23:35:32] anyway ;) we currently have transcodes set a soft limit at 2 gb (admins can override it) and a hard limit at 3. if we raise both to 5 is it going to explode, or just add a few minutes to the hours-long jobs that will encounter the case? [23:35:36] <_joe_> if you really like file handling and media management... [23:36:22] <_joe_> bvibber: tbh I would first try to test ourselves how long it takes to transcode a 5 gb file on our infra [23:36:31] <_joe_> there is a chance it would never complete [23:36:37] heh [23:36:42] lemme check some of those films [23:36:42] <_joe_> because we have a hard limit at 24 hours [23:38:14] <_joe_> It's something that would be easier if we start running jobs for videos as cli scripts that can run forever [23:38:57] yeah that'd be much nicer [23:39:32] lessee here's a 45-minute 4k video with the 1440p and 2160p transcodes blocked by the 2gb soft limit: https://commons.wikimedia.org/wiki/File:The_Earth-_4K_Extended_Edition.webm [23:39:38] the 1080p look about 2 hrs [23:40:10] i would estimate 4x longer for 2160p so 8 hours [23:40:25] which means it might complete :D [23:41:34] lemme see if it's willing to run them on a manual reset... [23:41:52] 59 min not 45 min sorry [23:42:09] The file that was being complained about on my talk page was file:politparade.webm [23:42:30] ok it's running the 1440, but it estimates the 2160 as too large: estimated file size 4526793 KiB over hard limit 3145728 KiB [23:42:39] so that's probably our target for testing [23:45:10] (added a note on https://phabricator.wikimedia.org/T357184 )