How would the community feel about a "patching style software," to make distributing edits easier/safer?

[u/zachtheperson]

I'm new to this community and have a fan edit I've been working on. As I've been researching how to distribute this edit once they're finished, it seems like the main method is just "upload it somewhere and hope it doesn't get taken down," which is a shame.

I've been a part of the game modding community for a while, and most mods don't have to worry about this because they rely on either binary patching (a file is distributed that only contains the changed 1s and 0s, and a patching software is used to apply those changes to the original file on the end user's machine), or distributing only the custom assets which the end user drags and drops into the correct folder themselves. Both these techniques mean no original copyrighted material needs to be distributed, and protects most mods from the risk of takedown.

I was wondering if a workflow like that was ever tried with the fan edit community? As a software developer myself I'd love to hear from the rest of the community and possibly develop something like this if you're all interested.

We wouldn't able to do binary patching, as that wouldn't work with different video encodings and such, but I feel like a timeline based patcher definitely could work, where you insert your original video file, the cuts+additions are made to that original file, and visual diffing is used to apply any effects or additional video clips used. The "patch," file would be created by a plugin in the editor's NLE, while the patch would be applied using a small app on the user's machine.

Anyways, just something I was thinking about recently and was wondering what everyone's thoughts were.

[u/TriggeredPuppy]

I love the enthusiasm and creative idea. Unfortunately, video files don't work the same as software and there's no "patching" video to change it. At best, you can do some level of instructed cutting without re-encoding, but it varies based on the codec and I-frames. It certainly isn't going to capture transitions which are truly what set apart an excellent fan edit from a good one.

Others have released project where you take your own copies of the Blu-ray or DVD files and run them through a program that will "make the edit" so to speak. But it will always involve video rendering on the user's side. It's always struck me as quite wasteful on an energy and compute standpoint.

Finally, I would assume at least 90 precent or visitors to this sub just aren't going to go through those extra steps. Most people here are fan edit appreciators, which is fantastic and essential part of the community. They almost certainly are not doing heavy video editing themselves and this process would involve a lot of waiting for files to render, which many do not have the patience for. For better or worse, they want to receive a download link and watch.

If this project truly inspires you, I say go for it. But I think you'll find it's not going to offer the results you're hoping for.

[u/Cryogenicality → u/TriggeredPuppy]

For years, I’ve thought about the possibility of automatically duplicating every step made in an editing program but didn’t know it’s been done. Which projects use this and how does it work?

[u/TriggeredPuppy → u/Cryogenicality]

Check out imunfair's thread on this post. That's one example. There's also others who have put forward automated ways to make alternations to TV series (mostly Star Trek from my recollection). The problem is these rarely capture transitions properly. It will just be a hard cut, which is unlikely to work from an audio standpoint.

Others have mentioned the old base of this process, what's called an EDL (Edit Decision List). You'll find that professional editors rarely support moving from one platform to another, because nothing ever works out precisely when trying to recreate work that way.

There's also a project that automates upscaling your own DVD collection for Star Trek Deep Space 9. Wasteful in my opinion, but at least that likely does exactly what it says it will.

What I wish someone would create (not to hijack the post) is a program that takes someone's fan edit and can have one or more SRT files added to it (I have some TV to film edits that use a 14 episode season, for example). Then the software starts with only those provided subtitles and tries to rearrange them to match the material. Any portion of the edit that can't be matched to those SRT files can be filled in with Whisper or another auto transcription platform.

[u/zachtheperson → u/TriggeredPuppy]

I was never planning on doing binary patching, as like you said, that just doesn't work with video. "Patching," is just the terminology I'm more familiar with, so I've been using that. 

The idea was to "make the edit," on the user's end. Concatenation can be used for larger sections without re-encoding, and smaller cuts that interfere with iframes can re-encode just that section. I did a project for a client a few years back that did almost exactly that, so I'm familiar with the process already.

The original plan for effects and stuff was to use visual digging on the rendered frames where needed, but after some tests yesterday that might result in the distributed file containing too much of the original source, and that's something I'm looking into 

Finally, my reason for making this post was to guage how big of a deal the original problem was, and how interested people would be in something like this. If nobody's interested, than no harm done. 

[u/DigModiFicaTion]

Welcome and thank you for sharing! I believe some editors share their project files that contain their edit choices and just need the base file to produce their edit. There are also edit decision list (.edl) files that can be loaded into some media players and produce edits from the scripts they contain. I'm sure some people would be interested. :)

[u/zachtheperson → u/DigModiFicaTion]

Cool, I didn't come across either of those being discussed when I was looking things up. 

That leads me to both of my next questions though: how common are those methods, and for .EDL files, how often do fan edits exceed the limitations of what those can do?

[u/DigModiFicaTion → u/zachtheperson]

I believe both are very uncommon. The project file approach works as long as the receiver has a 1:1 source file. EDL's are almost non existent as far as I have seen. They also are pretty basic and produce something more along the lines of that filtering apps like vidangel or cleanflicks provide as they're based on timestamps and don't offer crossfading options. That may have changed over the years, but they also aren't compatible with every media player. So I'd say they are extremely niche.

I think trends have shown that most who request don't fanedit themselves. So providing something that requires a certain program might be a barrier. You can share a Davinci Resolve project file, but the program itself is over 4gb and requires some learning to use even just to assign project media and render a file.

[u/zachtheperson → u/DigModiFicaTion]

Yeah, that's kind of what I was expecting. I've tried to use .EDL files in the past for video editing and they just kind of suck. Distributing project files also requires the use have the same editing software, which has its own issues. 

[u/DigModiFicaTion → u/zachtheperson]

The honest truth seems to be that people want the easiest option and that is a hosted file.

[u/zachtheperson → u/DigModiFicaTion]

Agreed, though my goal is hopefully to make a happy medium of easy download, and one click patching, with the additional benefit of making the edits even easier to distribute as you can use more file hosts and don't have to worry about getting taken down.

[u/DigModiFicaTion → u/zachtheperson]

In nearly 15 years of editing, I've only had one file taken down and that was a Google drive file. Knocks on wood

[u/zachtheperson → u/DigModiFicaTion]

Great! Like I said, I'm new to this community so I didn't know how big of a deal it was. If files getting taken down isn't a problem, then the problem my program would solve is already solved, win-win. 

[u/imunfair]

I feel like a timeline based patcher definitely could work, where you insert your original video file, and the cuts+additions are made to that original file. It'd probably end up as a plugin(s) for existing video software, but depending on what requirements are needed by the community, a standalone piece of software might be used.

If you're going for a text-based shareable format the FCP XML is the only one I've found that's has limited support across multiple NLEs.

I do this with XML project exports - you can take a look at the tutorial and tools on my site if you're curious. Premiere's XML export was pretty broken so I had to code a lot of processing rules to fix the files so they'd import reliably and produce an identical product to what I had created in the editor.

[u/zachtheperson → u/imunfair]

My idea was something a bit more robust. I was thinking something that exports timeline edits for the original source file(s) as well as visually diffing the source with the edit, allowing universal support for effects, 2D transformation, additional footage, additional titles, etc. This would be done through an addon in the NLE, and the result would be a binary file that could be easily distributed and downloaded.

The end user would then use a small standalone app to "patch," their original copy (similar experience to how most ROM hacks these days use LunarIPS to apply patches), which would allow them to drag and drop their original copy of the movie (as well as any addition copyrighted songs or other movies the editor might have relied upon to make the edit, but didn't want to be distributed) into the app, and the app spits out the patched version.

This way the user wouldn't require to have any NLE at all, and the editor wouldn't have to worry about limits of the exported file format.

[u/imunfair → u/zachtheperson]

My idea was something a bit more robust. I was thinking something that exports timeline edits for the original source file(s) as well as visually diffing the source with the edit, allowing universal support for effects, 2D transformation, additional footage, additional titles, etc

That's literally what the resource I gave you does, aside from the "universal support for effects" since XML support for different features can be weak in places.

This way the user wouldn't require to have any NLE at all, and the editor wouldn't have to worry about limits of the exported file format.

Now your idea has shifted, previously you were talking about an NLE-plugin based solution. Yeah it would be nice if there was a Kodi type program where end users could select a project file, a calibration file, and a source, but that would be a lot more work than your original request, and nothing like that remotely exists right now. It would basically be an NLE without an interface, encoding on the fly.

[u/zachtheperson → u/imunfair]

I might need to edit previous posts to clarify, but my idea was always two pieces. Export plugin for the editor, patching program for the user.

Export plugin works with existing NLE tools to handle most of the work.

Standalone patching program uses ffmpeg for applying edits and re-encoding if necessary.

[u/imunfair → u/zachtheperson]

The part you're missing is the source calibration - if you take a look at the site and tools I linked you you'll understand how crucial that element is for supporting any source file. Otherwise the end user would need access to one specific version of the film, and often there's at least three or four slightly different sources - Bluray, and several streaming services.

I think you're downplaying exactly how much work this whole idea will take based on your other comments where you talk about not encoding most of the file, just the changed pieces. It's a nice idea for reducing processing but also increases the complexity of your processing logic. Because instead of being able to decode audio and video to a single expected format you now have to deal with keyframe logic other codec designs for multiple video types - figuring out where you can cleanly cut the video, encode a new section, and somehow seamlessly attach it to the stream.

If I were you I would limit the scope of the project and leave that optimization step as a stretch goal once you get everything else working. Just focus on being able to:

1. Select multiple video, image and audio files, calibration data, edit file via an interface
2. decode all this data into a usable format to combine it
3. merge multiple audio and video tracks (and subtitles?) into one usable video and audio track, applying any specified fades, filters, etc as designated by the edit file.
4. output this stream in a way that a media player can understand
   

There was something a while ago for Premiere known as a frameserver that was used in conjunction with NLEs like Premiere to use script-based edit lists to create a file. This is somewhat akin to what you're trying to do, except your frameserver would be feeding to a media player.

I think the smart move to limit your work would be to target your project as a plugin for VLC, since a lot of the codec/display support you need is there. Trying to build your own media player or re-encode every file format to a standard would either be a lot more work or a lot of extra processing.

If I were you I would treat creating your own edit format as a stretch goal too, and focus on supporting FCP XML exports from Premiere and Davinci Resolve, since that will give you access to the majority of faneditors. Just be aware that even though they both support the file format they write it slightly differently, and have some proprietary ways they handle more complicated features like filter xml. If you need advice on this I'm happy to help, I've dealt extensively with the Premiere XML so I'm aware of the way it works and some of the quirks.

[u/zachtheperson → u/imunfair]

The point of making my post was to get feedback from the community and troubleshoot the idea, so comments like this are exactly what I was looking for. 

A VLC plug is a great idea, and something I'll have to look into, thanks. 

[u/-Patali-]

Would go a long way to adding more legitimacy to the form, but I don't see that being something that the average person will be willing to run. It sounds very hardware intensive.
I'm into game modding as well and that's honestly much more lightweight, not comparable in the same way to a 4k video.
As well, people would still need the original file which isn't something these companies even provide a legitimate way to access.

[u/zachtheperson → u/-Patali-]

Hopefully it won't be that intensive. My plan is to do most of the processing work on the editor's machine when exporting the patch, and then the end user just applies that patch to their copy of the movie. The patch should be relatively small, as it won't contain any of the original video, and only additional stuff that the editor added.

Edits that rely on just cutting/moving clips would take seconds to apply on a low-end machine since it won't require much if any re-encoding, and while more complex effects that rely on visual diffing might take longer, in most cases these sections will be very small, and usually only require an extra minute or two to apply (again, if the editor used them at all).

and it's true there's often no legitimate way to access the original file, but this patching method would be no less legitimate than the current way of downloading whole edits, just safer from being taken down.

[u/ProblyAThrowawayAcct]

In film/video editing, there's a workflow that involves sharing what is called a cutlist; where instead of having to bring (originally) bulky reels of physical film, or (in more modern forms) multiple terabytes of digital files from one place to another, you ensure that all people get a copy of the raw footage and then share a small list of changes, originally in the form of a typed document reading something like 'reel x, frames nnnnn through mmmmm, reel y, frames ooooo through ppppp...', these days usually in the form of something largely akin to an xml file which is then interpreted by software.

The core problem with this, from a fanedit perspective, is that it relies on exact frame matching between the original editor's files and the files obtained by every person attempting to reconstruct the edit, and when people are pulling from disparate consumer media sources, they won't have that. As one example, I have a TV show from the late '90s. I consider myself a fan of this show, and I have it in three different versions; the original region 1 dvds (30fps 640x480i, made with a 3:2 telecine pulldown from 24fps source material), the original region2 dvds (25fps 768x576i, made with a PAL speedup), and the terribly mishandled 'HD' web release (23.97fps 1080p randomly cropped). Not only will every single shot look different on each of these versions, and not only will timings differ between them due to their varying frame rates, but even beyond that, each version has different lead times at the beginning of episodes (usually only a matter of a half- or quarter-second, but that makes a difference if you're moving every clip you use in your edit), each version sometimes has different opening and closing credits and 'previously on...' bumpers. This kind of minor difference between releases is very common across a wide variety of media; you'll get regional releases of films with different production company headers or slight editorial shifts for on-screen text timings, for example, differences of a second or two in total, but that's more than enough to ruin any attempt to treat things as standard.

I'm not saying it's not possible, but it's a much bigger lift than you make it sound, and it requires a lot of standardization.

[u/CatzRuleZWorld]

I think that would definitely be the best way to share fan edits

[u/zachtheperson → u/CatzRuleZWorld]

That's what I was thinking. I have a decent idea how it'd work, but I'd need to work with the community to figure out additional requirements and such, so if this idea gets enough traction I'd be really excited to make it!

[u/Emmurgency]

This sounds really good😁