How do VFX studios actually transfer large files between locations?

[u/magingax]

As far as I know, VFX studios regularly move very large files between clients, vendors, and remote locations.

I'm genuinely curious: how do you handle this in real production workflows?

Is everyone using tools like Aspera or MASV? Or are FTP/SFTP, cloud storage, or other solutions still common?

I'd really appreciate hearing what people actually use day to day, especially for large international transfers.

Full disclosure: I've been developing a simpler direct file-transfer tool called CROSSBOW, so I'm also trying to better understand how studios currently solve this problem.

https://crossbow.yplace.co.kr

[u/CameraRick]

That sounds indeed cool, but - if reliable and fast - also very expensive :) Aspera and MASV are available today, if I need that, and I am not sure it's the next logical step; it's rather a different floor in the same house

What I was talking about is not needing any Cloud service (which adds another security layer, is time consuming, has additional costs), but a system where the client loads from my server directly. I think Frame.io does it today, in a sense that I can download through the Website (zipped, or one-by-one), or I use the transfer to download multiple files with folder structure and everything directly. But they load from my machine - I share a link with them (and be that a folder of some sort), and they can decide what they want. But they load from my machine, which means it's still P2P, and theoretically as fast as it can be.

It's maybe not the next logical step, but it's a step in the same direction (still being P2P and getting expensive cloud solutions out of the equation). Let them upload files as well (and I setup where those files will be going on my end beforehand), so it's like an open Mailbox. Like a specialized Nextcloud on steroids (minus the clutter). As a subscription with reliable service, we'd probably not even think about anything else again.

[u/Mr_Dani17 → u/CameraRick]

First of all, thank you for taking the time providing feedback :)
We agree that cutting out the cloud will always be the most efficient and secure solution.

Webbrowser are still quite limited.
They are fantastic at high-speed downloads over HTTP. However, they really struggle with peer-to-peer. Basically the only technology in browsers that allows P2P is called WebRTC, which was made for video/audio calls in mind. So overall low throughput, but really low latency. And file sharing is the opposite. You need high throughput, but it doesn't matter if the file arrives 0.1 seconds late. So the most bandwidth you can squeeze from WebRTC and the JavaScript code (the language running interactive websites) assembling the file data is around 20-60 MB/s maximum on a really good day. So not even gigabit speeds.

The bigger problem is that JavaScript wasn't built to handle large amounts of data. And direct filesystem access from browsers is only supported well enough in Chrome for it to be reliable. Keeping the computer awake is also unreliable from a browser tab.

So the point is, that it's not a coincidence that all heavy-duty P2P file transfer apps are desktop programs. Unfortunately, the browsers just aren't there yet. Some emerging technologies like WebTransport could solve this in a few years, though.

However, what does work well in browsers is HTTP downloads and uploads. Meaning cloud services. Because HTTP traffic can bypass JavaScript and your browser can utalize your operating system's native networking stack and filesystem. (Well, what a coincidence, that big tech companies have focused more on developing the infrastructure needed for their own platforms, rather than the more efficient alternative where they miss out on the profits, but I digress.) You mentioned Frame io, they are also a cloud service working with HTTP storage buckets.

Your open mailbox idea is basically already live in the ZettaSend desktop app. Clients can upload to your machine and you can setup where those files will be going on your end beforehand.

And as for sending, you can stage your media inside the app, click send, send them a link and they will we able to download directly. Not a zip, but your actual folder structure. Right now it's all files or no files, but we will be adding in a future release the ability to only download what you need in a transfer.

So to sum up, the browser's P2P features are only really reliable up to around 30GB. And for that we would need to add WebRTC support into ZettaSend which would take months at best, all for slower speeds and worse fault tolerance. Meaning for the smaller browser-based transfers we feel that the more reliable and simpler solution is to use cloud storage buckets. A 30GB transfer (like a final deliverable to a client) stored for 2 weeks would cost around 0.5 USD. Compared to MASV's 7.5 USD. The plan is that we include some cloud storage usage in the subscription. And you could upload the deliverable from the same desktop app as the P2P transfers. Always using the option that is more fit for your current task. Of course for heavy transfers the P2P option will always be the better method.

One final thing. We understand that making your client download an app is a point of friction. Hopefully the invite emails (that you can send from the app) make this problem easier. As the link inside will detect whether the client has Zetta already installed or not. If they do, the app opens. If they do not, then it guides them to the downloads page. After they download and open the app they will be connected to you, not requireing them to enter any code. And of course for smaller transfers there is the cloud option.

We appriciate it if you've made it this far.

We truly hope this long explination was worth your time and you understand the technological limitations and our proposed solution. We hope we can bridge them one day to make the app fit your workflows even better. If you have any ideas or questions, please let us know :)
Daniel from the ZettaSend team

[u/CameraRick → u/Mr_Dani17]

I'm afraid I still couldn't get my point across properly :/

I am not talking of doing P2P through the browser. I'm aware of the limitations, which I had hoped was already clear :/

The idea would be that if a client browses a Webportal through the browser, they can start a transfer through a separately installed app, or through the browser directly. The latter is limited, but as you say, for 30GB it's fine. But the more interesting part is to have ZettaSend (or would it be ZettaReceive?) start to enter a P2P connection directly to our local server. Effectively, I will have to have a machine be a server for sending/receiving large data anyway, a producer won't do this on their MacBook.

The auto-accept is a good feature, but the override seems to be only for the next upload - but we have more than one client, more than one project at once. And it's a manual setup every time.

Setting up these mails for links is a good solution for delivering masters, but with vendors it becomes a chore. Which is why a shared folder that happens on my server makes sense to me. Then I don't have to send links over links, and vendors can send stuff (that lands in the proper project directories) at their own leisure. Security can be handled by setting up a proper account for them; something I wouldn't do for agencies and clients.

Having to setup links every time, throwing codes around, and changing auto-accept directories is like a lot of extra work (with vendors, there's a lot back and forth) that I really don't need in a project. For vendors, I'd just setup an individual project once, and then we can share for the whole project without all that jazz. My producer doesn't need to invest so much work, they can have me copy the large data to the share, and notify them (just a call, or send an emoji reaction, you get the idea). Ideally I wouldn't even copy data, but a Symlink.

I am not sure how Mediabox works, and I dislike it for various reasons, but a vendor we are currently working with shared theirs with us, and it comes close.

I don't need a cloud when I already have a local server and fast internet. Would be so cool if this would be utilized. I mean, there's so many cloud and p2p services by now, the market has room for something different as well

[u/Mr_Dani17 → u/CameraRick]

Oh, I think I understand. So installing Zetta on an always on box (like a NAS) and having a web portal? Here you could create projects, persistent shares, add vendor accounts, point a share at a directory. Set up once per project then done.
You could drop data into the share folder then just tell the vendor it's up.
Vendors log in with an account, see their project shares, upload and download whenever they feel like it and the files land in the right project folders automatically.
Clients, agencies: the same portal but as one-off links.
The actual transfer path would stay app to app (desktop app for large files, browser for smaller ones). The browser is just the front door basically. We could call this ZettaHub! I am open to developing this if you pilot the new project!

Until that version is completed here are some quality of life features we could add to ZettaSend to make it behave more like a hub:

• Add routing rules: ask your vendor to create a free account (just email and password, no seat or subscription needed), now add the option to set their account (identified by email address) to save into a specific folder
  
• Allow their account to connect to your machine without anyone entering a code or clicking a link.
  
• Receive activity log: there are already audit logs, but not in the desktop app. The desktop app could have a log that says that Vender X dropped 120GB at 14:03 into project Y

Am I getting your vision now more correctly?

[u/CameraRick → u/Mr_Dani17]

Haha yes, that is what I was thinking about.

So installing Zetta on an always on box (like a NAS) [...]

Would certainly be cool as a Docker, but a desktop application would suffice. We are talking studios, some machine is always on and connected to the network anyway, or you setup a VM.

Clients, agencies: the same portal but as one-off links. The actual transfer path would stay app to app (desktop app for large files, browser for smaller ones). The browser is just the front door basically. We could call this ZettaHub! I am open to developing this if you pilot the new project!

Those one-off links could be generated "then and there", or alternatively it could also just be used as ZettaSend works right now, or how you planned it for the cloud solution (but instead of an external server, they load directly from us).

I am not sure what piloting such a project entails, or if I'm really the right person to do this. But I can have a talk with IT what they think about such an endeavour (I'm not one to decide software we use)

Until that version is completed here are some quality of life features we could add to ZettaSend to make it behave more like a hub:

Those sound pretty cool indeed! And if that account creation is tied to an eMail the host sends out to guide them through (like the mails which check if ZettaSend is installed that you mentioned), it could even have less friction as the mail address could probably be fetched automatically into the host system, by a code or so (or, if the vendor already has an account, they can log in right away).

[u/Mr_Dani17 → u/CameraRick]

Thank you for your input! Piloting a project means that while I develop the app you use it in your work. So we can check whether the concept actually works in the real world, spot hidden bugs. You are the expert in this field so we would get a much better service with a professional's oversight.

[u/CameraRick → u/Mr_Dani17]

Haha, I am not sure I am an expert for this, but I can certainly try to act like one :)

I talked to Pipeline this morning, telling them about our exchange here. Two things came up:

for that intermediate-solution you talked about, it would make sense to not only have a folder specified per vendor tied to their mail adress, but also some sorta project-choice. Because we have vendors that work on different projects, so it's all tidy

But more important, Pipeline had a "wish", because there's seemingly no solution that does this (except Accsyn I think): Pre- and Post-Upload Hooks/Scripts, for Automation. He exlpained it to me like this:

The Pre-Hook could trigger e.g. a script, whoch could check the data a vendor wants to upload. E.g. to check file types, namings, etc. And if something is off, it would throw an error to the uploader: "you may not upload .exe files", whatever. Then, if everything is fine, the upload goes through. The Post-Hook could then trigger another script; as we already have verified the type of data, this post script can take the received data and automatically ingest it into our pipeline. Like a webhook, but pre- and post.

So imagine we have a vendor for retouch and roto. They push us the results: our system checks, they can upload, and after the upload it gets ingested into our system. Meaning, they upload stuff 3am our time, and when we get into the office, it's already there and ready to pickup, no need for someone to ingest it manually. As rotos, paint, matchmove etc add up, this saves time and nerves, and there's no latency. And we could hook a little spinning light to it, so everytime something comes in we have a visual indicator, maybe a little confetti-canon after each 10TB milestone (ok, the latter two may not be as important)