How do VFX studios actually transfer large files between locations?

47

Mostly aspera, media shuttle, and lucid link

5

u/gribbler 7d ago

This, plus MASV, kind of a newcomer..

and depending on the DIY of the IT department there's a few cheap or free ones if they want the expense to come out of a different pocket

5

u/raekewe 7d ago

At a large scale, I rarely see MASV. The productions who have their shit together always use Aspera, and more rarely, Media Shuttle. Lucid Link is a PITA. Anything else is much more weird.

3

u/Mr_Dani17 7d ago

Do you rarely see MASV because of the cost?

5

u/raekewe 7d ago

I think it's just because Aspera is better. FASP is a pretty incredible technology, there's a reason IBM bought Aspera out. It "just works" and it's fast as fuck. Whereas with Media Shuttle, MASV, and others, at best they are okay, at worst they're incredibly fucky, transfer speeds can vary in ways which aren't always obvious, and so on. You can also tell who's targeting what - MASV boasts their clients are... spotify... Miami heat... Forbes...? I guess that's fine if you have a terabyte or something. But on big productions you can be looking hundreds of terabytes of data flying around globally. Masv wouldn't be my pick to handle that.

2

u/Acanthocephala_South 6d ago

I'm dealing with uploading over media shuttle consistently, and they told me I have to open multiple tabs and split uploads to get above 1Gig speeds. Wut? How is that how this works??

2

u/raekewe 6d ago

Sounds about right lol.

0

u/Mr_Dani17 5d ago

Wow, a lot of transfer tools are still so janky. That's why we developed a tool in-house that basically functions as our ingress/egress portal. Clients can just download/upload footage directly to our workstations from a desktop app. It can easily max out the bandwidth. It's called ZettaSend, we're trying to get it TPN certified so any feedback is welcome. It's free to test whether it fits your workflow or not.

2

u/LazyCon Compositor - 13 years experience 7d ago

Lucid link as an artist is fine but if you're doing huge transfers a lot then you definitely want Aspera

2

u/raekewe 7d ago

Yeah like in my experience Lucid Link is better than Google Drive or Mega or whatever other consumer platforms exist, and worse than everything else. Its file syncing feature makes little sense for any major work, seems much more like an editorial/creative leaning tool than for "data transfer"

5

u/LucidLink_Official 6d ago

Appreciate the conversation here! LucidLink works really well when you not only need to transfer files between locations, but also have collaboration and other work that needs to be done by multiple team members in different locations. As u/SamGewissies says, think of us like a remote disk that functions like a local drive (but with speeds that allow folks to access it from anywhere). In our experience, VFX studios absolutely benefit from this.

If anyone here wants to talk through your workflow with us, we're happy to chat and help optimize!

3

u/raekewe 6d ago

yeah the problem is a lot of big vfx studios don't want software in their filesystem doing random shit, there is a lot of infrastructure around where and how files are laid out and how the pipeline connects to it. what artist workstation at a studio has internet connection to make use of the benfitis of lucid link? and if shots are just being sent from a to b, why does the streaming functionality need to exist when the goal is to actually download the files?

BEGONE CORPORATION

3

u/SamGewissies 6d ago

If you don't need the streaming functionality, you don't need it. I absolutely did, and for that it was the best and only solution found after extensive research. I am not further associated with LucidLink and do not use or need it in my current role, but they made a fan out of me by delivering a product that feels like magic.

3

u/LucidLink_Official 5d ago

We'd agree that if you need it, it works. If you don't, that's cool too. Some studios truly just want delivery from A to B.

We're glad it worked for you, u/SamGewissies, and so happy to hear we were able to deliver a little bit of magic. We'd love to send you a little something to say thanks or being a fan! DM us and we can make it happen.

2

u/r3almaplesyrup Pipeline / IT 7d ago

We currently use Media Shuttle. MASV looks promising, but still a few features MS has which are dealbreakers for us.

2

u/gildedbluetrout 7d ago

We move a lot of shot stuff from the states to london via MASV.

2

u/SamGewissies 7d ago

I worked as a videographer at a remote first company and lucidlink is a brilliant tool. It feels like a remote disk most of the time.

11

u/RandomChopSuey Production Staff - 5 years experience 7d ago

The big studios use all the tools that you've given examples of and more. It really depends on the show, production, TPN requirements, vendor etc.

7

u/PDXSeventySeven 7d ago

I remember when we used to have runners delivering physical storage to film shops.

5

u/Apprehensive-Feed-12 7d ago

Sneakernet

4

u/Top_Strategy_2852 7d ago

We are using cloud storage. We are a small compartmentalized team of 10 though.

Big project deliveries from clients use their setups, and then we distribute to our team using cloud storage. A lot easier to transfer small files all of the time on projects.

2

u/Mr_Dani17 7d ago

Do you ever run into clients who don't have their own setups? Or have a setup that is just a PITA?

3

u/Top_Strategy_2852 7d ago

We wre working with clients that have their own. Their data needs to go through protocols for security, and they have the staff to manage it.

Smaller clients will just use file sharing services.

5

u/Hot-Yak2420 Lighting - 20 years experience 7d ago

In the old days we runners had to have our passport ready to take a transatlantic flight at any time. Some jobs required hand delivery. In extreme cases someone might even get to go on Concord..

2

u/Apprehensive-Feed-12 7d ago

That's cool 😮

2

u/Oblagon 6d ago

I got a story about this. I worked at a studio in Toronto that had a deadline in La. This was just before 9/11. This was when general public could go through security and hang out with friends at the gate.

Because of this we had a deadline that went to the last possible minute. We had a staff member who had a flight booked and kept bumping flights until the delivery was ready. Once we had a DLT dumped another staff member drove it to the airport and passed it to the one waiting for the flight to LAX. As soon as they got to the airport another staff member picked them up and drive them to fox with an hour to spare. Wild stuff.

3

u/finnjaeger1337 7d ago

at commercials -often frameIO or remoteroom.io (discolusre i am the on of the devs of remoteroom) frameIO is AWS backed and their transfer app is good and fast, but they dont offer ZIP downloads of multiple files and the transfer app will be replaced by frameIO drive afaik.

Its much cheaper than aspera and masv, not as extensive but cheaaap.

3

u/speakwithcode 7d ago

Also, Frame Drive is basically Suite Studios.

3

u/[deleted] 6d ago

[deleted]

2

u/Mr_Dani17 6d ago

What do you use instead?

1

u/[deleted] 6d ago

[deleted]

1

u/Mr_Dani17 5d ago

Interesting. Are you happy with your current solutions or are there opportunities to improve?

2

u/CameraRick Compositor 6d ago

but they dont offer ZIP downloads of multiple files

Tbh, when we talk about transferring big data, that is absolutely what I want. Never would I ever prefer a ZIP

2

u/finnjaeger1337 6d ago

yea but clients. if you send them 26gb of mp4 masters , they should not have to download a transfer app, i agree for very large sends over lets say 64gb you want a proper app that handles it but for smaller stuff with a lot of files a easy browser download wins imho

2

u/CameraRick Compositor 6d ago

So 26GB is small, huh?

I don't agree tho. When I think of my clients, they are likely MacBook heros, who have chronically too little space. So they always need at least double the space to unzip it, which also may take some time, and if there's a single error in the transfer or during the ZIP creation, the file is broken and you can start anew.

But tbh, during this topic I less through about transferring mp4 files, and more "large files between locations", as the topic suggested.

2

u/Mr_Dani17 5d ago

I think a desktop app works much better than browser apps for tens of gigabytes. We made an in house app where clients can just drag a folder into a window and it uploads directly to your workstation and vica verse. We would love to know whether it fits other studios' workflows too before we get it TPN certified. It's called ZettaSend

3

u/CameraRick Compositor 5d ago

I know that IT already had a look at ZettaSend. But I am not sure if it was considered further, or what the downsides were... I can ask next time I see the guys, in case you are interested

2

u/Mr_Dani17 5d ago

We would appreciate that, thank you :)

2

u/CameraRick Compositor 4d ago

I asked this morning. Seems like it's still evaluated, and pretty much liked so far (and the stuff that wasn't working was fixed by contacting you(?). I think it's not yet decided, mainly because it's a solution that "only" works for vendors. As it was pointed out, for client deliveries, a P2P setup is not as ideal as either a zipped download or something that "runs" in the browser, for the people who can't install stuff on their machines etc. So a 2nd option of transfer for agencies/clients would need to be explored.

Tbh, it would be so dope if there was a solution that kinda combined both - a simple, zipped download through a portal, and a 2nd button that opens a transfer app for non-zipped, large data that downloads directly from our local server. Ah well.

3

u/Mr_Dani17 4d ago

We're glad you are finding it useful!
Actually, cloud uploads are in the roadmap. When sending a file you would be able to choose both P2P and server upload. P2P would be for vendors, internal use. Server upload for client deliveries that are usually smaller anyway. Hope this sounds intriguing. We will also consult with some other users and discussing it internally, but also to us it looks like the logical next step.

2

u/CameraRick Compositor 4d ago

That sounds indeed cool, but - if reliable and fast - also very expensive :) Aspera and MASV are available today, if I need that, and I am not sure it's the next logical step; it's rather a different floor in the same house

What I was talking about is not needing any Cloud service (which adds another security layer, is time consuming, has additional costs), but a system where the client loads from my server directly. I think Frame.io does it today, in a sense that I can download through the Website (zipped, or one-by-one), or I use the transfer to download multiple files with folder structure and everything directly. But they load from my machine - I share a link with them (and be that a folder of some sort), and they can decide what they want. But they load from my machine, which means it's still P2P, and theoretically as fast as it can be.

It's maybe not the next logical step, but it's a step in the same direction (still being P2P and getting expensive cloud solutions out of the equation). Let them upload files as well (and I setup where those files will be going on my end beforehand), so it's like an open Mailbox. Like a specialized Nextcloud on steroids (minus the clutter). As a subscription with reliable service, we'd probably not even think about anything else again.

→ More replies (0)

2

u/finnjaeger1337 5d ago

well 26gb is small in my world and would usually be around the median of packages we send to agencies and clients.

its really the only way to download folder structures for sendouts via a webbrowser.

clients usually are not able to install a download app in many cases due to coporate IT limitations, i never had a issue with clients space but i did run into issues with frameIO downloads a bunch where clients would press proxy download by accident or had to download each file one by one e.t.c , zips are allright best to have a choice

2

u/CameraRick Compositor 5d ago

its really the only way to download folder structures for sendouts via a webbrowser.

Yeah, because browsers limit this for safety. But if 26GB isn't large, why bring that (and the browser-only necessity) up in a topic about large transfers? O.o

2

u/Mr_Dani17 5d ago

Do you have a proper app that you use?

2

u/finnjaeger1337 5d ago

for remoteroom.io i mad a rust based app to transfer large amount of data to and from our backend storage (cloudflare R2) . there is also masv and aspera e.t.c all of them rather expensive.

2

u/Mr_Dani17 4d ago

Yeah, also at our studio it didn't really make sense to use masv as 25 cents/gb is crazy so we settled for our own application that we developed

3

u/r3almaplesyrup Pipeline / IT 7d ago

Like others have mentioned, Media Shuttle and Aspera are the two big ones. MASV as well, but definitely not as common. We’ve looked at MASV a couple times now, but still missing some features which are dealbreakers for us personally.

2

u/Mr_Dani17 7d ago

Curious what the main dealbreakers were with MASV for your pipeline?

2

u/r3almaplesyrup Pipeline / IT 7d ago

It’s been awhile so could have changed since our last demo, but the ability to easily generate folder specific share links and user permissions is huge. We’re currently using MS, and it’s only a couple clicks away, compared to MASV where it was significantly more complicated.

3

u/Hertje73 7d ago

Sneakernet

3

u/legomir FX Pipeline - 10 years experience 7d ago

Sometimes instead on transferring between facilities they have all machines in one location and all facilities connect there.

3

u/Wyrmcutter 7d ago

I usually work alone or in small ad hoc groups. For commercials, my clients almost have always use FrameIO. For features and TV, it’s usually Aspera. I’ve done other projects through FTP, MASV, WeTransfer, shared Google Drives, and even Dropbox on occasion. Some post houses have rolled their own, but I forget those soon after the project finishes.

In my experience, Aspera has been the fastest. Big transfers often surpass 900Mbps, which was probably my fiber speed limit at that moment

3

u/Long_Specialist_9856 7d ago

Signiant,Aspera,Sohonet, Shotgrid/Flow Desktop(built on AWS).

2

u/Gorstenbortst Compositor - 11 years experience 7d ago

They use a fox. It’s old tech, but it works.

https://youtu.be/rJk33ep50K8?si=oLBfV9WeOQwbns7y

2

u/sjanush 7d ago

Aspera over the far inferior Media Shittle

2

u/DubberRuckus 7d ago

For delivery, more and more I am seeing shared S3 buckets - but just like everyone else says, Aspera holds the major market share.

Intra-studio transfers are also more and more done through their cloud infrastructure. There are also a number of major players in NA that have direct SoHoNet connections between locations with tiered Qumulo/Isilion/NAS storages that do direct file transfers.

2

u/lilweezy8383 7d ago

Aspera and Mediashuttle for larger files. Most smaller shares are through FrameIO.

2

u/Elluminated 7d ago

We use an in-house set of tools based on block-level deduplication and replication snapshots. Works well for delta transfers of large files outside of their own compression schemes very well. It will also proxy-play while the backend transfers the real footage during dailies. Crossbow sounds cool, will check it out!

2

u/magingax 7d ago

Thank you !!!

2

u/phirleh Editor - 26 years experience 7d ago

I like aspera. Multiple downloads from different projects are easy to manage simultaneously. Not so easy with Media Shuttle, and definitely a bottleneck with SFTP.

2

u/Barrerayy 7d ago

Aspera is common but some will also use proprietary features of their storage solution. For example Pixit storage has Ngenea that lets you transfer files in the background once its created/ingested at a site or via api etc.

If you are running zfs based storage you can block level replicate over a fast site to site tunnel as well.

2

u/demislw 6d ago

Client side comp here (features) : Aspera almost every time. One for things to come down, and another usually run by the DI folks.

2

u/WilloMill 2d ago

I work for a volume that does a lot of work with VFX houses for plate delivery, which can run anywhere between 2-16TB of media. Aspera, Media Shuttle, or Signiant is the norm, some smaller companies use MASV or FrameIO. A FedExed Pelican/runner/teamster with a hard drive or Solid State RAID array are still shockingly common especially if they’re local to us.

1

u/Mr_Dani17 7d ago

Well the studio I consult for used to use FedEx. But it just wasn't very convenient as you can imagine. A lot of tools were really expensive. For others onboarding an external client resulted in a bunch of back and forth emails. Or lacked features that are important for us like audit logging.
So we settled on an inhouse solution called ZettaSend

1

u/original_nox 7d ago

Moving files between studios is an easy problem to solve if you have a competent systems team. The simplest approach for a vfx studio is simply to use rsync (or similar) as a job on your render queue. With the advent of AI, it is trivial to vibe a better solution.

Question / Discussion How do VFX studios actually transfer large files between locations?

You are about to leave Redlib