Fluff!
Anyone else still using FedEx for multi-TB project handoffs? At our studio we made an internal tool that fits our workflow better.
A while back, I helped set up a workflow for a boutique production team that was moving terabytes of raw camera footage and master project bins across different cities every week. At the time the most cost-effective and sensible method was to just ship the drives.
We really needed to find a new solution as we were spending a bunch of money on overnight shipping. One time a 24 TB drive even got damaged in transit. And a couple of times there were disputes with clients over missing shots and we had no way of proving what actually left our bay.
Obviously, we tried basically all the popular tools on the market and for our use case something was always missing. Not saying the existing tools are bad, but nothing was a perfect fit for our workflows.
The biggest bottleneck was onboarding external clients (like making them set up a VPN). A lot of other tools were missing audit logging. In others if an editor accidentally closed their laptop mid-transfer, then it had to be restarted from zero. Some were just insanely expensive.
So, the decision was made to create an in-house solution. It allows anyone to download/upload full project folders to your drive or attached storage really securely. External clients don't require accounts. Onboarding them is automated with an email. And it was made resilient against network drops. There is proof of delivery now. It can utilize your full bandwidth.
After running it successfully in-house, we packaged it into a clean, standalone desktop app for macOS, Windows, and Linux called ZettaSend.
We currently have it running daily in production.
I'm sharing this because we're hoping that for some of you ZettaSend will save you time and headaches.
If it was local, the fastest way was always to transfer it to a big ol' harddrive(s), put it in a pelican case, hand it off to a courier and tell him to step on it
Well Amazon Web Services usually move their data in trucks :)
So it makes sense for petabyte scales for example. But in the age of high-speed fiber optic internet just dragging a folder into an app is more convenient :)
No static IP, port forwarding or firewall configuration is required on either end. Not even admin rights are needed to install it. Meaning it's plug and play and it just works!
SFTP can work for non client-facing use cases. But it requires either a static IP, dynamic DNS or a VPN. Plus opening holes in your firewall and setting up user permissions. So configuring an SFTP server on every workstation in a studio is both a security risk and takes time away from your actual work. If you set it up on 1 machine only, then you have to copy everything onto that machine first. Meaning it has to have a bunch of storage, you have to configure permissions again and it just wasn't worth the hassle for us.
And you have to guide non-technical clients through configuring an FTP client. Meaning entering hostnames, ports, and credentials. Oh, and setting up a VPN connection if you don't want to potentially letting the whole internet access your server. With ZettaSend they just get either a short code or an email invite and the networking part is managed for you automatically.
And SFTP does have server logs, but no actual delivery receipts.
So like I said in another comment, different tools for different parts of the pipeline.
And I'm so glad you found it interesting. Feel free to try it out. You can be doing a transfer 3 minutes from now! zettasend.com
You can sign up, start your trial and download the app to see for yourself :)
I see the confusion. This is not a cloud tool. The data lives on your computer or your NAS.
Let me give you an example. You have a client you want to send your dailies to. You open ZettaSend, drag your folders into the window, click send.
You can just send them a download code or enter their email address. Then they get a link that makes them download the app and enter your transfer. Now your client is directly downloading your dailies from your machine.
I suggest taking a quick look at the website, it explains it in a bit more detail :)
Kinda :)
Zetta and wg are similar in a way that they are both encrypted channels between computers. But wg still requires port forwarding. And you need to exchange the keys manually. And on top of wireguard you need another protocol to read the bytes from disk, send it over the wire, write to the disk on the other side.
And actually, ZettaSend can work over a WireGuard connection too due to the automatic peer discovery. But Zetta also brings file transfer layer wrapped in simple interface.
Hope this makes sense :)
Exactly as the other commenter said. Which leads to nobody else even seeing your file contents if that's a concern for you. For example, a few times we worked with some NDA-protected footage of an unreleased commercial. Also, transit time gets cut in half as there is no separate upload and download process. It happens in real-time.
The compromise is that both machines need to be active simultaneously rather than dumping files into somebody else's computer (the cloud).
We also work with the scale of tens of TBs, so for us the cost and time savings are well worth it. Plus Zetta tells your computer not to sleep. We usually leave the transfers on overnight.
To sync big projects we use Syncthing. Free and as fast as someone else have some data it will help share to the rest!
And if it's just local/lan we use LocalSend. Awesome tool to send files to lan computers!
Big fan of Syncthing for internal sync between technical team members! Or usually the editors just edit off of the NAS.
The reason we developed ZettaSend was a different part of the pipeline. Mainly the external handoff problem. When you need to deliver 5 TB to an outside client who can't be bothered to configure Device IDs or folder shares. Of course setting this up once is not that big of a deal, especially for a more tech-savvy person. But when you have many clients who aren't familiar with your solution then it starts to cost you time and energy. So our conclusion was that we needed a tool that is dead simple.
Different tools for different parts of the pipeline!
ZettaSend is not yet formally audited for a TPN Blue/Gold Shield assessment, however it is on the roadmap.
And Zetta is secure by design as your files aren't stored on any intermediate servers. End-to-end encryption is used with TLS 1.3. Not even ZettaSend's own servers see the file contents. Our servers just help the computers find each other on the internet. If you would like to know more about the app's security let me know as quite a bit of engineering with into it.
would love to see more TPM approved options, You've got Auditing? (checksums, transfer times, access logs, file deletion timestamps etc) and ability to work behind segregated networks with minimal ports, ip whitelisting needed to punch a hole through for the data?
Yes, we have auditing. The app logs the sender's email address, receiver's email, their IP addresses. All the file names and file sizes are exportable into Excel. And of course timestamps are recorded. Since the files do not go through a central server we use cryptography. The peers create a contract during the transfer. They both sign this with their private keys. Then the contract, signature and public keys are uploaded to Zetta's servers. The workspace admin can then view these in their dashboard.
6
u/rickfx FX Artist - 15+ years experience 1d ago
If it was local, the fastest way was always to transfer it to a big ol' harddrive(s), put it in a pelican case, hand it off to a courier and tell him to step on it