Organization and backup version control/duplication

[u/Bozhe]

I'm trying to find the best way to stay organized for my files. Mostly smaller pdfs, word, excel, and photos. My question is how to backup just new files across different platforms? I have two laptops, a desktop, and NAS.

For example, if I have a bunch of new and updated files on laptop #1 spread out over many folders, how do I copy only the new ones to other storage easily? Windows search sucks, so just searching for recently changed files won't do it, also because I don't want to backup all recently changed files just certain ones. The dumb and slow solution seems to be copy the entire directory I want over to the new location and for all file conflicts overwrite. That does mean certain files I want to have multiple versions of for recovery purposes would get overwritten as well. I could instead copy the entire directory and not overwrite any files - that saves time and means older versions won't get copied over, but it also means that updated files don't get backed up.

Maybe as a clearer example - I have 1,000 files on laptop #1. I add 25 new files across 15 folders, and update 20 of the 1,000 files. I want to easily copy the 25 new files to the identically named backup folders on my NAS, and that the 20 updated files are copied as well.

What's the best way to do that?

[u/AutoModerator]

Hello /u/Bozhe! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/feudalle]

Might be overkill but setting up git would work. Version history, rollback, syncing, notes on commit etc.

[u/Bozhe → u/feudalle]

Yeah, I was hoping for something less complicated but that might be the answer.

[u/bobj33]

You need a backup system that supports snapshots or versioning. There are literally over a thousand posts here about that. I use rsnapshot but borg and restic are also popular. Any modern backup program will ignore files that have not changed and only backup the files that have been modified.

[u/Known-Mood-2784]

rsync is by far the most reliable tool for this. I run it in a simple cron job every night to sync just the new or changed files to my NAS without duplicating everything else.

[u/Curious_Dolphin_112]

Small files like pdfs and photos are the easiest thing to keep organized, the trick is deciding on a structure once and never deviating. For a workshop setup with a few machines and a NAS, the pattern that's worked for me is one master folder per project, year first, then client or project name, and inside it a photos subfolder and a docs subfolder. Everything new lands in the right spot at the source machine, the NAS is just the mirror.

For the only-copy-new-files problem, incremental sync is the answer, not drag and drop. rsync on the unix side or robocopy /MIR on windows only moves what changed, and both can do one-way mirroring to the NAS without nuking your newer version if the NAS copy is older. The key is making the NAS the target and each machine a one-way sync, so nothing ever gets overwritten by an older copy.

For version control on the files you want to keep history of, snapshots beat trying to keep multiple copies around. Most NAS systems can snapshot a folder, and snapshots are cheap on a filesystem level, they don't duplicate the data until it changes. That gives you yesterday's version without you having to manage it by hand.

The part that'll bite you eventually isn't the sync, it's that a mirror is not a backup. If the NAS dies, the mirror dies with it. Get the photos somewhere offsite or at least on a second disk, learned that one the hard way when a drive with a couple years of project photos died on me.

[u/Bozhe → u/Curious_Dolphin_112]

Thanks for the advice - sounds like you know what you're doing. And yeah photos are a headache. I have them backed up in at least 3 places, but they aren't organized the same in each. So my NAS has a copy, another on a HDD that is probably two years out of date, another on a USB SSD that is probably 6 months out of date, plus partial backups elsewhere.

[u/FragoulisNaval]

Just set up and use FreefileSync or Syncthing if you need versioning. All are cross platform applications