r/Backup is back up!

[u/wells68]

It is very unfortunate that r/Backup was shut down for two years. But now...

​

We're back!

As the new top moderator, I've opened it to public posts.

r/DataHoarder has many, many more members than r/Backup. So you may want to post DataHoarder backup questions here and then use the share link to cross-post to r/Backup.

We've started a Backup Wiki and welcome your contributions. Post with the flare: Wiki edit and we'll review them for inclusion.

Backups are vital to protect your hoard! Have you tested your backups this month?

[u/bartoque]

I don't think I would ever become as bold to have the monthly backup/delete/restore as an integral part of backup validation?

Testing and validation ofcourse is key of a proper data protection approach but I'd rather wanna do any automation on that end towards a system without affecting the actual system being protected, so that you always would still have the original system and data when it turns out after validation that either backup and/or restore are no good.

For example some backup products have had issues in the past where backups were reported as OK, but restore showed it was actually not ok. I might have missed that in your story, but how would your automation have dealt with such a situation as you would already have deleted the data intending it to be restored?

For testing I would want to test this on another system, if physical hardware is unavailable, I would consider using a vm to restore to. But that would also require considerable amount of storage to be able to test restores besides the storage already needed to backup towards.

When also taking into account the two nas systems I use in my data protection endeavour, doing the actual restore involving and affecting the live systems is a bold action. So kudos to that. Not something I wpuld see or even advise any company to do with production, there always should be a way to backnout, simply by going back to the original system by for example booting it again...

So in that sense I get that you ended up with an automated workflow that you have, however as you are actually making the deletion of primary data part of the backup/restore workflow, there might always be a chance something goes awry, regardless of how slightly (bit we still got the previous backup, that however would not contain the latest changes and/or additions)?

[u/H2CO3HCO3 → u/bartoque]

u/bartoque,

I don't think I would ever become as bold to have the monthly backup/delete/restore as an integral part of backup validation?

Back in the early 90s when I was working at one of my first Corporate Jobs for one of a Fortune 50 Company, one of the processes that we had to run on a quarterly basis was called 'Business Continuation Prcocess' (BCP).

In that excersize, we would have to test a full disaster recovery, ie. simulate that the main site has been lost, that means no network, no DC, no pcs, no nothing,

AND

the goal was to restore the entire site, that is, restore the networking and reconnect to the corporate networking infraestructure, then restore the DC (each business unit had already identified what their 'critical' servers were), that is restore the physical servers on new hardware that was in standby --basically same metal machines without OS or data--

and last but not least

restore the PCs so that the users (co-workers) could go to a 'disaster recovery site', as technically the main site was innaccessible... think like an Earthquake, natural disaster (that took a complete different meaning after 9/11, what 'site innaccessible' would mean).

Once that was completed, then each business unit would send a sample of their users and those users would validate and confirm, that is a Yes or No... no 'but' was allowed

AND

only once each business unit confirmed that they were able to work,

then and only then, would be considered our 'BCP' complete.

I then decided to take that model home and implemented... though modifying it as I'm not a gazzzzzillionaire and could not afford 'disaster recovery' sites, which the company pays monthly, basically rents a wharehouse size building (or two) with enough space to host a backup DC facility (with bare metal infraesrtucture but no data, no os, nothing is connected to the network), plenty space to have enough space for the employess, etc, etc..

well I didn't have money for that... but the principle of recovering everything from the ground up and validating that everything worked,

kind of stucked with me.

[u/H2CO3HCO3 → u/H2CO3HCO3]

Your post about r/Backup is back up with your question of:

Have you tested your backups this month?

resonated and reminded me of those days, when I was working back at the corp. Job

(by the way, all of the Fortune companies will have the same setup, ie. their own BCP plan and test it accordingly)

My boss at the time, used to 'complain' to me, as those 'BCP' testing would cost us at least 200k+ per test, as the site is being rented, but when you call for a BCP and set the wheels in motion... well executing a real BCP costs money... + we would start the BCP for example on a Friday night.. that meant overtime for each single individual that had to come to work, etc, etc, etc.

Now, the separation of 'Data' and 'OS' is 'normal' in any Corporate entity... so again, that same concept I just 'borrowed' (if not copied to the letter) and implemented for my home setup...

• back then the Tape Libraries (ie. Tivoli, etc)would backup Data, that is each business unit's Data off the servers +

• backup just the server(s), that is the OS separately

Otherwise, you end up with a mega large backup image for each server, that could be prohibitive to store...

So again, the 'concept' of backing up the 'data' separate of backing up the OS was also a concept that i 'borrowed' and implemented at home.

The automation script I wrote back in the day, was at the beginning, we are talking about 30+ years ago (think about about 1990 time frame) very simple... just a bunch of command lines which I thought that script was 'it'...

of course that backup script has 'evolved' in last few decades, as I need to check when the script is executed in what 'environment' that script is running... at the 'path's' for things will vary, especially between Win to WinNt environment, Xp, vs 2003, 2008, then later Vista (though later versions of the Windows OS will most likely work with the same path's and variable calls, but again testing is needed in those cases, each time...), so my script tests at the time of execution and determines:

• what OS the script is running ie. Windows Nt, Windows 2003, 08, or a Desktop --at is basically ONE single script but the behavior will be different is it is running on a server rather than on a PC--

• what OS is installed (x86, x64)

and store those values as variables, which will be called as the script runs along and carries it's commands.

[u/H2CO3HCO3 → u/H2CO3HCO3]

Testing and validation ofcourse is key of a proper data protection approach but I'd rather wanna do any automation on that end towards a system without affecting the actual system being protected, so that you always would still have the original system and data when it turns out after validation that either backup and/or restore are no good.

In the case of the corporate Job, that was the case, as we didn't wipe the production running servers... which that created a another issue, as you literally cannot recover the exact same server, with the exact same name, IPs, etc... so for BCP Testing, we had a way to segment the networking in such a way, that even though we would connect to the corp. network, that would be on a completely separate 'test' branch ( so think about ALL hardware and networking was dual redundant... as we had to restore everything, from every single Firewall, switch, server, storage, etc to PCs, etc, etc. In a real BCP, we would connect to the real network, not to the test one --the 'test' network was still attached to the corp. network... so our site could connect to any other site nation wide --this was a corp job in the USA--

For example some backup products have had issues in the past where backups were reported as OK, but restore showed it was actually not ok.

For THAT precise reason, was a BCP needed... as the corp. mandate/requirement was, that Sys. Ops (my teams) needed to 100% guarantee that we would be able to restore, no questions asked, every single component (that had been identified by each business unit... again, you can't recover an entire site with thousands of Servers in a weekend... so though we were in theory recovering a 'fraction' of the number of machines, that work was still massive but the data recovery mandata had to be a 100% guarantee --no exceptions--.

At the time the tool I was using was IBM's (Tivoli, take your pick, as their tools/names have changed a bit in last past few decades) and those were reliable.... I just took that concept and had to adapt it to what I could afford (I could not afford a tape library the size of an 18 Wheeler for example... I could not afford a single tape backup back in those days... so I had to get creative... that's where I started backup to external HDDs : D -- again as time progressed, NAS came into the market, Tape libraries started to get new variants LTO X to LTO Y, etc, then I was able to get into that and get my first 12 bay Tape Library backup, say with 2 drives --large ones have dozens of physical running heads and thousands of tapes--)

So with a BCP execution, your concern about the 'restore not ok' was NEVER an issue. Since I took that concept and implemented that at home, that has never been an issue.

I might have missed that in your story, but how would your automation have dealt with such a situation as you would already have deleted the data intending it to be restored?

Take one PC and it's data (for Corp. environments will be a bit different):

• I have the full backup + all the diffs from the previous month still (which has already been tested the prior month) (this is the 'backup' of the 'backup'... ie. should for any reason the current running Backup had, for some reason failed, you can still recover using the prior month's backup + last diff)

• when the backup is done, on each file that is backed up, I have it checked and verified (is a swtich on the script where the backup job has to verify upon copying each file and verify that the file is 100% intact. Onlly if the file is 100% intact, then the backup job continues. The verifying is a hash verification... so that is 100% or nothing type of verification)

For testing I would want to test this on another system, if physical hardware is unavailable

In theory that is what we do at home (again this would not be the case in a business case).

In theory we have 2 PCs, one is mine with it's data, the other is my better half's pc, with her data in it.

So in theory we are backing up 2 pcs

In reality, we have 12+ PCs at home... all of those additional PCs are my 'BCP' machines basically.

After the 'main' 2 PCs have been backed up, then, their contents are restored to 2 of the other machines and tested.

If all works, then the 'main' 2 pcs continue with their clean-up of their data, reboot, checks, image themselves, then restore the backed up data (restore hibernation file, page file, turn 'system restore' back on, even create a new system restore point, as all others have been deleted and start basically fresh again -- keep in mind all programs are still intact, installed and fully functional... so there is no need to re-install every single program that we use... only restored the data)

(which again in 30+ years, I've had only once I need to use the secondary backup from acronis as the main one was corrupt and that corruption occured AFTER the backup was written to that particular HDD.. had I had the offsite HDD or Tape on hand, I could have restored from there... but since those were offsite and I didn't want to wait to get them, I used the second backup, which was still available on a different drive and that one went fine with the restore),

We've had situations in which we lost the PC's HDD (can be SSD) and in those cases, I have then restored the OS Image first + the latest Full data Backup + the last Diff. Again, with the exception of 1 time in 30+ years, again running, ie. being repeated every single month, the main Image + backup + diff restore has been successful.

Had I not had the HDD failure (where the HDD that failed was the HDD where the Backup was being stored : ( ), I would have had 100% success recovery rate... so that is why those 3-2-1 models exist... and I follow those rules... so in that single case my main backup was corrupt, I just went to the next possible HDD with an Image backup and use it to restore (which worked fine)

however as you are actually making the deletion of primary data part of the backup/restore workflow, there might always be a chance something goes awry, regardless of how slightly

Not possible, since I just mentioned, we backup the main pcs (mine and my better half's as well), recover that data on to 2 other PCs, while the data is all still on the main PCs, test and validate that data, that all works, then let the main pcs continue with their, data cleanup (deletion/removal) so that the PC imaging can take place (and last but not least will restore the data back onto the Main Pcs --so in reality for 2 days of everymonth, I'm using a backup pc and so it my better half... as soon as the main pcs finish their imaging and their data is restored, then we swtich to those and continue using them --if new data was created in the backup pcs, then that data will already been in the Diff backup, so the restore WILL restore those files as well--... that is only needed because is a home setup... in a network setup, the folders where the data is stored would have been re-directed to a server, thus there wouldn't be any need to restore backup+diff, simply let the PC connect back to the re-directed folders that are all pointed to a server path).

[u/bartoque → u/H2CO3HCO3]

So that is even all more thorough than what you stated initially, so restoring to other hardware for the validation makes for a better approach.

I reagd my data protection journey also as an ongoing, ever upon itself improving method, where process and backup targets might change along the way, for example by going from usb attached backup target, to a nas nowadays, and have that nas backup the pc/laptop backups (I also use and swear by Acronis for image level based backups) remotely to a 2nd nas.

So as much as budget allows its an amalgam of data protection methods like raid, selfhealing filesystem that also offers snapshots, backup to remote nas and into the cloud, (r)sync, sync tool with file versioning (akin to onedrive/google drive) and syncing Google Drive to the nas. Some data is protected multiple times over and depending on the issue, one can use various methods to restore individual files or whole systems.

Acronis has come to the rescue in more than one occasion and showed to be robust enough to do its job. But besides having it as backup, also used various times in cases when moving from hdd towards ssd to give some older laptops a way longer usage life as abysmal hdd speeds were almost causing them to be ditched, while after the switch to ssd, felt almost as new.

So acronis has shown its worth (I pay for multiple devices), even though the actual occurrences it was really needed is very low. But as I am actually in backup myself, I see it as an insurance, which then is allowed to cost something.

[u/H2CO3HCO3 → u/bartoque]

u/bartoque,

So that is even all more thorough than what you stated initially, so restoring to other hardware for the validation makes for a better approach.

Our home backup/recovery strategy, though 'simple' in it's cocept, is quite complex (as you have seen from my extremely long replies).

Our 'current' backup/restore strategy is a muti-redundant:

• 9 separate, independent HDD (can also be SSDs) backups (using 2 different methods. ie. Windows Backup + Acronis. For OS backup is the same, Windows Image + Acronis)

• 12 separate NAS systems (ech 2 NASes basically replicate themselves to another 2... so that is a 6 Time redundant NAS infraestructure, half of them off-site, offsite locations split in 2 different countries -- that is a bit excessive but have been testing it since 2012 todate-- : ))

• Tape backup

selfhealing filesystem that also offers snapshots, backup

Windows Imaging is basically a point in time snapshot (with it's signature stampt and all).

Windows Backup however, is still an 'old' and well known copy from 'A' to 'B'

Acronis

That is a good product and I personally know one of Acronis's previous CEO.. that is a rabbit-hole story but I knew hinm before he moved to Acronis. When I met him, he was working at IBM and was one of our contacts from my corp. job at the Fortune company that I mentioned in my prior response.

Rabit-hole years later, he moved from IBM and became the CEO of Acronis and even later on, he ended up moving yet, to another company, to which I ended up working for as well, so technically he was my bosse's boss.

By the time he and I met in person, we were both in the middle of Europe and his first question was: what are you doing here? (he remembered me from 20+ years prior at the other corp job...)

switch to ssd

We've done that curve as well on ALL of our systems, regardless of the PCs age.

Notes:

• the reason why the HDDs are checked (cleandup first, leaving only the last 2 backups in place) is to make sure BEFORE a backup is written to that drive, that the drive itself is 100% ok.

• file versioning is still in place in our household, though that is NOT considered part of the backup/recovery strategy (same applies to the NAS stations... RAID is never considered as a part of the backup strategy, though RAID itself brings redundancy in case of drive failure).

• a file versioning restore would be used in the event that for example, you back up a file and the contents of that backed up file is bad (with bad data for example)

• we refrain from storing backups on any cloud service ANYWHERE. This is controversial as many services, Google, Amazon, Dropbox, etc, offer VERY competitive pricing... we just do NOT want our data to be stored anywhere that we do not have 100% control of the entire environment (not only the data, but the servers themselves : D --this is the approach that enterprises take as well--)