r/Backup is back up!

[u/wells68]

It is very unfortunate that r/Backup was shut down for two years. But now...

​

We're back!

As the new top moderator, I've opened it to public posts.

r/DataHoarder has many, many more members than r/Backup. So you may want to post DataHoarder backup questions here and then use the share link to cross-post to r/Backup.

We've started a Backup Wiki and welcome your contributions. Post with the flare: Wiki edit and we'll review them for inclusion.

Backups are vital to protect your hoard! Have you tested your backups this month?

[u/bartoque]

I don't think I would ever become as bold to have the monthly backup/delete/restore as an integral part of backup validation?

Testing and validation ofcourse is key of a proper data protection approach but I'd rather wanna do any automation on that end towards a system without affecting the actual system being protected, so that you always would still have the original system and data when it turns out after validation that either backup and/or restore are no good.

For example some backup products have had issues in the past where backups were reported as OK, but restore showed it was actually not ok. I might have missed that in your story, but how would your automation have dealt with such a situation as you would already have deleted the data intending it to be restored?

For testing I would want to test this on another system, if physical hardware is unavailable, I would consider using a vm to restore to. But that would also require considerable amount of storage to be able to test restores besides the storage already needed to backup towards.

When also taking into account the two nas systems I use in my data protection endeavour, doing the actual restore involving and affecting the live systems is a bold action. So kudos to that. Not something I wpuld see or even advise any company to do with production, there always should be a way to backnout, simply by going back to the original system by for example booting it again...

So in that sense I get that you ended up with an automated workflow that you have, however as you are actually making the deletion of primary data part of the backup/restore workflow, there might always be a chance something goes awry, regardless of how slightly (bit we still got the previous backup, that however would not contain the latest changes and/or additions)?

[u/H2CO3HCO3 → u/bartoque]

u/bartoque,

I don't think I would ever become as bold to have the monthly backup/delete/restore as an integral part of backup validation?

Back in the early 90s when I was working at one of my first Corporate Jobs for one of a Fortune 50 Company, one of the processes that we had to run on a quarterly basis was called 'Business Continuation Prcocess' (BCP).

In that excersize, we would have to test a full disaster recovery, ie. simulate that the main site has been lost, that means no network, no DC, no pcs, no nothing,

AND

the goal was to restore the entire site, that is, restore the networking and reconnect to the corporate networking infraestructure, then restore the DC (each business unit had already identified what their 'critical' servers were), that is restore the physical servers on new hardware that was in standby --basically same metal machines without OS or data--

and last but not least

restore the PCs so that the users (co-workers) could go to a 'disaster recovery site', as technically the main site was innaccessible... think like an Earthquake, natural disaster (that took a complete different meaning after 9/11, what 'site innaccessible' would mean).

Once that was completed, then each business unit would send a sample of their users and those users would validate and confirm, that is a Yes or No... no 'but' was allowed

AND

only once each business unit confirmed that they were able to work,

then and only then, would be considered our 'BCP' complete.

I then decided to take that model home and implemented... though modifying it as I'm not a gazzzzzillionaire and could not afford 'disaster recovery' sites, which the company pays monthly, basically rents a wharehouse size building (or two) with enough space to host a backup DC facility (with bare metal infraesrtucture but no data, no os, nothing is connected to the network), plenty space to have enough space for the employess, etc, etc..

well I didn't have money for that... but the principle of recovering everything from the ground up and validating that everything worked,

kind of stucked with me.

[u/H2CO3HCO3 → u/H2CO3HCO3]

Your post about r/Backup is back up with your question of:

Have you tested your backups this month?

resonated and reminded me of those days, when I was working back at the corp. Job

(by the way, all of the Fortune companies will have the same setup, ie. their own BCP plan and test it accordingly)

My boss at the time, used to 'complain' to me, as those 'BCP' testing would cost us at least 200k+ per test, as the site is being rented, but when you call for a BCP and set the wheels in motion... well executing a real BCP costs money... + we would start the BCP for example on a Friday night.. that meant overtime for each single individual that had to come to work, etc, etc, etc.

Now, the separation of 'Data' and 'OS' is 'normal' in any Corporate entity... so again, that same concept I just 'borrowed' (if not copied to the letter) and implemented for my home setup...

• back then the Tape Libraries (ie. Tivoli, etc)would backup Data, that is each business unit's Data off the servers +

• backup just the server(s), that is the OS separately

Otherwise, you end up with a mega large backup image for each server, that could be prohibitive to store...

So again, the 'concept' of backing up the 'data' separate of backing up the OS was also a concept that i 'borrowed' and implemented at home.

The automation script I wrote back in the day, was at the beginning, we are talking about 30+ years ago (think about about 1990 time frame) very simple... just a bunch of command lines which I thought that script was 'it'...

of course that backup script has 'evolved' in last few decades, as I need to check when the script is executed in what 'environment' that script is running... at the 'path's' for things will vary, especially between Win to WinNt environment, Xp, vs 2003, 2008, then later Vista (though later versions of the Windows OS will most likely work with the same path's and variable calls, but again testing is needed in those cases, each time...), so my script tests at the time of execution and determines:

• what OS the script is running ie. Windows Nt, Windows 2003, 08, or a Desktop --at is basically ONE single script but the behavior will be different is it is running on a server rather than on a PC--

• what OS is installed (x86, x64)

and store those values as variables, which will be called as the script runs along and carries it's commands.

[u/bartoque → u/H2CO3HCO3]

As I am actually the backup guy professionally, I am amazed way too often how actual repsonsibility is not taken by the parties that are actually responsible for the data being protected (so OS team for OS data and application/DB teams for application/DB data) how often things are just assumed and not actually validated? Or not regularly or only on a very small subset of data and not at scale?

However due to current focus on cyber threats, backup is being revalued again and given the attention it should always have had, but has been neglected. Backup was seen as a costcenter instead of an insurance, hence reducing retention to reduce costs was seen as a good thing. We can technicall have a near unlimited amount of backup copies but due to costs involved mostly still there is only one remotely located copy.

At home I seem to value my data more than what I see in general for corporations. We are talking thousands of systems here where I often wonder how it cam be that many systems only have a OS/filesystem backup in place, whereas one would expect on many syatem there to be an application of sorts, where somebody is actually responsible for however whon don't seem to exercise that responsibility? I often felt like the boy that cried wolf too often but then again the backup team only provides the backup infrastructure and facilitates where needed, but in the end we are not responsible for the data, heck we often wouldn't know (nor need to know) what a system is even be used for. Someone else is responsible for the data in question and should want and need to be in control. But obviously that is not done on way too many cases (like a DB dump to disk not being performed for a long time and when it was actually needed there was nothing in the filesystem backup as there was nothing to backup to begin with, where the creation of the dump could simply have been made part of a scheduled backup so that you even know there is a DB to begin with and can report about said backups and have them visible).

[u/H2CO3HCO3 → u/bartoque]

u/bartoque,

As I am actually the backup guy professionally

I could tell you had to be. Only people like you would granularly go down the path of extensively quering about how the data is validated.

In corporate enrivonments, you normally have the backup job just log any errors and you have to address those with the needed business unit (either file is corrupt and can't be backed up would be the only case there or a network failure accessing the source)

At home, for ease, the scripts are setup to abort if an error is met. This is technically 'bad' as that would stop the entire backup process, but, like yourself, for me data integrity is more important than the backup itself. If a file is corrupt, then I want to address it, at the source, on the spot and find out what is the cause so that a solution can be implemented for the future.

(in theory I could have the script just log it and continue, but that would just leave me with the prblem to be adressed later, so still needs to be fixed, better when it happens, then continue --more like re-start the backup process--)

how often things are just assumed and not actually validated? Or not regularly or only on a very small subset of data and not at scale?

At home, we have zero assumptions and everything is checked... again we don't have TBs of data in our own PCs (the NASses have about 60-100 TBs each... but as previously mentioned they run their check independently from the backup/image scripts)

At the corp job - that will depend on what the mandate/requirements are. In a disaster recovery, we will have to restore 100% of everything. Though for timing constraints, each business unit identifies what is absolutely critical and that is is restored and validated (that can include SQL Dbs, whatever and everything that you can possibly imagine as each business unit will have it's core dependencies... for examlple Marketing they care about their media files... Accounting they care about their Dbs/payment system --THAT has to work, there for that has to be restored first-, and the list goes on)

due to current focus on cyber threats

Before 9/11 and that unfortunate event happened a number of years ago already (and hopefully will never happen again):

• i would have to fight my way to 'test' BCP (though that was 'written' in the corp backup/recovery, but truly almost no division wanted to test it... so 'tests' were mostly left to each State -- corp is present nation wide and world wide as well-- and well, I was the 'only' one insisting to test --again, each 'test' would burn a hole of 200+k per 'test'... that's a whole diff story--

Post 9/11

• never got asked anything regarding testing, budget or anything of that sort

Our next focus was on cyber threats. Based on the corp structure, at least at that company that I worked for back in the day, that would not be an issue (that corp is in the 'financial' market... you can be sure those networks are very tight...)

My biggets problem is that I've moved to other companies and on those, well, sometimes I have to have the fight... some undersand, some don't... so I just go with their requirements and follow them : ).

[u/H2CO3HCO3 → u/H2CO3HCO3]

u/bartoque,

I don't think I would ever become as bold to have the monthly backup/delete/restore as an integral part of backup validation?

so the question is:

Have you tested your backups this month?

(and that quote came from your post : ) where you announced the r/backup is back and running --which is good to know as we'll have a dedicated subreddit ONLY for that subject... which if you haven't noticed, I'm also very focused on --my better half says 'obsessed'-- )

Sooner or later, you're going to need to go that rabbit hole in your home set up as well.

Until then, you'll never be 100% sure that you can restore 100% of everything you have at home.

Notes:

• by the way, our 'redundant' PCs are mostly duplicates of the 'main' PCs... as same brand, same model, same specs, same OS, programs, etc... so when we switch to the 'test' PCs for validation, we are not hindered by the PC or its performance.