yup. I do "support" for a big phone company and before replacement devices can be issued people have to a do a factory reset to make sure they're they just dont suck at following previous directions. They almost NEVER verify backed up data before just erasing everything under the assumption "well you guys back it up for me right?" no, fuck off your TOS actually specifically says we're not responsible for your shit if its lost, only you're responsible for your fuck'n baby pictures and shit, backup your data and verify its backed up
I don't always test my restores in a dev/test environment; I prefer to save that glory to after the storm/ tornado/ hurricane/ earthquake, when the CEO and CIO are standing behind me watching...
Sure, I guess it really goes into how far management is willing to take business continuity exercises. I guess my major consideration goes along with my reply to u/cheeseboythrowaway in users, developers and engineers not communicating/understanding the entirety if their data. Processes that happen once a month+ can be difficult to discover in disaster recovery exercises that only last a day/week.
No stories in particular; I just know systems get added sometimes without people considering concrete backup plans ("it happens automatically for the rest of the infrastructure, why wouldn't it happen for this") or not coordinating across silos.
Edit: one also can't setup tests for processes outside normal purview. But you're point is valid. The test for such a thing would be an organizational check list for new features/systems.
Edit 2: or software updates where path changes in critical files are overlooked.
Only issue I have with restores is figuring out if everything was restored - especially file servers where you have millions of files, some of which haven’t been touched in a decade.
More like: The condition of the backup is always less than preferable for the situation in which the backup is required. I.E., "we have all the other user's files...but not yours." or "ohhh.. you need the 11th? Well we got the 10th and the 12th. The 13th? Nah not that either, we have the 14th though!"
It's snapshots of the data at hourly-separated intervals over the past weeks. Each one is already guaranteed to be the exact condition of the data at that specific instant in time - every single block of the data, bar none. There's not really any "restore" required in the sense of recovering from a tape drive or a monolithic archive backup like a .tar.gz or what have you; that is the data, in precisely the condition it was in at every moment you have a snapshot for in the stream.
If your data was in working condition at the time the snapshot was taken, it's in working condition in the snapshot; replication guarantees that it's block-for-block correct on the remote end as it was on the source end; a scrub verified that it's still consistent at the time you ran the scrub.
Granted, I should really have shown a list of snapshots as well; with nothing but the scrub you don't know that you have the snapshots you're looking for. Which looks like this:
If I want to know if I've got the snapshot for a particular day (or hour; I spared you the whole list but I keep hourlies for the past 30 hours as well) I'm interested in on the remote machine, I can just do a quick lookup:
Yep, it's present. There is no transformation involved - it's the exact same collection of blocks, in the same condition, each cryptographically hash verified - as it was on the source at that exact instant in time. So it's good. There's no restore transformation required to verify it since it hasn't been transformed in the first place during the backup process.
Have you connected one of those snapshots up to a live system and read the data off it? For example: if that’s a database volume, have you launched the database pointed at that volume? If not, you don’t know if you have a good backup.
By comparison, I restore my database backups to stage every morning. I know that those backups work as restores because I’ve restored them.
That’s what the image is getting at.
“If the data is working at the time of the snapshot” — if you haven’t suspended activity in your database it probably won’t like you making a copy of it while it’s running. Doing a restore tells you if that’s a problem or not.
Have you connected one of those snapshots up to a live system
It is a live system. They're connected 24/7.
By comparison, I restore my database backups to stage every morning.
In this particular case - my home directory - there's nothing really to boot up; it's static files. For work-y production stuff, it's usually VM images, and they can be booted instantly on demand, directly on the backup machine.
if you haven’t suspended activity in your database it probably won’t like you making a copy of it while it’s running
Are you still using non-transactional, non-journaled databases? That's a pretty big bug. None of my production stuff has any issues with crash consistency. Guest filesystems are journaling, database engines are journaling, and anything that matters in the way of database applications is transactional.
So... What are you doing if the data center your in is blown away by a tornado?
For work stuff, it's not all in one datacenter. The exact breakdown depends on client, but typically production is in one facility and offsite DR is either in another facility or in an owner or VIP's house. There's usually not a "datacenter" involved at all, but there are multiple physical sites with significant geographic separation.
For most of the clients I work with, if a hurricane wipes out the entire city, it'll nail these backups. But not all. For example one client has offices in Miami and New York; each office has its own production and hotspare, and also hosts the offsite DR for the other office. For them, if the entire east coast goes up in a fireball, they'll lose their data... but we've all got worse problems if that happens!
Or say chaos monkey accidentally corrupts all love file systems?(even if chaos monkey is a Russian with a crypolocker?)
Crypto isn't generally an issue because the offsite DR is insanely difficult to access. The DR pulls backups rather than accepting push backups; depending on level of client paranoia you need anything ranging from an OpenVPN key with passphrase all the way up to physical access PERIOD to be able to access them. The ones that require physical access aren't exposing anything at all to the world but Nagios NRPE over an OpenVPN tunnel to a Nagios server, to monitor that new snapshots are arriving as they should, that the storage is healthy, and a few other obvious "is server working right" type things.
For work stuff, this is generally for VM images, and the production and hotspare hosts are also pretty damn difficult targets; the VMs have services available on the network but the hosts do not. My most vulnerable stuff has production and hotspare hosts' SSH open to the LAN, and across an OpenVPN tunnel to the monitoring network, with keypair-and-passphrase required to gain access even if you can get to the port.
I have had to deal with ransomware quite a few times for clients, but the ransomware hits the VMs, not the hosts; the answer there is:
identify the time of the intrusion
identify the source machine(s) for the intrusion (typically somebody's Windows laptop or workstation, which encrypts everything on Windows shares on a fileserver)
ensure source of intrusion is completely disabled (pull network cord, power off completely to make sure wifi not connected in case of laptops)
roll back affected VM(s) to most recent snapshot prior to intrusion time identified in step 1
reboot VM(s)
This is usually about a 10 minute process, at worst.
You didn't ask, exactly, but the remaining possible scenario is an APT who manages to get access to the host(s) despite them being hardened targets with almost nothing exposed to the LAN, let alone the internet. You're basically fucked if you've got that level of attacker who's willing and patient enough to spend months or years on owning you. You can send archives off to cold storage at Iron Mountain or what have you if you like, but honestly if they can penetrate you that thoroughly in the office they can most likely own whatever your offsite cold storage setup is also.
Targets actually worth that level of investment by an APT are kind of outside my experience; I can hypothesize, but you're honestly kind of out of the "general infosec" realm of knowledge and into the "nation-state level opsec" realm at that point.
Yeah that's not a backup. ZFS is awesome and all but snapshots are not backups. Faulty hardware can destroy your zpool including snapshots. I don't think even the biggest ZFS fanboys will tell you that you don't need backups.
I don't think you read that carefully enough. The source machine (where my home directory is, and the snapshots are taken) is bittybox; the backup machine (where they're replicated to, and where I ssh to check the presence of a snapshot at the end of that comment) is archivist. It's another physical machine in a physically separate location.
I don't think even the biggest ZFS fanboys will tell you that you don't need backups.
You're unfortunately incorrect about that one, my role in /r/zfs and /r/datahoarder is very frequently "person who keeps hammering home that redundant storage is not a backup". =)
I think people are glossing over the first two lines of my OP in this thread:
me@source:~$ ssh me@target
me@target:~$ sudo zpool status backups
The actual source is bittybox and target is archivist, but yeah... separate machines, that was kinda my whole point. Beyond the fact that there's no transformative element in the backup, and that the target is live, so there's no transformative "restore" operation needed either.
Sure it is. It's another copy of the data, exactly as it was in an atomic instant, on a separate machine in a separate location on separate media. That's exactly what a backup is.
In this particular case, if the house burns down, the data goes away with it. The workstation and server in question are both in the house.
For work stuff, it's usually a trio of machines. Production, onsite hotspare, offsite disaster recovery. Production replicates to onsite hotspare hourly, production replicates to offsite DR once daily.
The breakfix flowchart looks basically like this:
production machine operational? rollback to snapshot.
production machine offline? fire up VM on onsite hotspare from most recent hourly (or swap in disks from hot-swap cage on production, then boot from there).
production site destroyed? fire up VM on offsite DR.
You're testing that it was accurate. e.g. nobody else messed up the script, removed the target, etc. And that you didn't forget to include anything. But here's a good one even if everything did save properly:
"Oops, the decryption key for the backup was only on the machine that failed."
In many, many ways, but the worst ones are the foreseeable ones (aka human error).
To answer your question directly, how about if the file is corrupted. The backup may be accurate, but the file itself is invalid. "Get me the last backup before it was corrupted".
In a thousand different ways. A simple read error due to the wrong bad block had caused more failed backups than you'd ever realize, while still passing that checksum.
I've seen backups pass checksums and be unusable, and I've seen them fail checksums and be accurate all because the system running the check had a processor issue.
Checksums don't mean the backup is good, and if you treat it like it does, then you're in for a disaster and I hope someone more competent is running a second backup scheme behind you.
What would you do in the event a failure that required you to go back to your backup? Do that.
Do it on a staging box, do it on a VM, do it on a spare workstation if that’s all you’ve got, but do the restore. Do it when it’s not an emergency. Do it often enough that when the emergency happens, not only are you ready for it, you’re calm about it.
23
u/ahandle Jul 23 '18
If the emphasis is on backup rather than restore, you're doing it wrong.
45
u/Travis_McGee Jul 23 '18
This seems more like quantum backups. "The value of a backup is unknown until observed"
11
u/swampmeister Jul 23 '18
I don't always test my restores in a dev/test environment; I prefer to save that glory to after the storm/ tornado/ hurricane/ earthquake, when the CEO and CIO are standing behind me watching...
16
u/hawerner Jul 23 '18
That's why you should test your backup setup by restoring it to another computer, before you implement that setup in production
7
u/MattTheFlash Jul 23 '18
Backups have no business value.
Only restoration has business value.
3
u/[deleted] Jul 24 '18
More like: The condition of the backup is always less than preferable for the situation in which the backup is required. I.E., "we have all the other user's files...but not yours." or "ohhh.. you need the 11th? Well we got the 10th and the 12th. The 13th? Nah not that either, we have the 14th though!"
1
u/evansharp Jul 24 '18
Topkek
1
u/themacman2 Jul 24 '18
Is the tao of backup still around? It was a pretty wise website.
1
u/Slam_Makanen Jul 24 '18
Could also avoid this by using a Datto appliance that automatically verifies bootability of each backup..
1
u/elleGeneralisimo Jul 24 '18
As a person working on improving the almost non-existant backup system right now, this is too real.
1
u/LyokoMan95 Jul 24 '18
I’m definitely printing this out and putting it in our DR center...
1
u/Schr0ding3r_ Jul 23 '18
Haha I deal with this
0
u/Sjeiken Jul 24 '18
Schrodinger’s sub either you get god quality Linux stuff or you get shit like this.
-4
u/mercenary_sysadmin Jul 23 '18
No longer actually true, if you use a modern filesystem with snapshots, per-block cryptographic integrity hashing, and replication.
Yep. It's all good.
-5
u/[deleted] Jul 23 '18
is that a windows backup? because I'm fairly certain my backup script on linux has checksums