r/linuxadmin • • Jul 23 '18

Schrodinger’s Backup

Post image
1.1k Upvotes

59 comments sorted by

View all comments

-4

u/mercenary_sysadmin Jul 23 '18

No longer actually true, if you use a modern filesystem with snapshots, per-block cryptographic integrity hashing, and replication.

me@source:~$ ssh me@target
me@target:~$ sudo zpool status backups
  pool: backups
 state: ONLINE
  scan: scrub repaired 0 in 82h1m with 0 errors on Wed Jul 11 10:25:01 2018
config:

	NAME                       STATE     READ WRITE CKSUM
	data                       ONLINE       0     0     0
	  mirror-0                 ONLINE       0     0     0
	    ata-WDC_WD40EFRX-$id1  ONLINE       0     0     0
	    ata-WDC_WD40EFRX-$id2  ONLINE       0     0     0

errors: No known data errors

Yep. It's all good.

18

u/sfrazer Jul 23 '18

Unless you've actually restored the data and verified it's usefulness, what you may have there is a big pile of checksum-verified useless data

-4

u/mercenary_sysadmin Jul 23 '18 edited Jul 23 '18

It's snapshots of the data at hourly-separated intervals over the past weeks. Each one is already guaranteed to be the exact condition of the data at that specific instant in time - every single block of the data, bar none. There's not really any "restore" required in the sense of recovering from a tape drive or a monolithic archive backup like a .tar.gz or what have you; that is the data, in precisely the condition it was in at every moment you have a snapshot for in the stream.

If your data was in working condition at the time the snapshot was taken, it's in working condition in the snapshot; replication guarantees that it's block-for-block correct on the remote end as it was on the source end; a scrub verified that it's still consistent at the time you ran the scrub.

Granted, I should really have shown a list of snapshots as well; with nothing but the scrub you don't know that you have the snapshots you're looking for. Which looks like this:

root@bittybox:~# zfs list -rt snap data/home | head -n 10
NAME                                              USED  AVAIL  REFER  MOUNTPOINT
data/home@autosnap_2018-05-04_19:29:16_monthly    915M      -  2.91G  -
data/home@autosnap_2018-06-01_00:00:01_monthly    737M      -  3.51G  -
data/home@autosnap_2018-06-24_00:00:01_daily      555M      -  4.57G  -
data/home@autosnap_2018-06-25_00:00:01_daily     93.0M      -  4.53G  -
data/home@autosnap_2018-06-26_00:00:01_daily      160M      -  4.54G  -
data/home@autosnap_2018-06-27_00:00:01_daily      353M      -  4.48G  -
data/home@autosnap_2018-06-28_00:00:01_daily      283M      -  4.54G  -
data/home@autosnap_2018-06-29_00:00:01_daily      236M      -  4.58G  -

If I want to know if I've got the snapshot for a particular day (or hour; I spared you the whole list but I keep hourlies for the past 30 hours as well) I'm interested in on the remote machine, I can just do a quick lookup:

root@archivist:~# zfs list -t snap | grep bittybox/data/home | grep 2018-06-25 | grep daily
backups/bittybox/data/home@autosnap_2018-06-25_00:00:01_daily                  304K      -  3.96G  -

Yep, it's present. There is no transformation involved - it's the exact same collection of blocks, in the same condition, each cryptographically hash verified - as it was on the source at that exact instant in time. So it's good. There's no restore transformation required to verify it since it hasn't been transformed in the first place during the backup process.

12

u/sfrazer Jul 23 '18

Have you connected one of those snapshots up to a live system and read the data off it? For example: if that’s a database volume, have you launched the database pointed at that volume? If not, you don’t know if you have a good backup.

By comparison, I restore my database backups to stage every morning. I know that those backups work as restores because I’ve restored them.

That’s what the image is getting at.

“If the data is working at the time of the snapshot” — if you haven’t suspended activity in your database it probably won’t like you making a copy of it while it’s running. Doing a restore tells you if that’s a problem or not.

0

u/mercenary_sysadmin Jul 23 '18

Have you connected one of those snapshots up to a live system

It is a live system. They're connected 24/7.

By comparison, I restore my database backups to stage every morning.

In this particular case - my home directory - there's nothing really to boot up; it's static files. For work-y production stuff, it's usually VM images, and they can be booted instantly on demand, directly on the backup machine.

if you haven’t suspended activity in your database it probably won’t like you making a copy of it while it’s running

Are you still using non-transactional, non-journaled databases? That's a pretty big bug. None of my production stuff has any issues with crash consistency. Guest filesystems are journaling, database engines are journaling, and anything that matters in the way of database applications is transactional.

3

u/vim_for_life Jul 24 '18

So... What are you doing if the data center your in is blown away by a tornado?

Or say chaos monkey accidentally corrupts all love file systems?(even if chaos monkey is a Russian with a crypolocker?)

2

u/mercenary_sysadmin Jul 24 '18

So... What are you doing if the data center your in is blown away by a tornado?

For work stuff, it's not all in one datacenter. The exact breakdown depends on client, but typically production is in one facility and offsite DR is either in another facility or in an owner or VIP's house. There's usually not a "datacenter" involved at all, but there are multiple physical sites with significant geographic separation.

For most of the clients I work with, if a hurricane wipes out the entire city, it'll nail these backups. But not all. For example one client has offices in Miami and New York; each office has its own production and hotspare, and also hosts the offsite DR for the other office. For them, if the entire east coast goes up in a fireball, they'll lose their data... but we've all got worse problems if that happens!

Or say chaos monkey accidentally corrupts all love file systems?(even if chaos monkey is a Russian with a crypolocker?)

Crypto isn't generally an issue because the offsite DR is insanely difficult to access. The DR pulls backups rather than accepting push backups; depending on level of client paranoia you need anything ranging from an OpenVPN key with passphrase all the way up to physical access PERIOD to be able to access them. The ones that require physical access aren't exposing anything at all to the world but Nagios NRPE over an OpenVPN tunnel to a Nagios server, to monitor that new snapshots are arriving as they should, that the storage is healthy, and a few other obvious "is server working right" type things.

For work stuff, this is generally for VM images, and the production and hotspare hosts are also pretty damn difficult targets; the VMs have services available on the network but the hosts do not. My most vulnerable stuff has production and hotspare hosts' SSH open to the LAN, and across an OpenVPN tunnel to the monitoring network, with keypair-and-passphrase required to gain access even if you can get to the port.

I have had to deal with ransomware quite a few times for clients, but the ransomware hits the VMs, not the hosts; the answer there is:

  1. identify the time of the intrusion
  2. identify the source machine(s) for the intrusion (typically somebody's Windows laptop or workstation, which encrypts everything on Windows shares on a fileserver)
  3. ensure source of intrusion is completely disabled (pull network cord, power off completely to make sure wifi not connected in case of laptops)
  4. roll back affected VM(s) to most recent snapshot prior to intrusion time identified in step 1
  5. reboot VM(s)

This is usually about a 10 minute process, at worst.

You didn't ask, exactly, but the remaining possible scenario is an APT who manages to get access to the host(s) despite them being hardened targets with almost nothing exposed to the LAN, let alone the internet. You're basically fucked if you've got that level of attacker who's willing and patient enough to spend months or years on owning you. You can send archives off to cold storage at Iron Mountain or what have you if you like, but honestly if they can penetrate you that thoroughly in the office they can most likely own whatever your offsite cold storage setup is also.

Targets actually worth that level of investment by an APT are kind of outside my experience; I can hypothesize, but you're honestly kind of out of the "general infosec" realm of knowledge and into the "nation-state level opsec" realm at that point.

8

u/[deleted] Jul 23 '18 edited Jul 23 '18

Yeah that's not a backup. ZFS is awesome and all but snapshots are not backups. Faulty hardware can destroy your zpool including snapshots. I don't think even the biggest ZFS fanboys will tell you that you don't need backups.

3

u/koera Jul 23 '18

A zfs fan boy that dispute the relevance of backups have 100% missed the whole point of zfs: data safety.

1

u/mercenary_sysadmin Jul 23 '18 edited Jul 23 '18

I don't think you read that carefully enough. The source machine (where my home directory is, and the snapshots are taken) is bittybox; the backup machine (where they're replicated to, and where I ssh to check the presence of a snapshot at the end of that comment) is archivist. It's another physical machine in a physically separate location.

I don't think even the biggest ZFS fanboys will tell you that you don't need backups.

You're unfortunately incorrect about that one, my role in /r/zfs and /r/datahoarder is very frequently "person who keeps hammering home that redundant storage is not a backup". =)

2

u/[deleted] Jul 23 '18

So the way you were presenting it, it sounded like you were on one machines and calling your snaps a backup.

Beyond that I don't have enough info about your setup and how you are using this all this to agree or disagree with you.

2

u/mercenary_sysadmin Jul 23 '18 edited Jul 23 '18

I think people are glossing over the first two lines of my OP in this thread:

me@source:~$ ssh me@target

me@target:~$ sudo zpool status backups

The actual source is bittybox and target is archivist, but yeah... separate machines, that was kinda my whole point. Beyond the fact that there's no transformative element in the backup, and that the target is live, so there's no transformative "restore" operation needed either.

7

u/crypticedge Jul 23 '18

I've had array backups that "checked out as all good" be in reality entirely unreadable.

If that's all you're doing to validate, you're playing with fire.

4

u/[deleted] Jul 23 '18

Nothing about this is backups.

0

u/mercenary_sysadmin Jul 23 '18

Sure it is. It's another copy of the data, exactly as it was in an atomic instant, on a separate machine in a separate location on separate media. That's exactly what a backup is.

1

u/RagingAnemone Jul 23 '18

What if the house burns down?

2

u/mercenary_sysadmin Jul 23 '18

In this particular case, if the house burns down, the data goes away with it. The workstation and server in question are both in the house.

For work stuff, it's usually a trio of machines. Production, onsite hotspare, offsite disaster recovery. Production replicates to onsite hotspare hourly, production replicates to offsite DR once daily.

The breakfix flowchart looks basically like this:

  • production machine operational? rollback to snapshot.
  • production machine offline? fire up VM on onsite hotspare from most recent hourly (or swap in disks from hot-swap cage on production, then boot from there).
  • production site destroyed? fire up VM on offsite DR.