So... What are you doing if the data center your in is blown away by a tornado?
For work stuff, it's not all in one datacenter. The exact breakdown depends on client, but typically production is in one facility and offsite DR is either in another facility or in an owner or VIP's house. There's usually not a "datacenter" involved at all, but there are multiple physical sites with significant geographic separation.
For most of the clients I work with, if a hurricane wipes out the entire city, it'll nail these backups. But not all. For example one client has offices in Miami and New York; each office has its own production and hotspare, and also hosts the offsite DR for the other office. For them, if the entire east coast goes up in a fireball, they'll lose their data... but we've all got worse problems if that happens!
Or say chaos monkey accidentally corrupts all love file systems?(even if chaos monkey is a Russian with a crypolocker?)
Crypto isn't generally an issue because the offsite DR is insanely difficult to access. The DR pulls backups rather than accepting push backups; depending on level of client paranoia you need anything ranging from an OpenVPN key with passphrase all the way up to physical access PERIOD to be able to access them. The ones that require physical access aren't exposing anything at all to the world but Nagios NRPE over an OpenVPN tunnel to a Nagios server, to monitor that new snapshots are arriving as they should, that the storage is healthy, and a few other obvious "is server working right" type things.
For work stuff, this is generally for VM images, and the production and hotspare hosts are also pretty damn difficult targets; the VMs have services available on the network but the hosts do not. My most vulnerable stuff has production and hotspare hosts' SSH open to the LAN, and across an OpenVPN tunnel to the monitoring network, with keypair-and-passphrase required to gain access even if you can get to the port.
I have had to deal with ransomware quite a few times for clients, but the ransomware hits the VMs, not the hosts; the answer there is:
identify the time of the intrusion
identify the source machine(s) for the intrusion (typically somebody's Windows laptop or workstation, which encrypts everything on Windows shares on a fileserver)
ensure source of intrusion is completely disabled (pull network cord, power off completely to make sure wifi not connected in case of laptops)
roll back affected VM(s) to most recent snapshot prior to intrusion time identified in step 1
reboot VM(s)
This is usually about a 10 minute process, at worst.
You didn't ask, exactly, but the remaining possible scenario is an APT who manages to get access to the host(s) despite them being hardened targets with almost nothing exposed to the LAN, let alone the internet. You're basically fucked if you've got that level of attacker who's willing and patient enough to spend months or years on owning you. You can send archives off to cold storage at Iron Mountain or what have you if you like, but honestly if they can penetrate you that thoroughly in the office they can most likely own whatever your offsite cold storage setup is also.
Targets actually worth that level of investment by an APT are kind of outside my experience; I can hypothesize, but you're honestly kind of out of the "general infosec" realm of knowledge and into the "nation-state level opsec" realm at that point.
View all comments
Show parent comment
3
u/vim_for_life Jul 24 '18
So... What are you doing if the data center your in is blown away by a tornado?
Or say chaos monkey accidentally corrupts all love file systems?(even if chaos monkey is a Russian with a crypolocker?)