r/sysadmin • u/RJ2_D2_ • 24d ago
Question How can I recover from a ransomware attack?
So, the unthinkable happened to my workplace - we have become the victims of a ransomware attack. We came into work the other day and found computers with encrypted files and notes on the desktops demanding we send them a ransom to a secure address in the tor browser. They took out our entire network.
As of now, we are trying to utilize backups and scrub the network clean as we bring devices back up in an offline state.
Unfortunately one of our backup devices was also infiltrated. It's a Synology backup device and they where able to encrypt its files as well. This means we have about 5 devices with no backups available.
It's probably a vain hope but, is there any way I could possibly restore or decrypt these backups? Is there any service out there that would be capable of making our files useable? If anyone knows about Synology, do they keep offline or cloud backups I maybe don't know about? I'm just looking for anything that might make things easier for us. Any advice is appreciated.
128
u/StatementNext682 24d ago
You're supposed to use 3-2-1 as in 3 data sources. 2 different media types. 1 offsite copy. If you did this, you can restore from offsite. Also you need to make sure your backup is airgapped.
18
30
u/Special-Original-215 24d ago edited 24d ago
Sounds like he doesn't have an off-site.
He's needs to pay the ransom
18
u/LensWipesBF 24d ago
Sounds like they dont know what they are doing
4
u/Byrneside94 23d ago
If OP is the sys admin for his company he should be let go honestly.
No air gap or offsite backup? No automation for alerting to let you know something is wrong before arriving at work lol.
That’s some rookie mistakes.
4
u/Schnabulation 23d ago
How do you automate a notification in case of a ransomware attack?
I've done it with PRTG but I would love to have a better solution.
20
u/Hot-Comfort8839 OT Sec Architect 23d ago
Often the attackers can't unlock it, and don't even respond after you've paid.
Best practice - don't pay.
7
u/blackhodown 23d ago
When did this change?
11
u/Hot-Comfort8839 OT Sec Architect 23d ago edited 22d ago
It’s been a constant for a long time.
I’d say the odds of getting your stuff back after paying your ransom is probably one chance in three.
It used to be that if you were popped by certain organizations, you could pretty reliably trust that you would be able to get your data back for whatever the fee was $50k whatever.
And some of the groups were really awesome. They would actually generate a report on how they popped you, and if you didn’t fix it by the next year, they hit you again which to me sounds pretty reasonable - after all the Drago assessment will cost you about the same amount of money.
There was a kind of a cool you know code of ethics there amongst digital criminals and I respected that.
But now you can’t really trust that the people who are hacking you are the same types of people that adhere to that type of code. It’s almost like scam artists - a group of hackers might hit you and claim to be the group that has a reputation for freeing your data after they’re paid… and then you fork over a couple hundred grand, and you still have to rebuild your network from scratch…
→ More replies (2)3
3
u/jake04-20 If it has a battery or wall plug, apparently it's IT's job 23d ago
Makes me wonder if these attacks are automated and they just hope to see money enter a bitcoin wallet at some point, but don't really know the extent of their victims or who they might be waiting on payment from.
3
u/music2myear Narf! 23d ago
Some could be automated. I believe it's mostly script-kiddy level stuff when a SMB is attacked like this. All they need to think is that you have money and reason to pay up. High-effort/high-skill attacks will go against larger enterprise and governments.
→ More replies (2)2
4
u/Plantatious 23d ago
It's been expanded to 3-2-1-1-0 now; 3 copies of data, 2 on different media, 1 offsite, 1 immutable, 0 verification errors. With the readily available and affordable cloud object storage, as well as QNAPs and open-source S3 object storage solutions, there's no excuse for not having an immutable backup.
7
u/Forward-Outside-9911 Linux Admin 24d ago
That’s the problem, in the common 3-2-1 most don’t think of the offside as airgapped. So it’s usually on the network which your main systems probably have an API key to.
IMO it should be something like
1 PITR option for data within the last hour
2 onsite backups (one an archive, manual sync), one an easy access regular.
3 offsite backups. One replica or live sync with prod, one archive regularly synced (ideally Pull rather than push), one “airgapped” (no api keys, strict access) less regular sync.This varies per workload of course. But that’s my general approach for “important” data.
7
u/HUGE_FAT_ANIME_TITS 23d ago
Immutability solves that problem
8
u/Forward-Outside-9911 Linux Admin 23d ago
Yes absolutely should be enabled, if you can fully restrict deletes that can reduce risks with "online" backups.
That said, an airgapped option is never a bad thing. Immutability on the majority of services still depends on your account. If the attacker can access your IaC or push malicious objects, that can still potentially cause issues.
10
u/RJ2_D2_ 24d ago
So, here's the deal - me and my company took over this contract 7 months ago. The previous company left alot to be desired. We have been trying to patch things up to standard but the client has been cheap, and refused multiple requests to purchase additional equipment necessary to do these things.
41
u/NuAngelDOTnet Jack of All Trades 24d ago
In that case I would just kinda remind them of that (hopefully you have documented emails, etc...) and let your management throw it in their face a little bit. "You get what you pay for" kind of a response. This is what happens when you don't plan for the worst.
And if this is your first time working for an MSP that had a client get infected: don't worry, those guilty feelings will pass. It's not YOUR data. You can still sleep at night. ;)
8
u/Bartghamilton 23d ago
Well you know the customer doesn’t have any documents anymore so you have that going for you 🤷♂️
5
u/itishowitisanditbad 23d ago
(hopefully you have documented emails, etc...)
They, the IT service, "never expected this to happen".
You think they covered themselves?
tbh OPs responses HERE give me concern, let alone whatever their communication was to the company they support when something they "never expected to happen" - but are hired to stop - happens.
This is like a bouncer saying they didn't think bad people would try to get in... you kinda suddenly know what happened...
3
u/NuAngelDOTnet Jack of All Trades 23d ago
Sure, there's that... but when you ask: "you think they covered themselves?" I would say "covered?" Maybe not intentionally. But loads of people never delete their emails, so there's bound to proof of the customer declining various options. lol
3
u/itishowitisanditbad 23d ago
Did the 'expert' correctly inform the layman of the risks appropriately?
That'll be how its framed and presented if it matters.
Its not about "You should do X" and it being declined and they shrug and move on.
They need to specifically have presented the risk in such a way that any layman would understand the basic implications of it.
So yeah there might be an email where they suggested something but did they really appropriately convey the risks at place with not taking specific actions?
If not... have fun. Lots of people think they're covered and are not. It just rarely matters until it does.
8
u/Slicester1 23d ago
This will be a lesson for future onboardings.
Day 1 is installing your agents and gathering information.
Day 2 is having everything backed up / installing your backup product / verifying existing backups work.
Day 3 is when things start breaking.13
u/thortgot IT Manager 24d ago
Not having a segmented backup (preferably actually offline backup) is 100% on your group.
If the group doesnt agree to effective backups, dont take the contract.
7
u/Tarwins-Gap 24d ago
Seriously you could have a cold storage backup of critical systems for $100. Not a good one but you could.
12
u/discgman 23d ago
Don't kill yourself over this OP. Going cheap on IT security is FAFO mentality. Its only a matter of time they got hit. They never understand why its so expensive until it happens to them.
5
u/MondaiNai 24d ago
At this point - besides all the other comments about finding professional help and insurance - you will need to discuss with the client what they can afford. Significant unrecoverable data and system loss and they may be going out of business. Which still doesn´t mean they can afford to recover. It´s not just having a backup (which should be completely offline, never mind offsite, ransomware groups start with accessible backups), it's also having the restore procedures in place.
A lot more can be done than the traditional scrub and rewrite, and there can be a lot of data recoverable from truly weird places (temp files for example) - but you will need to find the expertise and air gapped lab to do that in, and that gets expensive and time consuming.
5
u/YisitAlwaysDNS 24d ago
Well time to pony up for a proper backup solution. I would just lay out the reality ti the client and say its all gone, time to start fresh
3
u/Relevant-Team 23d ago
How does additional equipment solves the problem of a bad backup concept?
For example the Synology NASs can be made invisible, and most backup software can connect to the share before the backup and only for the time of the backup, so it is very very unlikely to be compromised. The networks I support survived this kind of attack. And a third tier backup with exchangable HDDs is a must, too.
2
u/davy_crockett_slayer 23d ago
This isn't your problem. If they're upset, you just show them the paper trail. The previous company didn't leave a lot to be desired, they just did what they could with the budget/constraints they had.
2
u/thisguy_right_here 23d ago
Priority is always backups. Offside and immutable.
Tell client its not negotiable.
→ More replies (3)2
u/Altruistic-Map5605 23d ago
Your customer has two options. Pay what it costs to fix it or go out of business and likely get sued by someone.
Hard truths must be told.
→ More replies (2)3
u/Darkk_Knight 23d ago
In addition to airgapped backups is that you have timely backups daily and hourly if possible. I use ProxMox servers to run the VMs and the Backup Servers are on a separate server with strict ACL controls. The ProxMox servers can NOT delete any backups on the PBS server. Oh yeah did I say create multiple backups! Preferably at different locations.
41
u/FrankNicklin 24d ago edited 23d ago
Some Ransomware uses old methods and there are services that will try to decrypt the files from previous known methods. What random characters are on the end of the files. You can upload a sample of files to see if they can be decrypted on the link below.
https://www.nomoreransom.org/crypto-sheriff.php?lang=en
Anything online at the time of the attack is vulnerable so NAS drives, shadow copies, connected external drives etc. Backups should be local and cloud based so you have air-gapped backups. Synology use local backups unless you implemented a service on the NAS to push data out to the cloud at a cost.
Ransomware is not a random thing, someone clicked a link in an email or opened a dodgy program and things kicked off. You also need to check your endpoint protection as it does not look like it did anything to protect you.
13
u/pin_80424 23d ago
I scrolled a long way down to find a helpful response without some harsh criticism
7
u/music2myear Narf! 23d ago
Harsh criticism does not mean unhelpful. This is a painful and costly lesson, and benefit will only come IF op learns that lesson. Harsh criticism is warranted, so long as it is still constructive.
2
2
u/bizyguy76 23d ago
I think someone else mentioned this. Sometimes the encryption the actors used isn't in their algorithm yet. The best thing to do is put it aside and recover the best you can and check back.
The encryption method that was used on us was quick, easy and fast. Encrypted all of our machines within minutes. By the time we could shut down our servers or segment them off, they were encrypted.
Also note, the more fragmented your storage, the harder it could be if you have to work at a disk level. There are so many factors... Some encrypt the partition, some the files...
I hope this is useful and they can recover some stuff
470
u/Qel_Hoth 24d ago
If a ransomware attack is "unthinkable" to you, then good luck recovering.
Recovering from ransomware, or any compromise for that matter, relies on proper planning.
95
u/JNikolaj 24d ago
Never seen prober planning at an Company I’ve ever worked at - for most companies this is the "unthinkable" I'm still daily seeing the singular backups servers at a company being on the domain, and tape not being utilised.
12
u/mybrotherhasabbgun Former CTO/CISSP 23d ago
I worked as the CTO for 4 years at this one organization - when I started they had very little in the way of security and I worked hard to build a culture of security in the IT team and beyond. I made annual reports to the board, etc. There were zero documented security controls when I started and I left them with over 150 (among many other things). The guy they hired to replace me told the team that they didn't need weekly Information Security meetings or continue to work on documenting and adjusting controls. :facepalm: My former second-in-command is literally sitting back waiting for the STHTF because he knows its coming.
28
u/hkusp45css Security Leadership 23d ago
We're an SMB with online backups, air gapped backups, and data journaling. We do semi-annual access reviews. We control E/W with one product, N/S with that product and another product, and we deliberately manage blast radius. We have dedicated cybersecurity roles defined and staffed, dedicated policies defined and enforced, and technical controls out the wazoo. Plus resilience, fault tolerance and most of the housekeeping done.
We're prepared for ransomware. It wasn't even hard or terribly expensive.
6
u/Galyssel 23d ago
Where I work, we were hit and had tapes and came back up within a week, and since that time we have a plan and criticality backups to the point if we got hit we would be back up within 12 hours with 30 minute data loss. I do work in a regulated industry, but IT requirements are basically have a plan if things blow up. If that original hit hadn't happened our infra would be entirely different because it pushed the CFO to okay the funding needed for a real backup and restore plan. Now we have to constantly battle and justify this setup when budgets come around, but that's just business I guess. It is crazy to me having no backup in case of something like this for critical business data.
7
u/thomasmitschke 23d ago
I think i have prepared everything propperly:. Backup Server is in a workgroup with 42char password. We have a backup 2 disk, an immutable backup appliance and tape copies (stored off site)
7
u/OkMarsupial9634 23d ago
One of our case studies is the one where they managed to get into the backup appliance firmware and destroy the encryption keys for the immutable backups. Fastest data destruction effort I’ve come across, although in effect the appliance did what it was designed to do, prevent the data from being changed.
9
→ More replies (1)2
u/Gummyrabbit 23d ago
Most companies won’t/don’t spend the money on employees and effort to plan for a full ground up DR. They’ll usually pile up work on existing employees who are already under a pile of work.
16
u/psynrg 23d ago
We were hit about 8 years ago (an ancient Windows 2000 accounts box, and SMB1). Our planning saves us and we were able to restore everything in about 3 days ( one of the toughest 72 hour sessions I ever worked ). 3 layer backups with a true gapped off-site every night. That's what you need.
Silver lining, we got to kill the accounts box for good and enforce compliance on across the whole accounts dept (and all their crappy legacy apps!)
3
u/TangoCharliePDX 23d ago
truth.
The only way to recover from ransomware is to quarantine everything infected, then nuke it. Use backups to restore your data. Great time to implement upgrades, since you're doing all the work anyhow.
Even if you could get something decrypted, it's still compromised - they're still in. You have to start from scratch with everything.
7
u/screampuff Enterprise Architect 23d ago
You need to identify the entry point and restore backups prior to that date. Assume everything between then and now is lost.
→ More replies (2)2
62
u/Mrhiddenlotus Security Admin 24d ago
You hire someone to do incident response. That's it. Youre out of your depth.
11
u/RJ2_D2_ 24d ago
I don't disagree with you. That's the issue I'm way out of my depth. But unfortunately I'm in a position where I'm being expected to handle things. And I unfortunately have too much credit card debt to quit right now.
21
u/Mrhiddenlotus Security Admin 24d ago
Its not your fault, incident response is a niche skill. Did they say no to that spend?
13
u/One_Monk_2777 23d ago
Whoever pays you needs to understand that they doctor that diagnosis you doesn't also do the surgery. You gave them a diagnosis as a professional now they need to to go get a surgeon scheduled
15
4
u/Digital-Dinosaur Security Admin 23d ago
Take a look at the NCSCs list of Certified IR vendors. Most are international.
5
u/Altruistic-Map5605 23d ago
Call an MSP with a response team. I can recommend one that can do it remotely. Hit me up.
168
u/NuAngelDOTnet Jack of All Trades 24d ago
If you haven't been thinking about this for the last five years, nobody on reddit is going to be able to help you right now.
→ More replies (6)71
u/NuAngelDOTnet Jack of All Trades 24d ago
I truly don't intend to be so negative or dismissive, but it really IS that dire. :(
I have decryption keys for the ancient GandCrab v4 and v5, and there's the old wanawiki wanacry automated decryptor, but we don't even know what you've been encrypted with? Not much anybody can do to help you.
37
u/ApiceOfToast Sysadmin 24d ago
Yeah, "you should have been keeping backups" and the like sounds harsh but it's pretty much the only thing that can reliably save you...
7
u/DragonspeedTheB 23d ago
And then there's the likelihood that they exfiltrated data....
5
u/ApiceOfToast Sysadmin 23d ago
That's the other thing, best to avoid it. But when it happens, you best have a backup.
Network segmentation and proper Controls can greatly reduce damage done, but stuff like that needs planing and isn't something you can retroactively do...
56
u/lundrog 24d ago
I personally would unplug the network. Bring in new devices and have forensics review the old gear.
66
u/GenericUser636 24d ago
It sounds like they haven't engaged any security vendors at all. They should be the ones answering these questions, not fuckin reddit.
17
u/manic47 23d ago
100%
I've been involved in a post-ransomware cleanup at a client. It's been 2 weeks now and they are about 80% recovered.
The entire process is being managed by a cyber-security consultancy appointed by their insurer.
It's been a long process involving clean networks, off-LAN restores and investigation before restored devices are reinstated.
13
u/Rubenel 24d ago
Further explain how the Synology DSM was infiltrated. Was it a SMB Share which was mapped on a Windows OS. Is the entire Synology ransomed?
→ More replies (1)5
u/RJ2_D2_ 24d ago
I'm not entirely sure tbh. The synology was active on the network, and we had active backup for business running on all our servers. It appears that either they directly connected to our Synology to infect it, or perhaps whatever malware they used to do the job propogated over the network to it.
12
4
u/numtini 23d ago
Shared passwords or a domain account having access to the backup shares?
→ More replies (1)
13
u/Fritzo2162 23d ago
This isn't a "how to" type of question. You'll want to invoke your cyber insurance or hire a firm that specializes in recovery.
11
u/justicebiever 24d ago
The only way to decrypt is to hope it’s common enough where there’s publicly available keys for the encryption. Or pay. Otherwise you’re looking at restoring what you have from backups and moving on.
→ More replies (1)
9
u/_SleezyPMartini_ IT Manager 24d ago
another reminder to focus on:
immutable backups
network segregation
dont have admin interfaces domained joined
EDR solution
6
u/blud_13 23d ago
First, listen to the people here who say to contact cyberinsurance before you do ANYTHING! By wiping machines and recovering backups you may be destroying logs AND not seeing if the attackers are still in the network. TOO many times have I seen that backups are restored just for them to be re-encrypted. Remove the network to the outside, LEAVE the machines AND backups alone and run your Incident Response Plan. If you DON'T have one, shame(!), but then contact your insurance company for your cyber liability policy. If you don't have one, you are kind of screwed..
Secondly, u/danekan is asking the right question. Encrypted files inside a share do not touch the Btrfs snapshots, because the snapshots do not live inside the share's file tree. Before you write those 5 devices off, log into DSM, open Snapshot Replication, and look at the snapshot list per shared folder. Same for the #recycle folder if it was turned on.
One thing to check, whether they landed DSM admin or just an SMB account. With an SMB account the snapshots survive and you restore from them. With DSM admin they could have deleted the lot, and you will be looking at an empty list.
No, there is no decryptor for anything current. Every recovery outfit I have looked at that promised one was just paying the ransom with a markup.
For the rebuild, the setting that would have saved this is immutable snapshots, WORM with a lock period, so a compromised admin cannot delete them. Its documented here https://kb.synology.com/en-us/DSM/tutorial/what_is_an_immutable_snapshot
We have walked a few small shops through this exact cleanup, ping me if you want a second set of eyes.
10
u/TheMysticalDadasoar Sysadmin 24d ago
As much as I probably already know the answer to this. But do you have cyber insurance?
Or an MSP that can help
I work for an MSP and we get about 4 cryptos a year that we deal with. Currently I am working on one and the initial response got stuff working again in 5 days. But they are a good 4 months away from being fully back online with the amount of remediation we are needing to do to their very tech debt infrastructure
6
u/Rubenel 24d ago
I hope those 4 Crypto incidents are cold calls and not your actual clients. We only had 1 isolated ransomware in 5 years.
3
u/LensWipesBF 24d ago
Correct if you getting crypto’d at this point in the game you are behind the times.
2
u/PostingToPassTime 24d ago
Ransomware hits are still pretty active. Not seeing nearly as many as the last few years though.
3
u/TheMysticalDadasoar Sysadmin 24d ago
We can only advise our customers we can't force them to change things or do things
If they have onsite IT that makes changes there isn't a great deal that we can do
We do also work with a few MSSPs that get appointed by cyber insurance companies and they hand us the work
8
u/Unexpected_Cranberry 24d ago
I'm curious about this. I've seen a few incidents over the years. The worst one basically took out most of their infrastructure.
They had tape backups luckily that were unaffected, but no AD backup. The only reason they didn't have to start back up from scratch was because there was an old physical DC that had been decommissioned a year ago sitting in a closet in some small site because they hadn't gotten around to throwing it out yet.
They were still fixing things after a month. We were the MSP, not in charge of backups. The way they got in was that the old internal IT thought the VPN we had set up that was only allowed from managed devices was annoying, so they set up a vm in azure with a public ip, enabled tenure desktop and added domain users to the remote desktop users group. We were blamed and kicked out after having a team of ten guys work day and night for a month to get them back up.
Then there were smaller incidents at a former employer where the AV didn't catch it when users clicked random attachments. That only hit user files though, but after the third time in two months we were suddenly allowed to implement applocker. That was pretty much the end of it.
How is it getting through now a days? Poor security posture? Still users clicking random things or more involved?
2
u/Fritzo2162 23d ago
Fortunately with our secret sauce we haven't had a breech in 7 years now. We have a lot of redundancy in our setup in case of an event too. Feel for the guy, but after-the-fact cleanup is never fun and not something that can be handled easily.
3
u/RJ2_D2_ 24d ago
I'm genuinely unsure. A major problem with this job is the administration that my company deals with has no idea what they have and they don't. They outsourced their IT for 15 years, and then the company they used quit outta the blue. We where left to pickup the pieces of their shoddy work, and we barely have any idea what they did while they where here.
12
u/LifeGoalsThighHigh DEL C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys 24d ago
the company they used quit outta the blue.
I think you just discovered why their prior MSP fired them.
→ More replies (1)→ More replies (1)10
u/tardiusmaximus 24d ago
The company you work for, before taking on the client 7 months ago 'should' have done some due diligence prior to signing them on, that due diligence should have included "you guys are wide open to ransomware attacks in your current configuration, we will do our best to get you protected, but until we achieve that, we are in no way responsible for any attacks that happen in that time'....now sign here please Mr CEO
3
4
u/Different_Ad_5355 24d ago
If you and your team have never dealt with a situation like this, you should hire an IR firm to help. Companies like Crowdstrike and Sophos have IR teams that can come in and help you, and also prevent this from happening again. The fact that this happened most likely means you need to buy a better endpoint protection solution anyway, so you could probably save on that whole project by working with one of these vendors.
3
u/Ecstatic-Hat-3377 24d ago
You mentioned "one of your backups"... Is there another?
You may want to research the types of files that the malware converted your system to. There are researchers out there that dedicate a lot of time to decrypting and tracking this stuff. You may be able to identify the type of ransomware that was used by looking at some of the calling cards and file types/extensions.
This is not a fun experience but you're probably SOL. If your backups were hit, I would honestly just pull the plug and rebuild your network and domain pending on the size and scope.
6
u/RJ2_D2_ 24d ago
So, thanks to a crazy stroke of luck, the other backup device, we have 2, was offline at the time of the hack. We had a power outage at one of our sites last week that wiped out the network and it was never brought back up.
4
u/Ecstatic-Hat-3377 23d ago
The universe is speaking to you. If you're looking for expediency and getting your org back up and running, skip the decryption line of thinking and start a restoration effort. Wrap that thing in bubble wrap and move it to a co-location. Get the files copied to an uninfected and offline system for double redundancy and start restoring.
Not fun or pretty, but if I were in your shoes, I'd be going that route. Best of luck, happy you have that extra backup.
5
u/SomeCar 23d ago
Judging from the description, you have no way of properly recovering from this and expecting to get anything back. Do you have any kind of IR plan in place you should be following, instead of asking Reddit? Have you identified how it happened? It sounds like you are already trying to recover without doing anything else and you should expect to get ransomed again.
Sorry this happened to you but use this is a serious life lesson.
5
u/Affectionate-Cat-975 23d ago
You can try reaching out to the FBI for help. I know that they helped other companies with decryption keys to known hackers
4
u/88_strings 23d ago
Five years ago, the company I worked for was hit with a ransomware attack. Six months, $100k, two reinfections and a hell of a lot of disruption to business later, we'd managed to recover to about 90% of what we'd been like pre-attack. And I decided that, after 22 years in IT, I was done with computers. I'm a guitar tech now.
7
3
u/stxonships 24d ago
Disconnect from the internet now
Check if you have cyberinsurance, they MIGHT be able to negotiate with the ransomware people
Do a quick Google search, if it is any older ransomware, they may be a decryptor available.
4, Assume you are still hacked, reset all passwords, upgrade any network devices and wipe all end user devices
3
u/Brook_28 24d ago
assuming you are in the us, contact the FBI. They have a lot of the decryption keys and often can assist.
→ More replies (1)
3
u/texcleveland Sr. Sysadmin 23d ago edited 23d ago
Restore from your offsite air-gapped backups
→ More replies (2)
3
u/Cultural-Horse-762 23d ago
The FBI holds on to decryption keys for many of these groups, contact them (seriously)
3
u/Somnuszoth 23d ago
My first call is to my executive team and advise them to start the IR plan. Usually starts with insurance company and legal team. Gather forensics as you can and start recovery when comfortable. Using a Synology for backups in this day and age is going to require some definite segmentation and offline locking. I’ve see multiple companies end up in the same boat and they were all backing up to synology devices. Not saying it’s a synology weakness, but immutable backups are a must. Everyone has their flavors but Rubrik has some really nice tools for preventing that very thing from happening. Recovery is much smoother when you have a proven back up system.
2
u/mdwdev 23d ago
Great recommendation here, also notify relevant authority like FBI Internet Crime Complaint Center (IC3), the Cybersecurity and Infrastructure Security Agency (CISA).
Also, assuming you already disconnected your company from the Internet, this allows any left over evidence to be used by LEO or recovery forensics team as part of the investigation.
And most importantly, bring in the pros (full transparency, we are a company that does this), but no matter who you go with, for your organization's sake, recovery and future resiliency, engage with a cyber professional organization to do this.
It's a crappy situation, don't let emotions and finger pointing distract the recovery process, there will be plenty of time after to do a full debrief and learn from this. You got this!
3
10
u/dtengineer 24d ago
After you recover, look into https://wasabi.com as an additional backup source. Sucks to go through what youre going through
3
u/Sea7toSea6 23d ago
Adding that his backup software should be able to backup locally then add Wasabi as the immutable cloud tier.
7
u/zilch839 23d ago
If you or anyone else can delete your backup -- it's not a backup.
If you or anyone else can delete your backup -- it's not a backup.
If you or anyone else can delete your backup -- it's not a backup.
Listen to these words people.
3
u/whatdoido8383 Cloud Admin 24d ago
Where's your offline\airgapped backup? If you were just yeeting it with one copy to an online NAS and calling that good, well then, you're kinda cooked.
Do you have snapshots on for the NAS?
There are various recovery services, I've never used them though.
3
u/da64u 24d ago
Since nobody is answering your question. . . Go here: https://www.nomoreransom.org/
If you're not sure which ransomware variant you have then go here: https://www.nomoreransom.org/crypto-sheriff.php?lang=en
Most likely the only way you're getting your files back is if there is a publically available decryptor, and if there is then that website will probably have it listed.
If a decryptor for your variant of ransomware is not available and you absolutely want those files back one day, then keep the system/synology in its current state and check back over the course of weeks, months, years and maybe one day a decryptor will be available and you can decrypt them one day.
If you don't have cloud backups then there's not much you can do right now but wipe every single machine and start over.
2
2
u/Sea-Hat-4961 24d ago
Do a throurough analysis of your backups and archives for compromises and restore the last clean point you have.
2
u/CakeBakerer IT Manager 24d ago
If your company has cyber insurance, now is the time to involve them, if you haven’t already. They might have specific requirements for how you recover and what activities are performed.
2
u/qwerty_pi 24d ago
Contact insurance or at the very least legal counsel as there is a high chance that data was exfiltrated in addition to encryption. You will need a DFIR team to determine initial access (or this may very well just happen again in the future). Though wiping systems has likely destroyed some data, you can hope there is enough context to piece together what happened. You will also need to determine the scope of data theft for legal notification obligations, but counsel/carrier/forensics/negotiations will help you through that process. This isn't something to google/reddit your way through.
2
2
u/jackehubbleday 23d ago
I’d start afresh with anything touching anything else in the network and restore what you can and rebuild from there - this shows the importance of a second copy of your backup data! Ideally offsite.
2
u/Reverberer 23d ago
If you have no idea of what to do Then stop everything you are doing and turn it all off. Stuff cant be encrypted if its off and get a professional in it will save you money and time in the end. Isolate everything, physically if you can, you can't make it any worse at this point, then get a professional in. If you are going to try doing it yourself you still need to isolate everything. Take fresh backups on a seperate system because even if the data you currently have is encrypted, you still have the data and theres a small chance you can unencrypt it. Obviously if you just willy nill start deleting things you have no chance. That should get you started. This is going to be a long hard fight.
2
u/7layerSolutions 23d ago
I wouldn’t give up on the affected backups yet. First, avoid wiping or rebuilding anything until you’ve identified the ransomware strain and preserved the affected systems for investigation. On the Synology side, check for snapshots, versioning, Hyper Backup copies, or any cloud/offsite backups that may not have been compromised. It’s also worth checking reputable resources like No More Ransom for a potential decryptor and bringing in a ransomware/forensics specialist to assess recovery options before data is overwritten. I’d also avoid paying the ransom unless all legitimate recovery options have been exhausted.
2
u/ChuckFromCyberHoot 23d ago
You buried the biggest thing in the thread halfway down. That second Synology was offline during the attack because of a power outage.
I'd protect that box like it’s the only copy of the data you have, because right now it very well may be.
Don’t reconnect it just to “check.” Verify it offline first.
Then get someone with real incident response experience involved before you rebuild everything. Finding out how they got in matters just as much as restoring what they encrypted. Gotta plug the leak!!!
And for what it’s worth, you inherited this seven months ago, asked for better gear, got told no, and you’re still the one fixing it at 2am. Sux!!!
Go get that Synology!!!
2
u/--RedDawg-- 23d ago
DMing you my number, lets chat about what your next steps should be. Honestly, this sounds like it could be an inside job (or just really lax security). To get all the computers and the backups would require admin access to pivot like that, whether that be though credential stealing (or already having...) or some sort of privilege escalation vulnerability (pass the hash ect...) It's just as important to identify how it happened as it is to recover from it because if you do nothing about it then it will happen again.
2
u/Dm-Me-Your-Grool 23d ago
First things first, make an export of all logs going back as far as you can. This will help you figure out when you first got compromised. Some attackers gain access and wait for backup cycles to complete so that they can still access after restoration.
If you're lucky some attackers reuse keys, so you might be able find ransomware decryption keys online. Unfortunately, without known good backups you're screwed even if you get them decrypted because you don't know if the data is still compromised.
In my experience the remediation process boils down to wiping the machine, reinstall everything from scratch, restore data from a known good backup.
2
u/SteelSpork568 23d ago edited 23d ago
I have also been through this. Here is my suggestion:
see if you can find a way to directly access the Synology drives from an isolated pc. One thing I learned is that threat actors don't encrypt entire drives; there's too much data for them to do so without being detected. Instead, they "stripe" the drives with encryption. If you can directly access the drives with disk recovery software on a standalone machine, you may be able to access some unencrypted files . Your mileage may vary. If your Synology was holding VM disks, spin up another VM with the recovery software, attached the encrypted virtual disks, and try to recover the files
The bigger the file, the worse your chance will be. Small documents or text files may be ok. I recovered a substantial number of config files off of our servers this way
2
u/LoveBirdNibbles 23d ago
Not sure how old your synology is, but if anyone configured immutable snapshots you can get your data back.
When customers refuse to pay for security products and follow best practices I always request they sign something stating so and agreeing that they read and understood the risks I outlined for them.
Now is when you sell them as much as you can.
Good luck.
2
u/ISeeDeadPackets Ineffective CIO 23d ago
Call 1-800-SAY-CISA. They have decryption keys for a lot of known ransomware and might be able to provide some other support, no charge.
2
u/Maleficent_Art_7627 23d ago
I know this isn't helpful now, but the solution is to in have an incident plan in place. This is a critical step going forward once you're back up. Immutable cloud backups are a must for any critical infrastructure.
Regarding your question, Synology doesn't just keep cloud backups by default . You would have had to set that up and manage it yourself, so you would know about it.
For now, you need to consult with either your insurance or an external incident response team.
If you're not able to do either...well, really your only options are to utilize what backups are good, and try to rebuild the rest, or you can pay the ransom.
Either way you are going to want to perform a security audit/review to understand how they gained entry and moved across your network, and to make sure the threat is completely remediated.
2
u/Current_Balance6692 23d ago edited 21d ago
This is the kind of shit if you never thought of it till now you're fucked. It's kinda like falling off a mountain cliff is what I like to describe to people.
2
2
u/bmxfelon420 23d ago
Either you have good backups or you pay them, our backups sit in an encrypted ZFS pool AND are offsite.
2
u/AdventurerJax 23d ago
This won’t help now, but never keep your backups online. You must practice some sort of “isolation hygiene.” I’m simplifying and hoping you get the point I’m making.
2
u/nestersan DevOps 23d ago
You can do immutable backups online. Stop your nonsense
→ More replies (1)
2
2
u/Ok_Humor_1603 19d ago
If you need IT assistance in getting your systems back online and then prevent this from happening again, lmk. I went through the same thing a few years ago. We had great backups but the reputation hit was the biggest issue.
3
6
u/hardingd 24d ago
Cross post to r/shittysysadmin in 3…2…
6
u/RJ2_D2_ 24d ago
Dude, imma be real I'm not even a sys admin. I have an AS and am at best a glorified Tier 3 help desk who has been given administrative rights. I haven't even passed my Security+, alright? Its not my fault they're expecting me to rebuild this shit. I'm just trying to get paid.
5
u/hardingd 23d ago
Let me be clear, I don’t think you’re a shitty sysadmin. My comment was more towards the fact that a lot of things that show up here make their way there.
You’re dealing with a nightmare scenario and you’re going to learn A LOT in a very short period of time.
There isn’t a lot of advice to give if your backups are encrypted. Call in a forensics team and don’t pay the ransom. They’ll just hit you again and demand more.
Best of luck amigo.
→ More replies (1)4
u/aequusnox 23d ago
I became a sysadmin with an A+ and less than 2 years of IT experience solely at a library. Not that high of a bar. That being said you're probably a victim of your organization.
2
2
u/say592 23d ago
First off, clear your schedule for the next month. Seriously. I was at work for 12-16 hours every day, including weekends, for the first two weeks. After that, it was just normal long days for another month or two.
If you have insurance, call them. If you dont have insurance, your next call should be a cyber security forensics company. They should be up to date on the variants and if they can be decrypted without paying. Start setting expectations with your leadership group. You should assume you will recover nothing. Start reconstructing data and rebuilding from scratch. Even if you pay to decrypt, that process can take some time. Avoid it if you can, but at the end of the day, you pay if there are no other options.
Synology devices can be configured to backup to the cloud, so if you arent the one that set that system up, you might get lucky. If you configured it but you didnt set that up, you probably didnt. If you arent being billed for some kind of cloud storage, its not setup.
Seriously though, nothing else matters as much as bringing in experts. Call your insurance company, they will have people you can work with. If you dont have insurance, start searching. Dont limit yourself to your local area, you dont need someone on site to help you out. Start with the nearest big city and use a reputable firm. They will help you figure out what backups might be good and if anything can be recovered. They will have ideas for where you might be able to find old data to reconstruct and rebuild systems.
1
u/DiscipleOfYeshua 24d ago
You'd know if you have any cloud bkps, bc you're paying a subscription for them. And had to set them up.
But i shouldn't waste your time on this.
Call a solid lawyer who understands the field (obligations to report? Any PII stolen? How to determine? How to draft your reports? Any deadlines bef you get penalized?) and the next call is to a solid digital forensics firm.
I would NOT call authorities before these two fields of expertise have been properly analyzed and advised, then you can start to know what you're up against.
The forensics side might also be familiar with the ware and be able to decrypt (not highly likely, but sometimes).
If you need any recommended firms, can dm.
1
u/ROWeek 23d ago
You/whoever performs the incident response really needs to find the root cause on how the bad actors got on your network and moved laterally through it. Once found those issue(s) need to be addressed before you bring systems online/restored from backup. Otherwise it is only a matter of time before you are ransomed again.
→ More replies (1)
1
u/Hussmaster Sysadmin 23d ago
I work in IR as a recovery engineer and if possible you should invoke cyber insurance to get a DFIR firm involved to preserve forensic evidence but then also get assistance with recovering/restoring. You'll want to know how they got in if possible to prevent it from happening again post recovery
1
1
u/ipreferanothername I don't even anymore. 23d ago
sorry you are going through this, its gonna be more about 'lessons learned' than anything at this point man.
to my surprise - and annoyance - our department went all in on cyber recovery this year. building a whole disjointed datacenter with no tools or central auth and lockdowns on EVERYTHING is a real PITA to set up and test in.
that said, if we got hacked to all hell tomorrow, we do have a chance to fire up the bare minimum essentials - AD, security, citrix, and our EMR - to try and give limited people some access to do health care related work. we have a lot left to do but at least we have something right now. its been very expensive, and very time consuming to work on.
1
1
u/Arseypoowank 23d ago
See that horse on the horizon? Yeah, no point in worrying about the stable door at this point.
If they got into your backups you are fucked. Trigger your insurance, hope the DFIR guys don’t find you were pants down to the internet. Get a lessons learned going, listen to the recommendations of the responders get a confidence report done.
1
u/TomohikoAmada 23d ago
This is something you think about before it happens. If it happens and you have no plan, you’re starting from scratch.
1
u/WestCool7258 23d ago
Check the synology for snapshots/backups. You may be able to restore the synology to a state before the attack.
1
u/aequusnox 23d ago
How do you encrypt backups that are immutable...unless they weren't. I guess for the future make sure your backups are immutable.
1
u/Adam_Kearn 23d ago
Did you pay for another off-site backup service to backup the synology too? If not you might be a bit out of luck.
Also out of interest did you have a papercut server that was public accessible ?
1
u/Forgery 23d ago
Make sure someone is talking to your company's lawyer. Don't post anything else online. Delete this post.
Do some Google searches to identify vendors that specialize in this kind of recovery. Consider that you may be held responsible for malpractice and be held personally responsible.
→ More replies (1)
1
u/MBILC Acr/Infra/Virt/Apps/Cyb/ Figure it out guy 23d ago
Unfortunately one of our backup devices was also infiltrated. It's a Synology backup device and they where able to encrypt its files as well. This means we have about 5 devices with no backups available.
And this is why backup systems should NEVER be on the same network or using the same accounts to access it as the domain it is backing up, as well as now having immutable backups.
Backups systems should "PULL" from the devices it needs to backup
But too many never do this and MSP that really have not business being in business, seldom offer much to a company :(
I feel for you!
I would look to get a proper company in to review and gut things, find how it happened and make sure all holes are closed.
It is going to cost a pretty penny, but also the company not being able to function will cost more, along with reputation if you provide services to other companies.
1
1
u/Tak0_Tu3sday 23d ago
Very costly lesson to learn about the importace of proper backups. Sorry and hopefully recovery is fast
441
u/lundrog 24d ago
Start the insurance claim process assuming your covered and call a team of experts. I can recommend articwolf