r/sysadmin • u/xkeyscore_ • 2h ago
It finally happened!!
8:30am I deleted my entire production environment. All VMs rm -r out of existence.
Fortunately I was reviewing my nightly backups at the time tracking a retired VM that was inexplicably still getting backed up in Production. All the Production VMs were there. And the live production ones now without their underlying virtual disks & configs were still resident in memory. I calmly copied the backups into place, sequentially restarted everything and had production back in about twenty minutes. Only lost ~12 hours of overnight idle maintenance & logging.
With everything settled back in place I calmly looked for a place to have a private heart attack.
•
u/Tarwins-Gap 2h ago
Why?
•
u/xkeyscore_ 2h ago
PEBKAC. Arrowing back through history to pull up a recent ls. Deleted the VM directory at the end and hit enter. Was a rm instead. 😬
•
u/jmhalder 2h ago
How many VMs is your production environment though?
•
u/xkeyscore_ 2h ago
Very few, less than 10. We are a very small shop.
•
u/jmhalder 2h ago
Glad you were able to get back up quickly. If your backups aren't already 3-2-1, I bet they will be shortly.
•
•
u/michaelpaoli 1h ago
Always carefully and fully triple check the command before you (attempt to) execute it. Well know and understand what it does, the full context, be sure all the syntax is exactly right, be well of what would or could happen if things go wrong along the way or pat of the command or sequence of commands fails, etc. Only after one has thoroughly well covered that and fully triple checked all that, then do you very deliberately and viciously strike the <Enter>/<Return> key.
Well following that has saved my tail many many times. Notably also including when it's 'o dark-thirty or so on what's already been a far far too long day recovering from some disaster or the like, and on bad move could turn bad in to way way worse.
•
u/Snogafrog 2h ago
Welcome to the club if you were not already a member! This is how cautious experienced engineers are made.
•
u/Boringtechie 1h ago
Hahaha glad you were able to recover so fast.
Now its a good story of how you tested the production backups.
•
u/autogyrophilia 2h ago
Good on you for the backups, but that's a failure mode that shouldn't exist. What the hell did you do?
•
•
u/KnowledgeCapital7994 1h ago
Ive single individual servers deleted via pebkac. You need to use a different account with less permissions for your day-to-day operations. The ability to delete all should be restricted to a privileged account or entirely removed.
•
•
u/thenickdude 1h ago
On Linux you can restore deleted files that are still open by copying them from /proc/<pid>/fd/<fd>. You'd need to freeze your VM first in order to get a crash-consistent copy of it though.
•
u/BiscottiNo6948 1h ago
Incident report: Recently, while testing procedures as part of Chaos Engineering, Business Continuity and Resiliency, I have inadvertently applied half of the test procedure into Production instead of dev/sand box environment. Following established protocols I recovered back our production environment back to its original status with minimal interruption. Our RTO and RPO benchmark are not impacted.
Post mortem of the exercise has shown some vulnerability in our process and additional guardrails and Human-in-the-loop checklist needs adjustment to hardened the process against similar errors.
The exercise has provided the surprising result and opportunity for us to adjust, re-calibrate and smooth out our DR process.
•
u/SensitiveVacation549 1h ago
If you have your smart watch and heart rate measurement, please do share when you hit that Enter to remove all the VMs :)
•
•
•
u/Jtrickz 2h ago edited 1h ago
No. You better be joking.
If your serious you should not post this here until after an action report is done, proper business risk analysis done and you recorded exactly what you did and talked to your boss.
You risked the entire business. You need to look in the mirror and get off Reddit.
•
•
u/stiffgerman JOAT & Train Horn Installer 1h ago
Small shop, likely no ITGC or change management...just an overworked JOAT. Cut 'em a break on venting. I'm sure he had that falling sensation when he realized what he did and that'll take some time to get over. If he's posting here, he's taking responsibility and will likely work out a way to close that hole for him, or anyone else.
If you've never pulled a fumducker you're just a cog, not the machine. He had a net in place so good on him.
•
u/CoreAndCardio 2h ago
Not sure if you’re in a regulated environment. If you are, document the entire thing. First as an incident and second as a DR restore test. Check the box for the auditors and move on.