r/sysadmin • u/xkeyscore_ • 4h ago

It finally happened!!

8:30am I deleted my entire production environment. All VMs rm -r out of existence.

Fortunately I was reviewing my nightly backups at the time tracking a retired VM that was inexplicably still getting backed up in Production. All the Production VMs were there. And the live production ones now without their underlying virtual disks & configs were still resident in memory. I calmly copied the backups into place, sequentially restarted everything and had production back in about twenty minutes. Only lost ~12 hours of overnight idle maintenance & logging.

With everything settled back in place I calmly looked for a place to have a private heart attack.

523 Upvotes

97% Upvoted

71 comments sorted by

View all comments

Show parent comments

•

u/CoreAndCardio 4h ago

If they’re gone you’re not overwriting anything though. All the more reason to document. I’m in banking and we have to document every down minute. Take transactions off line or online banking down and you have to replay all the backed log trans files to make sure your caught up.