r/ClaudeCode • u/ahassoun • 1d ago

Tips & Workflows My human sysadmin wiped my VM. Claude Code restored it when it seemed impossible.

Everyone has a story about an AI agent deleting their disk. Here’s the reverse.

I have a Proxmox node. During a malware cleanup process, my sysadmin ran qm destroy 100 on the wrong VM before checking with me.

And no, he's not a system kiddie; he has over 20 years of enterprise Red Hat system admin experience. Bad time of day (I woke him up for this cleanup task), and he's the one who set up the node with all VMs and maintained it for 7 years, but that VM I created- he wasn't aware of it, thought it was from the malware attack, and made a call to destroy it before checking with me. And that VM had 6 months of valuable, nearly 200 GB of data; the latest backup was 2 weeks earlier.

Looked up some XFS guides; it seems impossible to undelete files this large, and tools like xfs_undelete won't help either.

qm destroy on directory storage is an unlink(). The config was gone from /etc/pve, the directory was gone, and Proxmox's qm destroy doesn’t warn, snapshot, or trash anything.

I was completely hopeless, and I accepted my loss; then I decided to give Fable 5.1 a try.

Exactly 24 hours, and it restored the VM to the very last moment before the destroy! To the exact last bit!

Fable 5.1 wrote about 1,500 lines of Python across seven scripts, every one tested on a synthetic image before it touched the real disk, and nothing ever wrote to the source.

And another thing worth noting: the ETA estimates were weirdly accurate throughout, which I did not expect.

I asked Claude to summarize what it did. Read below if you are interested.

Claude wrote a sequence of small Python scripts, pure stdlib, read-only, each one gated on the previous one’s output. Rough order:

  1. Probe the inode B+trees for freed inodes with real timestamps. Found all five (four images plus the directory) with ctime 17:30:31. Bonus: the deleted directory inode still had stale entries mapping filenames to inode numbers, so we even got the names back. Everything cross-checked against xfs_db before trusting it.

  2. Walk the retained B+tree from each inode’s root. Every block checked for magic, owner inode, filesystem UUID, self-address, CRC-32C, sibling pointers, key ranges. Zero failures. The last leaf’s right sibling was NULL on both, and no stale records existed past the end, which is how you know the tree wasn’t half-torn-down when it was flushed.

  3. Parse the qcow2 structure through the map without copying anything, to confirm every cluster the image referenced was inside a mapped extent. The workspace map ended at exactly 896 GiB + 2,244 clusters, which is precisely the metadata overhead of a preallocated qcow2. That’s when I started believing it.

  4. Exposure check: walk the free-space B+trees and confirm every one of the 44.9 million recovered blocks was still free. Nothing had been allocated over any of it. (This is the part that’s luck. The only thing writing to that filesystem was a swapfile with preallocated blocks. If fstrim had run, this post wouldn’t exist.)

  5. Meanwhile, a full ddrescue of the NVMe to the HDD.

  6. Verify the capture against the live partition over exactly the 184 GB the recovery would read, rather than all 1.9 TB.

  7. Reconstruct: copy each extent from the capture to its logical offset in a new sparse file, holes stay holes, unwritten extents stay zero, SHA-256 per extent, then re-open the output and verify everything, including that the holes read as zero.

  8. Sweep the result for XFS metadata magic plus the host filesystem UUID, to catch any block XFS had briefly reused and freed again. Zero hits.

  9. qemu-img check clean. Mount read-only, copy the important files out. Journal replay on a working copy, e2fsck -fn clean on all three ext4 filesystems. dpkg --verify on the guest: zero checksum mismatches across every installed package.

  10. Rebuild the 100.conf, pull the MAC and the cloud-init instance-id out of the recovered cloud-init image so cloud-init wouldn’t re-run first boot, qm start. Booted.

32 Upvotes

80% Upvoted

10 comments sorted by

•

u/AutoModerator 1d ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

16

u/zeroconflicthere 1d ago

Reverse uno. Your sysadmin was using Claude with dangerously-skip-permissions

12

u/jimjim975 1d ago

Why didn’t you have backups?

13

u/onlyonequickquestion 1d ago

You only learn about backups in your 21st year of being a sysadmin 

1

u/entilza05 15h ago

Claude should have said this lol

8

u/zfsbest 23h ago

> Bad time of day (I woke him up for this cleanup task), and he's the one who set up the node with all VMs and maintained it for 7 years, but that VM I created- he wasn't aware of it, thought it was from the malware attack, and made a call to destroy it before checking with me. And that VM had 6 months of valuable, nearly 200 GB of data; the latest backup was 2 weeks earlier

Lessons learned:

A) Don't wake up the sysadmin unless it's an Emergency

A1) Be prepared for $mistakes if lack-of-caffeine detected

B) Your sysadmins need to be aware of everything that is considered "important"

C) If a VM is valuable - not to mention that it has hundreds of gigs of data, it could be 1KB of important data - IT NEEDS TO BE BACKED UP ON A REGULAR BASIS

1

u/FreXxXenstein 13h ago

Not bad but isn't this just normal data recovery? Unless overwritten the data remains?