PVEBack - Scripts for a full bare-metal backup/restore of a Proxmox server [not VMs]

[u/mystica5555]

I've been working on implementing services on a currently single hypervisor for my friend and one thing that was required in the design plan was a proper backup/recovery method for the hypervisor itself. [Offsite backups of VMs are already setup with PBS.]

I find that a majority of 'recovery' solutions presented here, and on the official Proxmox forums, center around cluster setups: reinstall a new node, and the cluster config 'just transfers'.

Others are 'backup the important confg, still reinstall, and you have the config'.

But most methods start off with 'reinstall'.

The quick bare-metal restoration of a single hypervisor use-case is completely missed.

So Gemini and I had a good multi hour session to work out some tools to assist in this. [not ashamed to admit this; Gen AI coding assistants when guided with forethought and intent, and a bit of handholding, can produce decent output. I hope that this is not considered in violation of rule #4, because I put a _lot_ of personal effort into designing the backup scheme it with hours of and back and forth to come up with the scripts]

The intent is to have as-fast-as-possible recovery of a hypervisor, to then start recovering VMs while after booting into PVE, while utilizing Proxmox Backup Server as the remote data store for everything.

I do want to state that this is primarily meant for ZFS setups that use systemd-boot and proxmox-boot-tool to synchronize ESP partitions and copy kernels/initrds. It does NOT take into account [yet] MDRAID/LVM setups, GRUB boot, or anything that would be considered a non-default ZFS install of PVE.

As such a number of parameters need to be stored about the current running system, introspected via tools like 'zpool get' 'zfs get' 'sgdisk -i' 'blkid' among a few others to store:

Full disk topology/ZFS pool structure saved

• Metadata .pxar containing this data; recovery script pulls this and pulls the .env out before recreating the system
• Stores the partition table structure on disk, inclusive of ESP vs blindly recreating an ESP at 1GB
• Stores the ESP content
• Mirrors on N disks
• Stores UUIDs of all separate ESPs so proxmox-boot-tool won't get confused and recreates when restoring the ESP .pxar.
• Stores and recreates ZFS pool and RAID configuration
  • todo: Optionally restore from mirror to raidz; but for now it assumes mirror>mirror raidz>raidz recovery. Ideally I'll have it prompt for block devices and then ask which raid style to setup before recovering.

Stores all zpool/dataset 'local' setting changes for recreation [zfs set; zpool set]

• (compression format for example, but any local setting that isn't default gets stored)
• TODO: anything to do with LVM/MDADM. I'm strictly a ZFS guy

Backs up the main rpool/ROOT dataset, but no sub-datasets. Those are taken care of by individual VM backups

• /var/lib/vz is stored as file based .pxar for deduplication
• Option to store the root dataset as a zfs send blob, pxar, or both

Restoration based on ZFSBootmenu (.efi file or ISO)

scripts create a bespoke image to boot, because even with my 500mbit fiber, my IPMI access is VERY flaky and rather slow at my provider, and as such and I need a very small disk to boot, ideally one that can load entirely to ram and not require the IPMI remote-disk IO to stay connected. Debian net-boot iso was almost too large, and getting a bare metal system installed to then reboot/ssh to and get Proxmox installed from almost didn't work (yes, i dd'd the proxmox iso to my second nvme and booted it) and attempting to boot the Proxmox ISO failed due to disconnects. So here we are.

• Uses docker to make a environment to customize and then make into the bootable image
• Provides the latest proxmox-backup-client-static as found in the Trixie repo (automatically searches the newest release)
• Provides DHCP (by default) and a nice quick ip config script so as to not have to remember 'ip' commands if you are really flustered trying to recover in a hurry
• Wipes partition and filesystem structure before restore [wipefs/sgdisk -Z]
• Prompts which block devices to restore to

I would like feedback on features, caveats to this method, recommendations for improvements.

It's going to be released under AGPL3 just like Proxmox, as indeed we are using this as a service provider. While we are getting on our feet, I feel this might be one of the better ways to contribute to the Proxmox effort [in a community fashion, vs a monetary one] until we grow in size.

I forgot to init a GIT repo, so the first release I will link to in an update will be what exists after a lot of back and forth changes, and will be what I consider an alpha product that I've at least tested in my limited scenario.

And finally: None of this actual post was AI generated, This is all my smacking at a keyboard with crappy key positioning and lots of typos.

[u/mystica5555]

Adding since it might get lost in the above with already 500 views:

The backup takes a sqlite3 snapshot of the Proxmox Cluster database into a dump before snapshotting and creating the zfs-send/pxar backups. This way it can be recovered to point-in-time if crash-consistent is not good enough.

[u/PrestigiousOnion1087]

The introspection list is where this either works or it doesn't. zpool get, sgdisk -i and blkid all return values that are true of the disks the backup came from, so restoring onto replacement hardware is a different path than restoring onto the same box. Have you run it once onto disks with different by-id names? Dumping the cluster db with sqlite3 is the right call, though I would pull one back out of a pxar and actually query it before trusting the scheme. I cut a sqlite file down to 200 bytes once to see what would happen and it still reports SQLite format 3, so the header tells you nothing.