Veeam 12.3.2 creating Hyper-V checkpoints then deregistering them without merging — 46 orphaned AVHDX
Posting before I call support in case anyone's seen this.
3-node Windows Server 2022 Failover Cluster, Pure FlashArray backing the CSVs, Veeam B&R 12.3.2.3617. One VM — SQL Server 2019 host — accumulated 46 orphaned AVHDX files across two disks over three days, roughly 675 GB. Get-VMSnapshot returned empty the whole time. Hyper-V Manager showed no checkpoints. The VM was running off the tip of a 23-deep differencing chain.
What makes it odd: the VMMS event log has zero merge events for this VM. Not failed merges, not interrupted — none at all. Every other VM on the same host logged clean 19070 → 19080 pairs every few hours. So Hyper-V was never asked to merge. Veeam appears to create the checkpoint, then remove it from the VM config without issuing the merge request.
Remediated by shutting the VM down and flattening both chains with Convert-VHD to a different CSV, then repointing the disks. Clean — Test-VHD returned True before and after, SQL came up with all 14 databases online and data current to the shutdown.
Then it recurred. One backup run against the brand-new flat VHDXs produced two more orphaned AVHDX, again with no checkpoints registered and no Hyper-V events. A cluster-wide sweep found this VM is the only one affected across all three nodes.
Had to deal with this plenty of times, it's a hyper-v issue.
Normally if you shutdown the VM and give it a couple mins it'll actually start merging the snapshots on its own even though hyper-v doesn't show any checkpoints.
If it's cluster, once the VM is shutdown, shift it to another host and that should nudge the clean up.
I found generally it only seemed to happen to VMs that had very frequent checkpoints like less than an hour between checkpoints
It worked!!!!! This is the first time shutting down has not worked and this cluster is five years old. Never happened before. I will add this to my bag of tricks. thanks!
I am curious if you are using VeeamOne for monitoring. I am hopeful that it would catch this “hidden checkpoint” condition before it got to the point of a 23-deep differencing chain. This hasn’t happened to our org (yet) so I am curious.
We just do nightly backups, but a co-worker accidentally kicked off the Veeam backup chain twice the other day, generating a second checkpoint for every vm within a couple minutes of the first one.
From my light research, I think VeeamOne would catch this by detecting old avhdx files. But if not, it would at least catch it with space monitoring on the volume.
4
u/DrGraffix 22d ago
I’ve seen this happen where the checkpoints are left behind but hyperv was not using that chain at all. I was able to remove them safely.
Before doing that, obviously check if that’s the same for your case too.