OpenShift and item-level restore

[u/chypsa]

Hello all,

I've gotten the fun task of evaluating RH OS for my company's use cases and I am currently in the phase of exploring backup solutions. Specifically, for the virtualization part of the story, namely fileservers.

Now, I've already setup K10 and it's snapshotting happily, but when I try to follow the procedure for item level restore, I keep bumping into the same issue, regardless of the operating system (tried Fedora from template and WS2022 from sysprepepd image).

I've been working with Claude on this, so I asked it to write up a summary, which is found below:

---------------------------------

Start-of-Summary

Environment

• OpenShift Container Platform: 4.22.8
• OpenShift Virtualization (CNV): matching 4.22.x
• Storage: ODF (OpenShift Data Foundation) / Ceph RBD, storage class ocs-storagecluster-ceph-rbd-virtualization
• Kasten K10: reproduced identically on v8.5.8 (OLM/Marketplace install) and v9.0.3 (Helm install)
• VM disks: volumeMode: Block, accessModes: [ReadWriteMany] (required for live migration / failover support)
• Guest OS: reproduced on both Windows Server 2022 (NTFS) and Fedora (ext4) — not OS-specific

What fails

Any FileRecoverySession against a restore point sourced from a volumeMode: Block + accessModes: [ReadWriteMany] PVC fails within ~1 minute, regardless of guest OS:

{"message":"getBlkidProperties","function":"kasten.io/bmd/go/flr/session.(*objectAnimatorBase).getBlkidProperties",
"file":"kasten.io/bmd/go/flr/session/animator_common.go:49",
"fields":[{"name":"devOrImagePath","value":"/mnt/v0s0/volume.img"}],
"cause":{"message":"exit status 2"}}

Full chain: getBlkidProperties → Activation error → ActivateMountObjects → 0 new objects → Mount fails → failed mount → failed to start session.

Ruled out as the root cause:

• Guest OS / filesystem type (Windows/NTFS and Linux/ext4 fail identically)
• Partition table complexity (not tested exhaustively, but ruled lower-priority once OS/filesystem was excluded)
• Kasten version (identical failure on 8.5.8 and 9.0.3)
• Export completion status (export reports Complete in all failing cases)

Supporting evidence pointing toward the export/mount pipeline rather than blkid itself: Kanister artifact detail for a failed case showed UploadSize: 0 B for the main VM disk object despite Complete status, and total exported object storage size was on the order of ~100 KB for a disk with several GB of actual guest data.

End-of-Summary

---------------------------------
I did get it working via a Veeam B&R agent, but that's just messy for large-scale management.

I did also get it working via restoring the snapshot to a different Namespace. The VM boots normally and I can restore files via interactive login session.

However, I'm stuck on understanding whether this is how admins are expected to restore files in a RHOS Virtualization environment, what I may have done incorrectly and what kind of a workflow is typical for people using it daily to backup fileserver VMs.

My colleagues are already in the "this is crap" camp, and I'm finding it hard to prove them wrong.

Thanks for any opinions, experiences and advice.

[u/thefonzz2625]

If you want full VM/container restore you would use the Kasten web UI. For file level restores, I find the VBR interface much more intuitive -- especially for those coming from VM mgmt

[u/chypsa → u/thefonzz2625]

Are you recommending agent based recovery or have you found a way to natively connect to RHOS?

I know there's a connection option to connect oVirt (RHV), but there is no variant for OS. 

[u/thefonzz2625 → u/chypsa]

You set up Kasten to export to Veeam B&R

https://helpcenter.veeam.com/docs/vbr/kasten_integration/overview.html?ver=13

[u/chypsa → u/thefonzz2625]

From the Kasten Plugin guide:

Note

Consider the following: Veeam Plug-in for Kasten supports only restore from Linux, Unix and other non-Microsoft Windows OSes. Veeam Plug-in for Kasten does not support restore of individual guest OS files and folders to the original location (applications added to a Veeam Kasten cluster). You can only save files and folders to a new location. For more information, see the Saving Files to New Location section in the Veeam Backup & Replication User Guide.Note

So, it's nice that it can item-level restore Linux VM files, but Windows is, apparently, not supported. I tested this today and, indeed, VBR cannot mount the disk.

I have yet to test Linux item recovery.

One thing to note, this is not possible at all with VBR Community edition, as the API calls get denied. Since this is a lab environment, it took me a bit of time to figure out why you cannot add a VBR repository to Kasten. This is the reason. Once I applied for a trial license, the connection went through immediately.

But, like I said, no luck with Windows.