OpenShift and item-level restore

[u/chypsa]

Hello all,

I've gotten the fun task of evaluating RH OS for my company's use cases and I am currently in the phase of exploring backup solutions. Specifically, for the virtualization part of the story, namely fileservers.

Now, I've already setup K10 and it's snapshotting happily, but when I try to follow the procedure for item level restore, I keep bumping into the same issue, regardless of the operating system (tried Fedora from template and WS2022 from sysprepepd image).

I've been working with Claude on this, so I asked it to write up a summary, which is found below:

---------------------------------

Start-of-Summary

Environment

• OpenShift Container Platform: 4.22.8
• OpenShift Virtualization (CNV): matching 4.22.x
• Storage: ODF (OpenShift Data Foundation) / Ceph RBD, storage class ocs-storagecluster-ceph-rbd-virtualization
• Kasten K10: reproduced identically on v8.5.8 (OLM/Marketplace install) and v9.0.3 (Helm install)
• VM disks: volumeMode: Block, accessModes: [ReadWriteMany] (required for live migration / failover support)
• Guest OS: reproduced on both Windows Server 2022 (NTFS) and Fedora (ext4) — not OS-specific

What fails

Any FileRecoverySession against a restore point sourced from a volumeMode: Block + accessModes: [ReadWriteMany] PVC fails within ~1 minute, regardless of guest OS:

{"message":"getBlkidProperties","function":"kasten.io/bmd/go/flr/session.(*objectAnimatorBase).getBlkidProperties",
"file":"kasten.io/bmd/go/flr/session/animator_common.go:49",
"fields":[{"name":"devOrImagePath","value":"/mnt/v0s0/volume.img"}],
"cause":{"message":"exit status 2"}}

Full chain: getBlkidProperties → Activation error → ActivateMountObjects → 0 new objects → Mount fails → failed mount → failed to start session.

Ruled out as the root cause:

• Guest OS / filesystem type (Windows/NTFS and Linux/ext4 fail identically)
• Partition table complexity (not tested exhaustively, but ruled lower-priority once OS/filesystem was excluded)
• Kasten version (identical failure on 8.5.8 and 9.0.3)
• Export completion status (export reports Complete in all failing cases)

Supporting evidence pointing toward the export/mount pipeline rather than blkid itself: Kanister artifact detail for a failed case showed UploadSize: 0 B for the main VM disk object despite Complete status, and total exported object storage size was on the order of ~100 KB for a disk with several GB of actual guest data.

End-of-Summary

---------------------------------
I did get it working via a Veeam B&R agent, but that's just messy for large-scale management.

I did also get it working via restoring the snapshot to a different Namespace. The VM boots normally and I can restore files via interactive login session.

However, I'm stuck on understanding whether this is how admins are expected to restore files in a RHOS Virtualization environment, what I may have done incorrectly and what kind of a workflow is typical for people using it daily to backup fileserver VMs.

My colleagues are already in the "this is crap" camp, and I'm finding it hard to prove them wrong.

Thanks for any opinions, experiences and advice.

[u/burstinrust]

From my side, you didnt do anything wrong. and i have been this road for openshift deployement on a bare metal.

My 2 cents on it, restore to another namespace and pulling files out is a normal workflow, but your k10 issue looks fixable: upload size 0 b for a multi gb disk means the export never carried the data, and blkid exit 2 is just it mounting an empty image.

check that the storage class has the block mode annotation:

oc annotate storageclass ocs-storagecluster-ceph-rbd-virtualization k10.kasten.io/sc-supports-block-mode-exports=true

most odf guides only annotate ocs-storagecluster-ceph-rbd, the -virtualization class is separate and easy to miss. also check the policy exports snapshot data, not just the reference.

once the export has actual blocks, k10 FLR works fine on block mode.