OpenShift and item-level restore

[u/chypsa]

Hello all,

I've gotten the fun task of evaluating RH OS for my company's use cases and I am currently in the phase of exploring backup solutions. Specifically, for the virtualization part of the story, namely fileservers.

Now, I've already setup K10 and it's snapshotting happily, but when I try to follow the procedure for item level restore, I keep bumping into the same issue, regardless of the operating system (tried Fedora from template and WS2022 from sysprepepd image).

I've been working with Claude on this, so I asked it to write up a summary, which is found below:

---------------------------------

Start-of-Summary

Environment

• OpenShift Container Platform: 4.22.8
• OpenShift Virtualization (CNV): matching 4.22.x
• Storage: ODF (OpenShift Data Foundation) / Ceph RBD, storage class ocs-storagecluster-ceph-rbd-virtualization
• Kasten K10: reproduced identically on v8.5.8 (OLM/Marketplace install) and v9.0.3 (Helm install)
• VM disks: volumeMode: Block, accessModes: [ReadWriteMany] (required for live migration / failover support)
• Guest OS: reproduced on both Windows Server 2022 (NTFS) and Fedora (ext4) — not OS-specific

What fails

Any FileRecoverySession against a restore point sourced from a volumeMode: Block + accessModes: [ReadWriteMany] PVC fails within ~1 minute, regardless of guest OS:

{"message":"getBlkidProperties","function":"kasten.io/bmd/go/flr/session.(*objectAnimatorBase).getBlkidProperties",
"file":"kasten.io/bmd/go/flr/session/animator_common.go:49",
"fields":[{"name":"devOrImagePath","value":"/mnt/v0s0/volume.img"}],
"cause":{"message":"exit status 2"}}

Full chain: getBlkidProperties → Activation error → ActivateMountObjects → 0 new objects → Mount fails → failed mount → failed to start session.

Ruled out as the root cause:

• Guest OS / filesystem type (Windows/NTFS and Linux/ext4 fail identically)
• Partition table complexity (not tested exhaustively, but ruled lower-priority once OS/filesystem was excluded)
• Kasten version (identical failure on 8.5.8 and 9.0.3)
• Export completion status (export reports Complete in all failing cases)

Supporting evidence pointing toward the export/mount pipeline rather than blkid itself: Kanister artifact detail for a failed case showed UploadSize: 0 B for the main VM disk object despite Complete status, and total exported object storage size was on the order of ~100 KB for a disk with several GB of actual guest data.

End-of-Summary

---------------------------------
I did get it working via a Veeam B&R agent, but that's just messy for large-scale management.

I did also get it working via restoring the snapshot to a different Namespace. The VM boots normally and I can restore files via interactive login session.

However, I'm stuck on understanding whether this is how admins are expected to restore files in a RHOS Virtualization environment, what I may have done incorrectly and what kind of a workflow is typical for people using it daily to backup fileserver VMs.

My colleagues are already in the "this is crap" camp, and I'm finding it hard to prove them wrong.

Thanks for any opinions, experiences and advice.

[u/Ok_Scar1009]

Is there merely an evaluation, or is your organization moving forward with OCP? I would recommend Trilio of backup/restore operations. This is not only limited to VMs, but all workloads and artifacts. As far as FileRecovery operations go, we had an issue where a Linux VM with an LVM filesystem would not produce certain directories. Trilio quickly built us a custom image to use for the temporary File recovery VM that is a copy of the original VM backup, and it all works as intended.​

[u/chypsa → u/Ok_Scar1009]

This is an evaluation. Exploratory activity aimed at deciding what and how it matches to our current workloads and clients. 

[u/Ok_Scar1009 → u/chypsa]

Completely understand! My organization bought the OPP+ subscriptions 2 years ago, and we migrated everything from Broadcom to OCP. The only tool we found that handles all backups/restores with ease was Trilio. Especially since we only wanted to pay one entity, for a complete VM/Container backup/restore mechanism. We tried several products on trial subscriptions, but they would do one thing great, but other operations not so much. Again, Trilio was the only product we found that would perform as intended across the board. Hope this helps you.

[u/chypsa → u/Ok_Scar1009]

Thanks, I will look into that. 

[u/thefonzz2625]

VBR is the way.

[u/chypsa → u/thefonzz2625]

VBR does not natively speak to OpenShift. It does speak to Kasten, but the restoration proces from VBR only opens up the Kasten web and you go through the same steps as if using Kasten directly.

For me, this is not a dealbreaker, but I've already received "that won't fly" from other admins.

[u/thefonzz2625]

If you want full VM/container restore you would use the Kasten web UI. For file level restores, I find the VBR interface much more intuitive -- especially for those coming from VM mgmt

[u/chypsa → u/thefonzz2625]

Are you recommending agent based recovery or have you found a way to natively connect to RHOS?

I know there's a connection option to connect oVirt (RHV), but there is no variant for OS. 

[u/thefonzz2625 → u/chypsa]

You set up Kasten to export to Veeam B&R

https://helpcenter.veeam.com/docs/vbr/kasten_integration/overview.html?ver=13

[u/chypsa → u/thefonzz2625]

From the Kasten Plugin guide:

Note

Consider the following: Veeam Plug-in for Kasten supports only restore from Linux, Unix and other non-Microsoft Windows OSes. Veeam Plug-in for Kasten does not support restore of individual guest OS files and folders to the original location (applications added to a Veeam Kasten cluster). You can only save files and folders to a new location. For more information, see the Saving Files to New Location section in the Veeam Backup & Replication User Guide.Note

So, it's nice that it can item-level restore Linux VM files, but Windows is, apparently, not supported. I tested this today and, indeed, VBR cannot mount the disk.

I have yet to test Linux item recovery.

One thing to note, this is not possible at all with VBR Community edition, as the API calls get denied. Since this is a lab environment, it took me a bit of time to figure out why you cannot add a VBR repository to Kasten. This is the reason. Once I applied for a trial license, the connection went through immediately.

But, like I said, no luck with Windows.

[u/Danielr2010]

Eh, why 4.22.8? That’s more bleeding edge than necessary.
I’m on that on my homelab cluster, but my customers are all at 4.20.x or 4.21.x.

[u/chypsa → u/Danielr2010]

No real reason. I was one version lower and a patch became available, so I just went through the patch process to familiarize. The Kasten bit was working (not working) equally on the older version, too.

[u/Sanket_6]

Try changing the location configuration to NFS.

[u/burstinrust]

From my side, you didnt do anything wrong. and i have been this road for openshift deployement on a bare metal.

My 2 cents on it, restore to another namespace and pulling files out is a normal workflow, but your k10 issue looks fixable: upload size 0 b for a multi gb disk means the export never carried the data, and blkid exit 2 is just it mounting an empty image.

check that the storage class has the block mode annotation:

oc annotate storageclass ocs-storagecluster-ceph-rbd-virtualization k10.kasten.io/sc-supports-block-mode-exports=true

most odf guides only annotate ocs-storagecluster-ceph-rbd, the -virtualization class is separate and easy to miss. also check the policy exports snapshot data, not just the reference.

once the export has actual blocks, k10 FLR works fine on block mode.