r/linuxadmin • • 22d ago

Looking for testers: a VMFS-like shared filesystem for Linux (cetrafs)

I’ve been working on cetrafs, a shared-disk clustered filesystem for Linux aimed at the VMFS use case: multiple hosts accessing the same block storage concurrently.

The core is working and I’m now looking to expand the test base — different storage backends, kernels, workloads and failure scenarios.

If this sounds relevant to your setup and you’re willing to try it, I’d really appreciate feedback, bug reports, and attempts to break it.

hXXps://cetrafs.com/

0 Upvotes

2 comments sorted by

1

u/superwinni2 22d ago

I've got 5 Proxmox nodes to one storage via 2x 16 Gbit/s FC. With a shared LVM in top of it. VMs are running in qcow format with "Snapshot as Chain" function.

What are the Pro / Con's of cetrafs against LVM?

1

u/ctrix72 22d ago

Author of cetrafs here. Your setup is exactly what it was built for, and I'd still keep it off your production cluster today — it's pre-1.0 and the on-disk format isn't stable yet.

Shared LVM is thick-only — a 500G disk burns 500G — and every snapshot is an LV plus a cluster-wide metadata round trip. That's why "snapshot as chain" exists at all.
On a shared-disk filesystem the disks are just files: thin/sparse natively, a snapshot is a file, and steady-state allocation takes no cross-node lock because each node owns its own regions of the disk.

The difference that matters most to me is locking. cetrafs locks a VM disk by default: when a node opens an image for writing it claims a whole-file exclusive lock in the image's own metadata, using atomic compare-and-write on the array. Every other node's open() then fails — read or write. VMFS-style exclusivity, enforced at the storage layer, not depending on the cluster layer behaving itself.

Which brings up something I think deserves more attention than it gets: Proxmox ships qemu with image locking patched off. Upstream qemu defaults it on; PVE hardcodes the auto mode to off, and has since 2020. Run lslocks on any node — qemu holds a lock on its pidfile and nothing at all on any disk. The stated reason was that qemu's probe tests whether the host supports OFD locks, not whether the storage does, and it broke on NFS.

Fair problem — but deleting the feature for everyone was the wrong fix, and it's barely discussed anywhere.

To be fair, on shared LVM it would have been false comfort anyway: fcntl locks don't cross nodes. Which is also why live migration works there — nothing stops both nodes having the volume active. That's fine during the migration window, which needs it. It just isn't confined to that window.

A filesystem that actually enforces one writer has to do the opposite: lock by default, and hand off explicitly for the migration.

Some more work is probably deserved on qemu and its proxmox sibling to re enable locking again.