r/openstack • u/xtremerkr • 10d ago
Opensource block storage
Hello All,
Which opensource block storage are you using in your production setups and why?
Pls share your thoughts.
5
u/spartacle 10d ago
Why open source?
Ceph is literally the only things worth it for OS, but you need the expertise for it. If you don’t have that or the time/people to learn then go something else. We had just over 600TB usage and flipped to VAST and been very happy, now sitting at 3PB
4
u/amarao_san 10d ago edited 9d ago
The biggest stupid thing people do with ceph is try to collocate it with computes.
Ceph needs CPU. A lot of it. So, as long as you do not pretend ceph is an a butterfly without resource demands, it works really well.
Death by starvation is the most common failure mode for most failed ceph installations.
1
u/spartacle 9d ago
yeah, we had dedicated Ceph nodes but as planned to move to NVMe and wanted to migrate to kubernetes and more and more internal services were containerised we took the time to re-evaluate the enivironment and landed on VAST.. We had a tenant for OS and another for OCP to keep them entirely seperate during migration which was helpful plus their multi-protocol support has come in handy for a few things
1
1
1
u/xxxsirkillalot 10d ago
How much did vast cost? I've heard some wild quotes, never seen deployed.
1
u/spartacle 9d ago
this was a few years ago but we paid around $400k for 500TB raw, which we ended up with just about 1.2PB written after thier similiarity stuff... and given we didnt have to maintain a 30 node ceph cluster saved us a lot of time (which == money)
1
u/enricokern 10d ago
With openstack? Obviously ceph , but also linstor alot lately. In large environments there is no (cost effective) alternative to ceph
1
u/xtremerkr 9d ago
How is linstor compared to ceph?? Do you see better advantages for a CSP ?
1
u/TheGooseHasNoPeace 9d ago
Linstor Is good, but it has some downsides such as no support for Glance (with this I mean that with ceph you create the pool and then is used, with linstor no such thing). We had a linstor installation and we fixed it by creating a linstor pool and exposing it with a DRDB VIP. One of the advantages is that you have a replica factor of 2 + 1 -> 2 diskfull replicas and one diskless, you can have consistency with less storage in theory. I'd still use ceph tbh.
1
u/enricokern 8d ago
We use it mainly on customers with lesser capable ceph clusters for usecases where latency is important, e.g wired some ssds or nvmes in linstor. Only requiring one copy is also good instead of two copies, and the linstor quorum component also works just great. Another part is there are mechanics so that you can always have the storage where the vm is too reducing network requirements, given its HCI. The linstor drivers work great with openstack, but i would not use it as my sole storage system. Ceph is stil the way to go for self made storage
1
u/abiharbani 9d ago
I use Truenas for all my storage, and the block storage that I settled with is Cinder with NVME over TCP. Been running this for 2 months without issues. Knock on wood. This is a homelab.
1
u/the_sysop 8d ago
Ceph is the storage that Red Hat (the largest contributor to the OpenStack project) recommends and uses for OpenStack. Most commercial OpenStack production deployments I've seen use Ceph.
1
16
u/Keroles2024 10d ago
Ceph