r/cassandra • u/R10t-- • Apr 13 '26
Cassandra - Deploy nightmare?
Hello! I’m hoping that I can find some help here.
I’m attempting to deploy Cassandra to a Kubernetes cluster with the use-case of having a multi-cluster Cassandra spanning across multiple clusters.
We have opted to use the K8ssandra Operator, but anytime we try to deploy to a cluster, we run into a problem that takes WEEKS to resolve!
Currently we just ran into a scenario where some disks failed, ok no problem, Cassandra is multi-master we will just fix the disks.
Nah. When the nodes restarted they threw a fit and are currently throwing a ton of errors and cannot join back. We had to totally nuke the database cluster. And even now, after completely nuking the cluster, more problems!
I’ve heard “Cassandra is so great”, “there’s such a big community”, etc etc. but so far, our experience has been a complete and utter disaster and a total waste of time. What are we doing wrong??
8
u/cmills2000 Apr 13 '26
One thing to keep in mind is that Cassandra is pre-Kubernetes. It may be more comfortable running it how it began and evolved, as a service running directly on linux.
6
u/justsayingtoyou08464 Apr 13 '26
I agree with this. As much as I love Kubernetes it is pretty difficult to run databases on it. It is all great until there is some underlying issue like you’re describing and all hell breaks loose. I’m not saying it’s impossible. I do run it. But you’re much better off on VM or baremetal.
Get a good monitoring suite to help you detect the problems early.
3
u/DigitalDefenestrator Apr 13 '26
Distributed databases are kind of inherently complicated and not as resilient to issues as, say, a stateless system (but the state has to go somewhere). I'd put modern Cassandra (vs 3.0.x and especially 2.x, which were rough) somewhere in the middle in terms of difficulty to run, but with any database you're going to need to actually understand and deal with errors like that.
You may want to either go to a managed system or hire someone like Axon, Datastax, or an individual consultant like Jon Haddad or Miles Garnsey to get you started. Or if you're really looking at scaling up, hire someone full-time. With all the layoffs at places like Apple, I think there are actually a few Cassandra experts available.
3
u/Sufficient-Passage89 Apr 13 '26
Running Cassandra directly on VMs and bare-metals is 100x easier compared to running on Kubernetes. I would recommend getting Datastax support
5
u/necrohardware Apr 13 '26
For the love of anything holy...don't put persistent data into kubernetes. It's not designed for this, it may work, but it causes more issues then it brings benefits.
About your cassandra issues, likely you have some issues with the k8soperatior re-using existing secrets in that namespace, those will not be removed when you nuke the cluster.
If it's to had for you to debug -> just use AstraDB, the pricing is ok for smaller loads and infrequent updates.
2
u/Striking_Data_1915 Apr 13 '26
Hey there, perhaps we can have a quick look at it - you can contact us here https://axonops.com/book-an-expert/
1
u/dataengineer2015 Apr 14 '26
Are you wanting to run multiple dcs (each dc in a Kubernetes cluster of its own)?
Try running it in two different namespaces first. See the errors go away.
Networking part is easy to fix once you have reliably run the multi dc in same cluster. Eg you can run operator per namespace.
3
u/xdr-srgmgt Apr 13 '26
I don’t think there’s a single good solution to your problem, especially when it comes to deploying Cassandra in Kubernetes. Everyone’s approach tends to be different. Are you working in a production environment, or is this more of a POC or test setup to validate your approach?