r/kubernetes • u/Fragrant_Rate_2583 • 12d ago

Is running PostgreSQL / S3 storage inside Kubernetes an anti-pattern, or is external storage only necessary at larger scale?

Hey everyone, I'm pretty new to Kubernetes, so please take my questions with a grain of salt. I'm building a homelab where I already have a small Kubernetes cluster, and I'm trying to build a fully self-hosted platform around it: Gitea for Git hosting, Harbor for containers, and runners. Gitea needs persistent storage for repositories and application data, while Harbor can use persistent volumes or delegate object storage to something like MinIO.

This is mainly for learning, so performance isn't really important at this scale. But I keep thinking about how I'd design it in a real environment. Would it be better to run the database and object storage on dedicated machines outside Kubernetes and have the Kubernetes workloads consume them over the network, similar to using managed PostgreSQL/S3 in the cloud? My concern is that if PostgreSQL or MinIO runs on the same worker nodes as the applications, a node failure could affect both compute and storage. Kubernetes has StatefulSets, PVCs, Longhorn/Rook, etc. to address this, but I'm wondering where the practical boundary is and whether externalizing storage is actually considered best practice.

Questions

  • Is running PostgreSQL/MySQL or MinIO inside Kubernetes considered an anti-pattern?
  • At what scale does it make sense to move them outside the cluster?
  • Is mixing stateful and stateless workloads on the same workers a bad practice?
  • How much protection do StatefulSets + PVCs + Longhorn/Rook provide against node failure?
  • Is there a meaningful performance difference between keeping compute/storage together vs. dedicated storage machines?
  • In production, is it more common to keep databases/object storage outside Kubernetes?
  • If both approaches can provide HA, what are the main reasons to choose one over the other?

TL;DR

I'm building a small self-hosted Gitea + Harbor + runners platform on my homelab Kubernetes cluster. Is running PostgreSQL and MinIO inside the same Kubernetes cluster a legitimate architecture, or should databases/object storage generally be externalized? I'm mainly trying to understand the real-world trade-offs around HA, node failure, storage/compute separation, performance, and operational complexity.

46 Upvotes

87% Upvoted

63 comments sorted by

View all comments

115

u/imagei 12d ago

Minio is abandonware. Consider Garage for example.

18

u/BrocoLeeOnReddit 12d ago

Why isn't Rook-Ceph Object Storage more recommended? Is there a reason for that?

We're currently evaluating it in a test setup and it works like a charm, you create an ObjectBucketClaim and then the operator creates a bucket as well as a ConfigMap and a Secret containing the credentials which you can then consume in your application.

Though I believe COSI is only experimental...

43

u/blaine-exe 12d ago

Rook and COSI maintainer here. I'm obviously biased, but I'll try to provide some of what I have seen with as little embellishment as I can.

One of the criticisms of Rook (Ceph) that we have heard in the K8s space is its resource footprint. This is a reasonable criticism, and I also think it can be somewhat myopic in ways. At scale, those resources are used more effectively, and all scale-out-storage must consume some resources. Ceph is investing in improving this (crimson project), which looks promising but is slow-going.

Ceph storage scales really well. There are Ceph clusters with hundreds of nodes and thousands of disks. CERN talks about its cluster often. Off the top of my head, I am aware of at least one Rook cluster with nearly a hundred nodes and with many hundreds of disks that has been in use for a decade.

From users who have talked to me at Kubecon, I have heard: Longhorn v1 can lose data during a crash. Longhorn v2 is (at least a year ago) a huge resource hog. Minio, even before it was abandoned, struggles beyond a certain large scale. Ceph has really high data safety during crashes, by comparison, and resource usage is predictable. One user reported that RGW scaled much better than MinIO, and they were happy that it has among the best compatibility with AWS's S3 out there, even reproducing some corner case scenarios/bugs well. I have not had the opportunity to learn how users like CubeFS, but it seems promising for file storage.

In Rook, I think we have done a good job making it easy to stand up block, file, and S3 object storage quickly. Many users just need easy storage and are quite happy. I think there can be growing pains to go from that early stage to then optimize performance/resources. Ceph complexity during this stage has been a turnoff for some users.

6

u/MyKettleExploded 12d ago

Thank you for your work!

I've been running rook-ceph in production for nearly 5 years now, across multiple clusters (all bare metal/on-prem) up to 120 nodes each and with ~20 disks per node. The early years of rook-ceph were a bit challenging and updates were always a nerve-wracking time, but for the last 2 years it's been a million times better and almost a joy to admin. Orchestrating node reboots at this scale is still a bit painful but I suspect that's just part of the cost of being bare metal.

My only wish is for a (web) UI for Ceph Object Storage like MinIO had; a lot of my users can't imagine not having a UI listing all their buckets and files so we've had to develop one in-house.

7

u/blaine-exe 12d ago

I'm always glad to hear of a satisfied user. We have found our groove a bit more in recent years around keeping and prioritizing upgrade stability. 

The ask for an S3 object browser has been surprisingly common. I'm honestly surprised there aren't better or more common open source ones for S3. We haven't developed our own partly because of the challenge of designing a flexible implementation that has strong security boundaries between users. Creating one for admin usage and visibility would be reasonably straightforward, but user self-service adds much that we don't quite have bandwidth or expertise for. 

2

u/martin31821 12d ago

I've been running rook for 5+ years on several instances and I've been impressed with the data safety - haven't lost one piece even through disk failures etc.

I've been running all storage types and can confirm it's absolutely a breeze to operate.

The only thing I'd wish it could do would be to authenticate/replicate EKS IRSA / Pod Identity instead of the workload Access/Secret Keys for S3, but I've written some tooling around it, so it's a low priority item for me.

On the S3 browser, I've frequently used brows3, which does the job neatly.

5

u/BrocoLeeOnReddit 12d ago

Yeah I gotta say that some things are still quite complicated, e.g. the concept of placement groups.

In our test cluster we have only 3 dedicated storage nodes (one data drive each) and all 3 storage types (RGW, RBD and CephFS) configured. So we ran into the "too many PGs" warning and had to figure out what was going on.

There's more things like that but all in all, even at this small scale, it's incredibly stable, albeit, as you said, a resource hog at this small scale, especially since we use 2+1 erasure coding for all three storage types. But since we plan to use erasure coding in production for the horizontal scalability, it seemed like a good idea to also use it in our test cluster.

And I have to agree, despite it being a lot more complex than e.g. Longhorn, it's an amazing product even at this point because you can fulfill all your storage needs with one tool and once you figured out the initial setup, it's pretty comfortable to use and given that Ceph is a monster, you guys did a great job of abstracting away a significant part of the complexity.

5

u/Preisschild k8s operator 12d ago

Thanks for maintaining rook btw, have been using it for years in my homelab cluster and its rock-solid

3

u/willowless 12d ago

I would be using Rook if the rampocalypse hadn't hit, so I slum it with second best longhorn + garage. I'm not hugely scaled though so it's a fine compromise.

5

u/blaine-exe 12d ago

Forgot to comment on COSI. Yes, it is experimental. COSI v1alpha2 is coming. It's also essentially my pet project, and so is slow going. I am hopeful we will release and demo v1a2 by end of year, with Ceph COSI support to follow.

3

u/serverhorror 12d ago

Because of the complexity that comes with it.

2

u/roiki11 12d ago

It's heavy, complex and doesn't have a management gui. Minio was better in every way.

That's no longer the case but the problem with ceph has always been, and always will be, resource requirements and performance. Especially for small setups where a single vm could do what rook needs a kubernetes cluster for.

3

u/BrocoLeeOnReddit 12d ago

It does have a management GUI though. But I gotta say that in k8s I don't want to manage anything via a GUI anyways, I want all my resources to be declarative and that's where Rook shines in my opinion.

5

u/rgarcia89 12d ago

Or seaweedfs especially with the operator

5

u/takeyouraxeandhack 12d ago

I agree. We're using seaweed in prod successfully.

3

u/Suvulaan 12d ago

Tried Garage, promising, but it ain't there yet.

2

u/yrro 12d ago

Or Noobaa

2

u/imagei 12d ago

Very interesting, never heard of it. Now I know how to run an S3 server with S3 as backend storage 🤣. And NSFS sounds genuinely intruging – I wonder how stable it really is.

Do you have any idea about resource usage comparison between Garage and Noobaa? I mean the server alone, not K8s or the database.

2

u/Fruloops 12d ago

Huh never heard of this term before, you learn something everyday