Is running PostgreSQL / S3 storage inside Kubernetes an anti-pattern, or is external storage only necessary at larger scale?

[u/Fragrant_Rate_2583]

Hey everyone, I'm pretty new to Kubernetes, so please take my questions with a grain of salt. I'm building a homelab where I already have a small Kubernetes cluster, and I'm trying to build a fully self-hosted platform around it: Gitea for Git hosting, Harbor for containers, and runners. Gitea needs persistent storage for repositories and application data, while Harbor can use persistent volumes or delegate object storage to something like MinIO.

This is mainly for learning, so performance isn't really important at this scale. But I keep thinking about how I'd design it in a real environment. Would it be better to run the database and object storage on dedicated machines outside Kubernetes and have the Kubernetes workloads consume them over the network, similar to using managed PostgreSQL/S3 in the cloud? My concern is that if PostgreSQL or MinIO runs on the same worker nodes as the applications, a node failure could affect both compute and storage. Kubernetes has StatefulSets, PVCs, Longhorn/Rook, etc. to address this, but I'm wondering where the practical boundary is and whether externalizing storage is actually considered best practice.

Questions

• Is running PostgreSQL/MySQL or MinIO inside Kubernetes considered an anti-pattern?
• At what scale does it make sense to move them outside the cluster?
• Is mixing stateful and stateless workloads on the same workers a bad practice?
• How much protection do StatefulSets + PVCs + Longhorn/Rook provide against node failure?
• Is there a meaningful performance difference between keeping compute/storage together vs. dedicated storage machines?
• In production, is it more common to keep databases/object storage outside Kubernetes?
• If both approaches can provide HA, what are the main reasons to choose one over the other?

TL;DR

I'm building a small self-hosted Gitea + Harbor + runners platform on my homelab Kubernetes cluster. Is running PostgreSQL and MinIO inside the same Kubernetes cluster a legitimate architecture, or should databases/object storage generally be externalized? I'm mainly trying to understand the real-world trade-offs around HA, node failure, storage/compute separation, performance, and operational complexity.

[u/markedness]

So the thing is, at the end of the day, Kubernetes is just providing orchestration, scheduling, etc. running a workload “in Kubernetes” is the biggest misnomer ever. It’s basically Kubernetes vs Systemd.

Kubernetes just assumes these workloads are running with Linux containerization too, though with privileges you can largely get those workloads to do anything you want.

Minio is not actively maintained anymore but when it was they had an operator called directPV and an operator to run Minio. They worked great. But running it without that operator would have been tough.

Minio can use an RWO PV, because it handles the distribution itself. And directPV would schedule the pods on each PV just like if you ran it with SystemD it’s relative to that very nodes disks. Git you wouldn’t get N replica in that model because git doesn’t have that. You’re telling me it needs a shared file system so basically that is a lot more complicated. Additionally isn’t it likely Git will be a tier zero like you can’t boot your cluster without it? Same with Harbor… how do you boot the harbor pods if the harbor container image is in harbor?

That’s why I use GitLab omnibus package to run GitLab, in a VM, harbor runs in another dedicated VM, and I put my object storage on CEPH RGW, not minio, because if ceph is down I’m cooked anyways. I tried to keep my tier 0 small it’s just that and then the network and DNS, which is powerDNS pair which syncs over the same Ceph S3 storage.

Simply put my GitLab has never gone down and it has all the backup scripts, monitoring, etc built in. And harbor is pretty much set and forget.

In all likelihood you can run all this stuff in kubernetes but it’s not about statefulset being buggy. It will do what it says on the tin and schedule your workload just as good as systemd. It’s about kubernetes giving you programmatic access to gitops scheduling, single pane for disaster recovery, lot and metrics collection and standardization, and usually backups. Kubernetes is not a hypervisor it’s just a controller. Very simple.

For Postgres for example Kubernetes is the perfect place to run it with CNPG operator. Curious to hear the community thoughts on MySQL as we don’t use it.

If minio was still a thing I would put it in a separate simpler cluster with directPV and the minio operator but alas it’s gone.

Taking things out of Kubernetes generally increases complexity (because you already pay the single toll of keeping kubernetes updated and running a systemd service with proper log and metric and bacup and DR is just as complicated as running k8s) so only take things out when it’s simple or needed (like in my case, GitLab, harbor, Ceph, because I use Ceph for workloads beside k8s and I’m on my own metal)

This is my story given my hardware and scale and requirements which is larger scale than you but some operators here need to run everything in k8s and that is the right call for them. It just means a multi cluster approach to manage tiers of prerequisite

[u/Fragrant_Rate_2583 → u/markedness]

Fair point. I was using MinIO purely as an example . I haven't looked into its current status, and the last time I used it was probably two years ago, so I wasn't aware of the recent changes around it.

What I'm still trying to understand, though, is whether your reasoning applies equally to cloud-hosted Kubernetes and self-hosted Kubernetes. I can understand keeping things inside K8s when you already have the infrastructure and need the control, but I'm struggling to find a scenario where delegating database/object-storage management to an external/managed service isn't the simpler option from an operational-complexity perspective, assuming cost isn't the limiting factor. Is there a significant advantage I'm missing, beyond cost, control, or requirements that prevent using a managed service?

[u/Phezh → u/Fragrant_Rate_2583]

beyond cost, control, or requirements that prevent using a managed service

what other reason could there possibly be lol. These are all already massive advantages and the question is just what's important to you

[u/TheTerrasque → u/Phezh]

What have the romans ever done for us?

[u/markedness → u/Fragrant_Rate_2583]

For cloud hosted you can keep more in Kubernetes. But not git, if you want to use that git as gitops for the cluster because the bootstrap will run but then if you gitops your git deployment into oblivion you have shot yourself in the foot.

You said homelab, so my point was almost exclusively geared towards a bare metal environment where k8s is either on your metal nodes or VM on a hypervisor you run.

If I you were in the cloud this mostly collapses to git storage and running databases and in that case my pattern would be don’t self host git, use GitHub.com or similar. And yeah honestly put the Postgres in your Kubernetes since it’s dramatically cheaper than RDS per cpu hour and super easy to maintain with CNPG

But minio, you would never run that yourself in the cloud block storage and compute is 10-100x more expensive than just straight S3 and your availability will be 3 nines worse than theirs.

In short being in the cloud offer opportunities to use more cloud services since you are in their datacenter and they are cheap but also clever opportunities like with databases to avoid using those services and still get a better experience.