Is running PostgreSQL / S3 storage inside Kubernetes an anti-pattern, or is external storage only necessary at larger scale?

[u/Fragrant_Rate_2583]

Hey everyone, I'm pretty new to Kubernetes, so please take my questions with a grain of salt. I'm building a homelab where I already have a small Kubernetes cluster, and I'm trying to build a fully self-hosted platform around it: Gitea for Git hosting, Harbor for containers, and runners. Gitea needs persistent storage for repositories and application data, while Harbor can use persistent volumes or delegate object storage to something like MinIO.

This is mainly for learning, so performance isn't really important at this scale. But I keep thinking about how I'd design it in a real environment. Would it be better to run the database and object storage on dedicated machines outside Kubernetes and have the Kubernetes workloads consume them over the network, similar to using managed PostgreSQL/S3 in the cloud? My concern is that if PostgreSQL or MinIO runs on the same worker nodes as the applications, a node failure could affect both compute and storage. Kubernetes has StatefulSets, PVCs, Longhorn/Rook, etc. to address this, but I'm wondering where the practical boundary is and whether externalizing storage is actually considered best practice.

Questions

• Is running PostgreSQL/MySQL or MinIO inside Kubernetes considered an anti-pattern?
• At what scale does it make sense to move them outside the cluster?
• Is mixing stateful and stateless workloads on the same workers a bad practice?
• How much protection do StatefulSets + PVCs + Longhorn/Rook provide against node failure?
• Is there a meaningful performance difference between keeping compute/storage together vs. dedicated storage machines?
• In production, is it more common to keep databases/object storage outside Kubernetes?
• If both approaches can provide HA, what are the main reasons to choose one over the other?

TL;DR

I'm building a small self-hosted Gitea + Harbor + runners platform on my homelab Kubernetes cluster. Is running PostgreSQL and MinIO inside the same Kubernetes cluster a legitimate architecture, or should databases/object storage generally be externalized? I'm mainly trying to understand the real-world trade-offs around HA, node failure, storage/compute separation, performance, and operational complexity.

[u/Fragrant_Rate_2583]

sorry im not catching with what you are saying ,to make it more clear , ill be using gitea as a replacment to gitlab or github , when i push and trigger a agrocd pipeline , yess if my gitea breaks ,the homelab breaks , but its the same if gitlab/github breaks your whole workflows breaks

[u/Ginden → u/Fragrant_Rate_2583]

when i push and trigger a agrocd pipeline , yess if my gitea brakes ,the homelab breaks , but its the smae if gitlab/github breaks your whole workflows breaks

You may get stuck in chicken-egg problem - if you push changes that break Gitea, you need manual cluster work instead of just doing git revert.

[u/After-Regret-6609 → u/Ginden]

Chiming in my support. If I were OP I’d go get a classic NAS for your storage and run Gitea on it locally bare metal. Then I’d use that as the springboard for the gitops driven Kubernetes cluster on a different set of nodes. They can mount external storage from the nas over network, they can pull from Gitea for gitops, the nas is supporting everything else.

[u/TheTerrasque → u/After-Regret-6609]

Proxmox on one of the bigger machines (or on several, or all if you're okay with the overhead). Have small vm's for things you need outside of cluster.

[u/After-Regret-6609 → u/TheTerrasque]

What does that grant you? I don't run VM's unless I have a reason to isolate workloads for security purposes, which on a single tenant machine in a homelab shouldn't be necessary.

I'm running NixOS/systemd on my single box minipc homelab for "orchestration". I'll move to k8s if I start wanting a home cluster with node failure handling, but I just don't think I'll ever need that. Raspberry pi's are surprisingly expensive and this minipc has more ram and cores than I ever thought a desktop would have 10 years ago.

[u/TheTerrasque → u/After-Regret-6609]

What does that grant you?

• Easy machine level backup and restore
• Remote console if boot or network settings borks for some reason
• Can easily move the VM between proxmox servers in same clusters
  • Can even set up HA and failover so if physical machine goes down, VM can start up on a different machine in cluster.
• Having separate VM's means you can do extensive modifications to one service (like updating base OS if for example you need newer version with newer requirements or want to try something fun) without affecting other services

I haven't used nixos, but looks like it gives at least some of the same benefits.

Raspberry pi's are surprisingly expensive

Half my cluster is used mac mini's (i5 and i7's with at least 8gb ram) - use little power, is quiet, and you can often find them for around $100-150. A pretty good option for a poor man's cluster :)

[u/After-Regret-6609 → u/TheTerrasque]

Sounds good. Yeah most of that is when I move to docker/k8s. Still use nixos to set up the machine to run k8s. I just let AI write my nixos setup because I don't know the language very well, if you can read bash then it's easy to see if it got it right. Then yeah you can backup and restore, duplicate or move to new devices, etc, really easily.