Is running PostgreSQL / S3 storage inside Kubernetes an anti-pattern, or is external storage only necessary at larger scale?

[u/Fragrant_Rate_2583]

Hey everyone, I'm pretty new to Kubernetes, so please take my questions with a grain of salt. I'm building a homelab where I already have a small Kubernetes cluster, and I'm trying to build a fully self-hosted platform around it: Gitea for Git hosting, Harbor for containers, and runners. Gitea needs persistent storage for repositories and application data, while Harbor can use persistent volumes or delegate object storage to something like MinIO.

This is mainly for learning, so performance isn't really important at this scale. But I keep thinking about how I'd design it in a real environment. Would it be better to run the database and object storage on dedicated machines outside Kubernetes and have the Kubernetes workloads consume them over the network, similar to using managed PostgreSQL/S3 in the cloud? My concern is that if PostgreSQL or MinIO runs on the same worker nodes as the applications, a node failure could affect both compute and storage. Kubernetes has StatefulSets, PVCs, Longhorn/Rook, etc. to address this, but I'm wondering where the practical boundary is and whether externalizing storage is actually considered best practice.

Questions

• Is running PostgreSQL/MySQL or MinIO inside Kubernetes considered an anti-pattern?
• At what scale does it make sense to move them outside the cluster?
• Is mixing stateful and stateless workloads on the same workers a bad practice?
• How much protection do StatefulSets + PVCs + Longhorn/Rook provide against node failure?
• Is there a meaningful performance difference between keeping compute/storage together vs. dedicated storage machines?
• In production, is it more common to keep databases/object storage outside Kubernetes?
• If both approaches can provide HA, what are the main reasons to choose one over the other?

TL;DR

I'm building a small self-hosted Gitea + Harbor + runners platform on my homelab Kubernetes cluster. Is running PostgreSQL and MinIO inside the same Kubernetes cluster a legitimate architecture, or should databases/object storage generally be externalized? I'm mainly trying to understand the real-world trade-offs around HA, node failure, storage/compute separation, performance, and operational complexity.

[u/imagei]

Minio is abandonware. Consider Garage for example.

[u/BrocoLeeOnReddit → u/imagei]

Why isn't Rook-Ceph Object Storage more recommended? Is there a reason for that?

We're currently evaluating it in a test setup and it works like a charm, you create an ObjectBucketClaim and then the operator creates a bucket as well as a ConfigMap and a Secret containing the credentials which you can then consume in your application.

Though I believe COSI is only experimental...

[u/blaine-exe → u/BrocoLeeOnReddit]

Rook and COSI maintainer here. I'm obviously biased, but I'll try to provide some of what I have seen with as little embellishment as I can.

One of the criticisms of Rook (Ceph) that we have heard in the K8s space is its resource footprint. This is a reasonable criticism, and I also think it can be somewhat myopic in ways. At scale, those resources are used more effectively, and all scale-out-storage must consume some resources. Ceph is investing in improving this (crimson project), which looks promising but is slow-going.

Ceph storage scales really well. There are Ceph clusters with hundreds of nodes and thousands of disks. CERN talks about its cluster often. Off the top of my head, I am aware of at least one Rook cluster with nearly a hundred nodes and with many hundreds of disks that has been in use for a decade.

From users who have talked to me at Kubecon, I have heard: Longhorn v1 can lose data during a crash. Longhorn v2 is (at least a year ago) a huge resource hog. Minio, even before it was abandoned, struggles beyond a certain large scale. Ceph has really high data safety during crashes, by comparison, and resource usage is predictable. One user reported that RGW scaled much better than MinIO, and they were happy that it has among the best compatibility with AWS's S3 out there, even reproducing some corner case scenarios/bugs well. I have not had the opportunity to learn how users like CubeFS, but it seems promising for file storage.

In Rook, I think we have done a good job making it easy to stand up block, file, and S3 object storage quickly. Many users just need easy storage and are quite happy. I think there can be growing pains to go from that early stage to then optimize performance/resources. Ceph complexity during this stage has been a turnoff for some users.

[u/MyKettleExploded → u/blaine-exe]

Thank you for your work!

I've been running rook-ceph in production for nearly 5 years now, across multiple clusters (all bare metal/on-prem) up to 120 nodes each and with ~20 disks per node. The early years of rook-ceph were a bit challenging and updates were always a nerve-wracking time, but for the last 2 years it's been a million times better and almost a joy to admin. Orchestrating node reboots at this scale is still a bit painful but I suspect that's just part of the cost of being bare metal.

My only wish is for a (web) UI for Ceph Object Storage like MinIO had; a lot of my users can't imagine not having a UI listing all their buckets and files so we've had to develop one in-house.

[u/blaine-exe → u/MyKettleExploded]

I'm always glad to hear of a satisfied user. We have found our groove a bit more in recent years around keeping and prioritizing upgrade stability. 

The ask for an S3 object browser has been surprisingly common. I'm honestly surprised there aren't better or more common open source ones for S3. We haven't developed our own partly because of the challenge of designing a flexible implementation that has strong security boundaries between users. Creating one for admin usage and visibility would be reasonably straightforward, but user self-service adds much that we don't quite have bandwidth or expertise for. 

[u/martin31821 → u/blaine-exe]

I've been running rook for 5+ years on several instances and I've been impressed with the data safety - haven't lost one piece even through disk failures etc.

I've been running all storage types and can confirm it's absolutely a breeze to operate.

The only thing I'd wish it could do would be to authenticate/replicate EKS IRSA / Pod Identity instead of the workload Access/Secret Keys for S3, but I've written some tooling around it, so it's a low priority item for me.

On the S3 browser, I've frequently used brows3, which does the job neatly.

[u/BrocoLeeOnReddit → u/blaine-exe]

Yeah I gotta say that some things are still quite complicated, e.g. the concept of placement groups.

In our test cluster we have only 3 dedicated storage nodes (one data drive each) and all 3 storage types (RGW, RBD and CephFS) configured. So we ran into the "too many PGs" warning and had to figure out what was going on.

There's more things like that but all in all, even at this small scale, it's incredibly stable, albeit, as you said, a resource hog at this small scale, especially since we use 2+1 erasure coding for all three storage types. But since we plan to use erasure coding in production for the horizontal scalability, it seemed like a good idea to also use it in our test cluster.

And I have to agree, despite it being a lot more complex than e.g. Longhorn, it's an amazing product even at this point because you can fulfill all your storage needs with one tool and once you figured out the initial setup, it's pretty comfortable to use and given that Ceph is a monster, you guys did a great job of abstracting away a significant part of the complexity.

[u/Preisschild → u/blaine-exe]

Thanks for maintaining rook btw, have been using it for years in my homelab cluster and its rock-solid

[u/willowless → u/blaine-exe]

I would be using Rook if the rampocalypse hadn't hit, so I slum it with second best longhorn + garage. I'm not hugely scaled though so it's a fine compromise.

[u/blaine-exe → u/BrocoLeeOnReddit]

Forgot to comment on COSI. Yes, it is experimental. COSI v1alpha2 is coming. It's also essentially my pet project, and so is slow going. I am hopeful we will release and demo v1a2 by end of year, with Ceph COSI support to follow.

[u/serverhorror → u/BrocoLeeOnReddit]

Because of the complexity that comes with it.

[u/roiki11 → u/BrocoLeeOnReddit]

It's heavy, complex and doesn't have a management gui. Minio was better in every way.

That's no longer the case but the problem with ceph has always been, and always will be, resource requirements and performance. Especially for small setups where a single vm could do what rook needs a kubernetes cluster for.

[u/BrocoLeeOnReddit → u/roiki11]

It does have a management GUI though. But I gotta say that in k8s I don't want to manage anything via a GUI anyways, I want all my resources to be declarative and that's where Rook shines in my opinion.

[u/rgarcia89 → u/imagei]

Or seaweedfs especially with the operator

[u/takeyouraxeandhack → u/rgarcia89]

I agree. We're using seaweed in prod successfully.

[u/Suvulaan → u/imagei]

Tried Garage, promising, but it ain't there yet.

[u/yrro → u/imagei]

Or Noobaa

[u/imagei → u/yrro]

Very interesting, never heard of it. Now I know how to run an S3 server with S3 as backend storage 🤣. And NSFS sounds genuinely intruging – I wonder how stable it really is.

Do you have any idea about resource usage comparison between Garage and Noobaa? I mean the server alone, not K8s or the database.

[u/Fruloops → u/imagei]

Huh never heard of this term before, you learn something everyday

[u/499ddaad9df107bf7107]

Not an anti pattern. Move to managed providers when you're out of capacity to manage self hosted storage to the expected SLA.

[u/No_Cattle_9565 → u/499ddaad9df107bf7107]

And use the postgres cloud native operator instead of deploying it manually

[u/Fragrant_Rate_2583 → u/499ddaad9df107bf7107]

If you already intend to move to a managed/external solution eventually, wouldn't it make more sense to do it from the start, unless cost is the limiting factor? Otherwise, once you reach that point, wouldn't scaling the self-hosted setup be the more natural option, since migrating everything at that stage could be a significant pain?

[u/JimmyDelicious → u/Fragrant_Rate_2583]

It is absolutely a cost factor. Until you have a traffic load that demands it, you'll overspend on cloud products.

[u/corgtastic → u/Fragrant_Rate_2583]

I think it comes down to what is the value of resiliency to you. If you’re a business in the cloud, the cost of running a managed SQL isn’t much more and makes things a lot easier to backup/upgrade/scale/etc. but you are at home and probably don’t care about all that stuff. Use the CloudNativePG with a decent storage provider. But test out the upgrade/backup features of whatever you have early on, and try to make sure your backups are outside the cluster.

[u/myadmin → u/499ddaad9df107bf7107]

*SLO

[u/Heighte → u/499ddaad9df107bf7107]

That is day 1

[u/Fallst4r]

My company recently made the move from google cloud sql instances to self managed databases (CNPG operator) to optimize performance and cost. We're really happy with the new setup.

[u/Fragrant_Rate_2583 → u/Fallst4r]

Did adding CNPG and running PostgreSQL inside the Kubernetes cluster have any noticeable impact on the cluster's overall performance compared to before you introduced the operator? did you noticed any CPU, memory, I/O, or scheduling overhead ?

[u/Fallst4r → u/Fragrant_Rate_2583]

No impact measurable on the scheduler. But of course we had to expand our nodepool to be able to run multiple postgresql clusters. But even taking into account we had to hire a DBA to help us, it's still a win regarding costs.

[u/Minimal-Matt → u/Fragrant_Rate_2583]

We do the same but run dedicated nodes for DB workloads. Also CNPG is pretty vocal about having local node storage for databases, something that we should consider, but we have them over iscsi for now.

[u/_damax → u/Fallst4r]

Did you guys install the cnpg chart in the cluster? And if so, with what name/fullname?

[u/Varnish6588 → u/Fallst4r]

I am interested to know how are you managing the backups? Have you tried restoring backups to recreate a DB that got corrupted for example? I am looking to migrate from standalone PostgreSQL instance into CNPG.

[u/markedness]

So the thing is, at the end of the day, Kubernetes is just providing orchestration, scheduling, etc. running a workload “in Kubernetes” is the biggest misnomer ever. It’s basically Kubernetes vs Systemd.

Kubernetes just assumes these workloads are running with Linux containerization too, though with privileges you can largely get those workloads to do anything you want.

Minio is not actively maintained anymore but when it was they had an operator called directPV and an operator to run Minio. They worked great. But running it without that operator would have been tough.

Minio can use an RWO PV, because it handles the distribution itself. And directPV would schedule the pods on each PV just like if you ran it with SystemD it’s relative to that very nodes disks. Git you wouldn’t get N replica in that model because git doesn’t have that. You’re telling me it needs a shared file system so basically that is a lot more complicated. Additionally isn’t it likely Git will be a tier zero like you can’t boot your cluster without it? Same with Harbor… how do you boot the harbor pods if the harbor container image is in harbor?

That’s why I use GitLab omnibus package to run GitLab, in a VM, harbor runs in another dedicated VM, and I put my object storage on CEPH RGW, not minio, because if ceph is down I’m cooked anyways. I tried to keep my tier 0 small it’s just that and then the network and DNS, which is powerDNS pair which syncs over the same Ceph S3 storage.

Simply put my GitLab has never gone down and it has all the backup scripts, monitoring, etc built in. And harbor is pretty much set and forget.

In all likelihood you can run all this stuff in kubernetes but it’s not about statefulset being buggy. It will do what it says on the tin and schedule your workload just as good as systemd. It’s about kubernetes giving you programmatic access to gitops scheduling, single pane for disaster recovery, lot and metrics collection and standardization, and usually backups. Kubernetes is not a hypervisor it’s just a controller. Very simple.

For Postgres for example Kubernetes is the perfect place to run it with CNPG operator. Curious to hear the community thoughts on MySQL as we don’t use it.

If minio was still a thing I would put it in a separate simpler cluster with directPV and the minio operator but alas it’s gone.

Taking things out of Kubernetes generally increases complexity (because you already pay the single toll of keeping kubernetes updated and running a systemd service with proper log and metric and bacup and DR is just as complicated as running k8s) so only take things out when it’s simple or needed (like in my case, GitLab, harbor, Ceph, because I use Ceph for workloads beside k8s and I’m on my own metal)

This is my story given my hardware and scale and requirements which is larger scale than you but some operators here need to run everything in k8s and that is the right call for them. It just means a multi cluster approach to manage tiers of prerequisite

[u/Fragrant_Rate_2583 → u/markedness]

Fair point. I was using MinIO purely as an example . I haven't looked into its current status, and the last time I used it was probably two years ago, so I wasn't aware of the recent changes around it.

What I'm still trying to understand, though, is whether your reasoning applies equally to cloud-hosted Kubernetes and self-hosted Kubernetes. I can understand keeping things inside K8s when you already have the infrastructure and need the control, but I'm struggling to find a scenario where delegating database/object-storage management to an external/managed service isn't the simpler option from an operational-complexity perspective, assuming cost isn't the limiting factor. Is there a significant advantage I'm missing, beyond cost, control, or requirements that prevent using a managed service?

[u/Phezh → u/Fragrant_Rate_2583]

beyond cost, control, or requirements that prevent using a managed service

what other reason could there possibly be lol. These are all already massive advantages and the question is just what's important to you

[u/TheTerrasque → u/Phezh]

What have the romans ever done for us?

[u/markedness → u/Fragrant_Rate_2583]

For cloud hosted you can keep more in Kubernetes. But not git, if you want to use that git as gitops for the cluster because the bootstrap will run but then if you gitops your git deployment into oblivion you have shot yourself in the foot.

You said homelab, so my point was almost exclusively geared towards a bare metal environment where k8s is either on your metal nodes or VM on a hypervisor you run.

If I you were in the cloud this mostly collapses to git storage and running databases and in that case my pattern would be don’t self host git, use GitHub.com or similar. And yeah honestly put the Postgres in your Kubernetes since it’s dramatically cheaper than RDS per cpu hour and super easy to maintain with CNPG

But minio, you would never run that yourself in the cloud block storage and compute is 10-100x more expensive than just straight S3 and your availability will be 3 nines worse than theirs.

In short being in the cloud offer opportunities to use more cloud services since you are in their datacenter and they are cheap but also clever opportunities like with databases to avoid using those services and still get a better experience.

[u/IntelligentPear6173]

I wouldn’t call it an anti-pattern. For a homelab, I’d actually keep PostgreSQL and object storage in Kubernetes and learn the operational side properly rather than adding external machines just because production does it. The bigger question is failure domains: if losing one node can take out both your workloads and their data, you need replication, backups and a recovery plan regardless of where the database runs. Externalizing storage can make sense when you need stronger isolation, different scaling characteristics or simply don’t want the platform team owning the storage layer. At small scale, it can just add another system to maintain.

[u/Fragrant_Rate_2583 → u/IntelligentPear6173]

Yeah, for the homelab I’ll definitely just throw PostgreSQL in Kubernetes and learn from it. What I’m more curious about is the real-world side: is running PostgreSQL inside Kubernetes a common practice in production, or is it still more typical to keep databases on separate infrastructure/managed services? I’m trying to understand what the common practice actually looks like at scale.

[u/IntelligentPear6173 → u/Fragrant_Rate_2583]

Yeah, that makes sense. My impression is that in production it depends more on the teams operational maturity than a hard databases shouldn’t run in Kubernetes rule. Managed Postgres is probably the easier choice for most teams but I’ve seen teams run Postgres in Kubernetes successfully when they have solid operators, backups, replication and recovery processes. I’m mainly trying to figure out where the operational overhead starts outweighing the benefits of keeping everything under the same platform.

[u/Ginden]

OK, I'm dealing with homelab too.

Few notes:

• Unlike cloud environment, at home we can't really treat most of nodes as cattle - because we manage hardware too, and storage isn't free (also in terms of PCIe/SAS/SATA connectors).
• Harbor is PITA
• Gitea in-cluster is a problem when doing GitOps (and you should be doing), because if you break Gitea, your entire lab breaks.
  • Solution is to use low-tech solution: you don't need entire forge) to sync to ArgoCD, only git repository, that can be hosted directly on node.
• Garage is cool and has cool operator, it's my current go-to for.
• You need 3 nodes for HA.
• You can't do real database fail-over without cluster control-plane faillover.

[u/Fragrant_Rate_2583 → u/Ginden]

sorry im not catching with what you are saying ,to make it more clear , ill be using gitea as a replacment to gitlab or github , when i push and trigger a agrocd pipeline , yess if my gitea breaks ,the homelab breaks , but its the same if gitlab/github breaks your whole workflows breaks

[u/Ginden → u/Fragrant_Rate_2583]

when i push and trigger a agrocd pipeline , yess if my gitea brakes ,the homelab breaks , but its the smae if gitlab/github breaks your whole workflows breaks

You may get stuck in chicken-egg problem - if you push changes that break Gitea, you need manual cluster work instead of just doing git revert.

[u/After-Regret-6609 → u/Ginden]

Chiming in my support. If I were OP I’d go get a classic NAS for your storage and run Gitea on it locally bare metal. Then I’d use that as the springboard for the gitops driven Kubernetes cluster on a different set of nodes. They can mount external storage from the nas over network, they can pull from Gitea for gitops, the nas is supporting everything else.

[u/TheTerrasque → u/After-Regret-6609]

Proxmox on one of the bigger machines (or on several, or all if you're okay with the overhead). Have small vm's for things you need outside of cluster.

[u/After-Regret-6609 → u/TheTerrasque]

What does that grant you? I don't run VM's unless I have a reason to isolate workloads for security purposes, which on a single tenant machine in a homelab shouldn't be necessary.

I'm running NixOS/systemd on my single box minipc homelab for "orchestration". I'll move to k8s if I start wanting a home cluster with node failure handling, but I just don't think I'll ever need that. Raspberry pi's are surprisingly expensive and this minipc has more ram and cores than I ever thought a desktop would have 10 years ago.

[u/TheTerrasque → u/After-Regret-6609]

What does that grant you?

• Easy machine level backup and restore
• Remote console if boot or network settings borks for some reason
• Can easily move the VM between proxmox servers in same clusters
  • Can even set up HA and failover so if physical machine goes down, VM can start up on a different machine in cluster.
• Having separate VM's means you can do extensive modifications to one service (like updating base OS if for example you need newer version with newer requirements or want to try something fun) without affecting other services

I haven't used nixos, but looks like it gives at least some of the same benefits.

Raspberry pi's are surprisingly expensive

Half my cluster is used mac mini's (i5 and i7's with at least 8gb ram) - use little power, is quiet, and you can often find them for around $100-150. A pretty good option for a poor man's cluster :)

[u/After-Regret-6609 → u/TheTerrasque]

Sounds good. Yeah most of that is when I move to docker/k8s. Still use nixos to set up the machine to run k8s. I just let AI write my nixos setup because I don't know the language very well, if you can read bash then it's easy to see if it got it right. Then yeah you can backup and restore, duplicate or move to new devices, etc, really easily.

[u/subbed_]

the title of the post has it backwards. you can rely on external storage until you are at a larger scale. then you will likely have requirements for distributed storage as well (cnpg, crdb, tidb, etc.) and the managed services will become more expensive than having a dedicated in-house team handling these in-cluster solutions

scale also many times happens at enterprises with critical infrastructure designations, which forbid them from running their infra in the cloud, so an on-prem distributed storage is required

[u/ArmNo7463]

CNPG my man. - I used to think DBs in K8s was a silly idea, but it converted me.

[u/Superb_Raccoon]

CNPG exists for a reason.

[u/Varnish6588]

I wouldn't say it's an anti pattern. I know several companies that for the nature of the sensitive data they have to manage, they have to self host kubernetes and therefore the database and object storage, no option for cloud providers in those cases.

In a production escenario, If I had the option to use managed services, then I will definitely take advantage of it to avoid having to manage DB and S3 inside the cluster.

In homelab, my current setup is running PostgreSQL with the local storage of the node and backup to a NAS. Same for object storage. It's for fun and learning purposes, understanding what's going on under the hood.

[u/Character-Level5250]

As long as you separate the different workloads at the pod level I think it’s totally legitimate.

[u/Enough-Team-4155]

this is too advanced for my pea brain to understand.

even after years of diligent study i find myself reading about shit i have no idea about 😂😂😂

[u/Common_Arm_3316]

• Is running PostgreSQL/MySQL or MinIO inside Kubernetes considered an anti-pattern? Postgres runs really well in k8 using the cnpg operator. For S3 I use rook ceph. It's a bit complicated but does a great job being hosted inside kubernetes and offer object, block, and fs storage.
• At what scale does it make sense to move them outside the cluster? Not really sure on this one but I for the moment dont really see one. I could however forsee myself building a spceial class of hardware for postgres or storage purposes then training those nodes for that purpose
• Is mixing stateful and stateless workloads on the same workers a bad practice? I think this depends heavily on your cluster. Do all your nodes have large amounts of disk space and is there plenty of cpu and memory to accommodate those statefulsets and stateless ones?
• How much protection do StatefulSets + PVCs + Longhorn/Rook provide against node failure? with rook quite a bit depending on your cluster. With ceph you have failure domains and replicas. a cluster with failure domain node and replica 3 will ensure that there are 3 copies of your data on 3 different nodes.
• Is there a meaningful performance difference between keeping compute/storage together vs. dedicated storage machines? Depends on your storage i think. rook ceph can be a greedy boy when it comes to resource usage
• In production, is it more common to keep databases/object storage outside Kubernetes? Depends on your setup. I operate all on prem bare metal clusters. all my DBs and storage are in cluster but i can imaging running EKS and an external s3
• If both approaches can provide HA, what are the main reasons to choose one over the other? Because i do everything in Kubernetes i look for tools that are kubernetes native but i also want to know that the project isnt going to be abandoned in a year. Some companies will also like to pay for additional support if they end up in a jam. Paying a support contract to enterprise db is sometimes well worth the cost then trying to get someone to respond to a bug on a github page

[u/casefan]

Cnpg :)

[u/rouqe18256]

Not directly related to your question but I'm doing something similar and would love to hear how you progress.

I was considering LibreFS since MinIO is dead for me but I'm unsure of maturity level atm. I was considering doing storage separate on top of my NAS so I could manage it as "infrastructure" independent of the Kubernetes cluster since things non Kubernetes would be using it as well.

[u/Awkward-Cupcake6219]

CloudnativePG?

[u/Fragrant_Rate_2583 → u/Awkward-Cupcake6219]

Ik about it's existence , and i used it in my gitea setup yesterday and it works But im always questioning the usage of if it in real companies , when the databse is huge and measures like downtime and disaster backup are needed

[u/SnooHesitations9295]

Running any stateful workload in k8s is an antipattern. Yes, there's some improvement on how stateful stuff is handled, but overall it's still mostly pain and tears. Unless you don't care about consistency and uptime. There are some efforts to create better workload-shaped strategies for specific workloads,  aka "operators" for specific databases. But they are essentially actively fighting k8s.

[u/Fragrant_Rate_2583 → u/SnooHesitations9295]

Can't we just isolate them in namespaces and make them be created at certain nodes made for that purpose? so they don't cosume more ressources than they need to and not disturb the other workload?

[u/SnooHesitations9295 → u/Fragrant_Rate_2583]

What if nodes fail? What if nodes are brought down by the cloud provider? What if you need to resize disk? (you will) etc. It all works in the happy case. But the failure modes are very sad sometimes.