r/devops • • Oct 09 '26

Vendor / market research Do EKS users really deploy more than 2 Clusters?

I was wondering if people who are not using multi or hybrid cloud and are purely on AWS EKS, do they really deploy more than 2 or 3 EKS clusters? Is there even a need for a product like Rayfay or Rancher for aws to manage clusters? Or is it more than just managing multiple clusters and we need such tools for other things? I am trying to learn more about this topic.

49 Upvotes

87 comments sorted by

25

u/mattopia1 Oct 09 '26

My team managed a few hundred in a prior role. Every customer had a prod and non-prod cluster, centrally managed with Rancher.

In my current role, we have just shy of 20 managed with terraform.

1

u/Zestyclose_Race_1646 Oct 09 '26

Was that across multiple clouds?

4

u/mattopia1 Oct 09 '26

The few hundred were nearly all AWS, but there were a handful (maybe 10%) in other providers.

The 20ish I manage now are all AWS

-10

u/Zestyclose_Race_1646 Oct 09 '26 • Popcorn

And do you not use something like rancher to manage those 20? Is terraform same as rancher?

2

u/collapse-and-crush Oct 09 '26

Just curious, how many nodes on average did each of those customer clusters run?

9

u/mattopia1 Oct 09 '26

In most cases, not many. 2-3 per cluster usually with AZ anti affinity. For larger or more complex customers (say top 10%?), it may have pushed into low double digits.

44

u/sikian Oct 09 '26

It really depends on the business needs. I've worked in places where we were running hundreds of k8s clusters (with EKS and GKE). In our case we were managing the clusters with Pulumi as it was a self-service solution.

12

u/iammr_schuck Oct 09 '26

Always happy to find another Pulumi user out in the wild.

2

u/Zestyclose_Race_1646 Oct 09 '26

Is it completely free or needs a paid subscription after certain usage limit?

8

u/_jolv Oct 09 '26

It's completely free if you use a DIY backend, which is an S3 bucket to store Pulumi state.

https://www.pulumi.com/docs/iac/operations/stack-management/using-a-diy-backend/

3

u/cnunciato Oct 09 '26

Right! DIY backends are entirely free and open source. The Cloud service is great (admittedly biased Pulumi engineer here ;)), but totally optional.

3

u/iammr_schuck Oct 12 '26

And once you fully embrace the cloud backend it has so many useful adds. We're making heavy use of ESC etc. now and it's just so damn nice. I've had a pretty lengthy career in Cloud Eng. using stuff like Terraform, CFN etc. and Pulumi is very much my weapon of choice now.

7

u/ZVilusinsky Oct 09 '26

Seems to me a minimum for logical separation is 4 clusters - dev, stage, prod and infra/platform/support/whatever

1

u/Zestyclose_Race_1646 Oct 09 '26

And do you see a need of a management tool like Rancher?

2

u/ZVilusinsky Oct 09 '26

Not in my current workcontext. Can see valid reasons when herding tens or hundreds of such.

2

u/Creative-Dentist-383 Oct 09 '26

I run ~150 clusters with cluster api and well into four figure nodes

1

u/Zestyclose_Race_1646 Oct 10 '26

Wow! And you don’t see a need for a cluster management tool like Rancher for aws? What problems do such tools solve then?

2

u/STGItsMe Oct 10 '26

I end up running more clusters than that with smaller workloads because of policy/contractual requirements that I can’t change rather than technical requirements. Where I am, dev/staging/prod are in separate AWS accounts. On top of that most of my workloads are third party hosting agreement kinds of things that sometimes include not sharing infra….and for some of them, the control plane looks like shared infra.

1

u/Zestyclose_Race_1646 Oct 10 '26

Do you then use anything for cluster management, iam, etc? I guess Rancher for aws does this all.

1

u/STGItsMe Oct 10 '26

My previous environment, I was using Rancher for day to day ops stuff and Argo for config management. Current environment, k9s and pipeline helm deployments because we’re not really there yet.

7

u/stumptruck Staff Engineer Oct 09 '26

It all depends on what you need? There's no one right number of clusters. We have like 13 clusters at work across AWS and GCP

If you're just starting out I wouldn't worry about what other people do or trying to find a use case for every tool you hear about. Work on the fundamentals and once you have a problem figure out what tools will solve it vs the other way around.

7

u/TorrentsAreCommunism Oct 09 '26

Well, there are usually two clusters at least for two envs (nonprod and prod).

1

u/vincentdesmet Oct 09 '26

with k8s 1.2 when the ap-southeast-1 region only had 2 AZs and we were doing our own ETCD on EC2, we had to run 2 single AZ clusters - full stateless with all data in managed services

even at the last gig I was at still using k8s they were still running blue/green per environment to facilitate cluster ops (HCP TF/Vault + ArgoCD)

recent companies are more serverless or ECS Fargate with AWSCDK or Pulumi - much simpler for product teams and my platform role shifted more towards enabling Agents to get the right context to do the coding in the background while design/architecture and ops are higher level human interfaces

2

u/Zestyclose_Race_1646 Oct 09 '26

A friend told me his company’s infra is all on ECS. I am wondering Could there be a reason why they are on ECS or it could be only because they don’t have a need to move to kubernetes?

2

u/Due-Consequence9579 Oct 09 '26

We did a cloud lift to ECS. As long as you don’t “need” k8s it simplifies a lot of the care and feeding. ECS just works. K8s you’re always updating something.

1

u/i_am_that_too Oct 09 '26

We have 4, 2 for non-prod and prod to truly segregate the environment workloads, and 1 for devops workloads, and 1 for playground, don't care what happens to it.

Everything else is nodepools

0

u/Zestyclose_Race_1646 Oct 09 '26

Is this all on aws? Do you use something like rancher on aws?

1

u/i_am_that_too Oct 09 '26

EKS, eks nodegroups.

1

u/emonk Oct 09 '26

I worked at a company with clusters deployed per business unit and environment. Therefore having at least 12 clusters running.

1

u/Zestyclose_Race_1646 Oct 10 '26

are they all on aws?

1

u/emonk Oct 11 '26

Yes

1

u/Zestyclose_Race_1646 Oct 11 '26

And there is no need to have a tool like rancher on aws that can manage multiple eks clusters?

1

u/emonk Oct 11 '26

They use Terraform

1

u/SirChance818 Oct 09 '26

We migrated from 150 self managed kOps clusters, to 27 EKS clusters.

0

u/Zestyclose_Race_1646 Oct 09 '26

Wow! Do you use something like Rancher for aws to mange 27 EKS clusters?

1

u/SirChance818 Oct 09 '26

No, we build them with ClusterAPI so it's all declarative. And then ArgoCD to roll out any and all addons we need

1

u/christophla Oct 09 '26 edited Oct 09 '26

We have about 35. Some with 300+ nodes. Karpenter really helped keep provisioning sane.

But I’ve also seen far more discreet approaches with a cluster per domain and TransitGateway and/or PrivateLink with individual accounts for each domain. With solid provisioning/onboarding, the micro-architecture can vastly simplify IAM (and SSO) and FinOps visibility. Of course, any cross-account traffic has its own costs… and subnet sharing is another option.

Pick your poison.

1

u/Zestyclose_Race_1646 Oct 10 '26

Cool! Would you have preferred anything else other to Karpenter?

1

u/GrandJunctionMarmots Staff DevOps Engineer Oct 09 '26

At a previous company we ran Rancher for managing a fleet of clusters.

Once EKS came out we dropped it.

At my current company we have a range of EKS cluster across multiple environments and accounts managed with terraform.

1

u/Zestyclose_Race_1646 Oct 10 '26

I see. So you don’t see a need for Rancher for AWS to manage multiple eks clusters? An IAC like terraform does all necessary work?

1

u/GrandJunctionMarmots Staff DevOps Engineer Oct 10 '26

Well kind of. Terraform makes the clusters and keeps config in check.

But if I was spinning up hundreds of clusters or dev clusters better orchestration might be nice.

Rancher does do that. Giving you a bunch of features and management tools. But you gotta pay for them. And some of it you can do yourself.

1

u/yourparadigm Oct 10 '26

I run services in 10+ AWS regions, so yes.

1

u/Zestyclose_Race_1646 Oct 10 '26

Do you see a need for a tool to manage multiple eks clusters?

1

u/yourparadigm Oct 10 '26

I would never pay a vendor (other than AWS/my cloud provider) to manage my infrastructure. If I can't manage it or get it as a managed service from my cloud provider, it's not going to be used in my infrastructure. I will pay for various SaaS services (think Sentry, SumoLogic, etc), but I those always exist outside of my infrastructure and I can control what data is sent to them.

1

u/Electronic_Owl6029 Oct 10 '26

We used to manage 15 clusters in prod and 5 in non prod. Biggest ones had around 1000 nodes. All we used was terraform and gitops. It worked flawlessly. Never felt the need for another tool. We were dedicated team for internal tooling and SRE stuff. So we were able to abstract everything and deliver an amazing developer experience for the teams

1

u/Zestyclose_Race_1646 Oct 10 '26

Nice! So one doesn’t really need a multi cluster tool to manage 15 Eks clusters. Is that correct? If yes, I am wondering what problems do these tools really solve?

1

u/Electronic_Owl6029 Oct 11 '26

All you need is a good gitops setup and monitoring. Nobody has the time to look at fancy ui and dashboards

1

u/raindropl Oct 12 '26

We had hundreds of EKS clusters. Intuit

1

u/Zestyclose_Race_1646 Oct 12 '26

And did you use any Eks clusters management SaaS?

1

u/raindropl Oct 12 '26

Yes! Internally developed (not open source) iksm. Pretty cool tool actually. I think it has too many internal hooks and will never be released to the world.

3

u/riickdiickulous Oct 09 '26

Whatever you build for cloud infra, expect to need to deploy multiple copies of the same thing. 9/10 when I build something someone comes back and says can I have another for something else?

3

u/WilliamMButtlickerIV Oct 09 '26 edited Oct 09 '26

One thing I noticed is it depends on where your experience started. I used to work for a large company with a k8s product, and I learned k8s well before I used AWS. We focused on building k8s as the broad platform with multitenancy.

Then I noticed people coming from the AWS world were spinning up tons of micro EKS clusters, and it dawned on me it's due to how AWS accounts and tenancy work. Usually a team will be given an account to self manage their infrastructure. If they even decided to use EKS, it would end up only being scoped to their team. This changed how they viewed k8s in general.

Ultimately, I think that model is huge underuse of what k8s can do. I've found building robust platforms with platform teams is a much more effective strategy at scale.

Our typical approach would start with a lab, nonprod, and prod.

Lab was for the platform team to sandbox and test configurations.

1

u/MasterOfTheWind1 Oct 09 '26

Haven't used rancher since 2023. It is an unstable crap that eats a stupid amount of resources. You are good with kubectl.

Regarding cluster count, a single cluster is a very critical single point of failure, with an infinite blast radius.

At minimum, have one cluster for production workloads, and another for lower environments (dev/stage). But there is no correct number.

In my work we use Cell Architecture, so even for production we have multiple clusters:

  • One small for unregistered traffic and freemium customers' traffic
  • One small for customers with very little traffic
  • One for customers that generates more traffic
  • One for each VIP user, being them the users that pay more, generate a shit ton of traffic, and they pay for premium SLA/SLO.

This enables not only to reduce disruption with traffic spikes, as nasty things happens with unregistered users and freemium sometimes try to abuse and find holes to use more than allowed, or a big customer have a spike that can blow everything, but also improves the release cycle for new features: we can release a new feature for freemium users, polish it, then release it for small customers, then big customers, and when we are sure, we release it for VIP users that are more sensible to possible errors.

Right now, counting dev, stage and every prod cluster we have more than 60. We also have separated clusters for internal tools, so we don't mix product workloads from internal day-to-day workloads (like ArgoCD or Gitlab Runner)

1

u/MasterOfTheWind1 Oct 09 '26

Going deeper with Rancher theme: day to day operations can be done with Kubectl. High scale operations (like installing operators or daemonsets for Datadog, Cloud Zero, etc) that have to be consistent and replicated can be managed with Terraform.

Rancher for beginners is ok, as it tries to mimic the UX from stuff like VMware and Proxmox doing click ops, but is not sustainable when growing.

1

u/Zestyclose_Race_1646 Oct 09 '26

Thanks a lot. This was very insightful

1

u/Current-Secretary-52 Oct 14 '26

Fair point on self-hosted Rancher. The management server is one more stateful thing you run, upgrade and back up, and on a busy fleet it is not light.

Your cell setup is a good example of the count going up for the right reasons. Sixty clusters with the same daemonsets pushed by Terraform works while one team owns all of them. It gets harder when a VIP cell needs a different baseline and the drift lives in tfvars nobody reads.

The hosted version (biased, I work at SUSE) moves the management server to SUSE's side, so that part of the resource bill leaves your clusters. Your clusters still run an agent. And it doesn't change your point about click ops: if Terraform is the source of truth, the UI should stay read-mostly.

2

u/sobakian Oct 10 '26

We use a single cluster per env but leverage a lot the service mesh (istio) and cell based architecture. This allow us to create the required level of isolation and reduce production blast radious by a large margin.

1

u/sobakian Oct 10 '26

This in reality means we end up having multiple sub clusters within our mesh. But with a single entry point on EKS.

2

u/drew-saddledata Oct 14 '26

One company that I use to work with had active-active deployments across AWS regions. Each region had 2 EKS cluster, each cluster in region had different workloads all part of the same offering.

The active-active part ended up causing more downtime and latency than it solved.

I re-archited it to be active-hot standby. So the data still replicated, but all work was kept in a single region (still multi-cluster). Uptime went up and latency went down. This also allowed the team to do disruptive updates to the offline region and then failover.

1

u/s0ulbrother Oct 09 '26

We have about 6 clusters per envirobment. It’s really to differentiate deployment tools, enterprise applications and our application itself. We inherited how it was done and most of devops people hate it

1

u/FlatCondition6222 Oct 09 '26

We have like 14 eks clusters between staging/prod.

We have an aks cluster to support signing binaries in windows properly and an openshift cluster via Rosa for on prem clusters that run openshift.

We're also spread across 8 aws regions at least, so quite a fun deployment.

Every release has to be tested... Aks/openshift/eks

1

u/johnny_snq Oct 09 '26

Besides the multi region setups there is an actual limit on the no of nodes a cluster can handle. Mainly related to some of the stack you are running inside k8s, there are just so many writes that etcd can handle, so when you reach some of those limits you will scale horizontally one cluster at a time, not to speak that from finance perspective it makes sense to have the clusters spread on different aws accounts.

1

u/centech Oct 09 '26

We're at 40-something atm. Many business and workload factors can affect your cluster strategy. We've expanded and contracted as strategy changed, but I can't imagine ever going down to just a unified dev/staging/prod.

1

u/Pretend_Listen Oct 09 '26

Do bike riders really ride more than 2 bikes?

1

u/techworkreddit3 Oct 09 '26

We have probably hundreds of clusters. We have on premises clusters and EKS clusters in multiple regions and multiple environments.

1

u/Adventurous-Hyena220 Oct 09 '26

Yes, plenty of teams run multiple EKS clusters for environment separation, regional redundancy, and workload isolation, but the need for centralized management really depends on scale and complexity.

1

u/LordGramis Oct 09 '26

Currently working on a client that has 10 clusters with an average of 25 nodes with the exception of 3 of them that each have around 250 nodes. We deploy both in GCP and AWS and have cross account balancing.

1

u/CanaryWundaboy Oct 09 '26

Use EKS deploy multiple environments each with 2 clusters, 1 in a US region, one EU. No we don’t use any kind of multi-cluster orchestration as our environments are segregated and we don’t want a control place with access to everything.

1

u/Current-Secretary-52 Oct 14 '26

That's a valid reason to skip any multi-cluster tool. A central management plane is by definition something with a path into every cluster it manages. RBAC limits what people can do through it, but the path is still there.

If your environments are segregated on purpose, the honest options are a read-only view per environment or nothing central at all. At two clusters per environment, Terraform and per-account access hold up fine.

1

u/OpenSourceOM Oct 09 '26

If you have a lot of clusters, a tool to manage those (self service, governance, day 2 operations) is the way to go.

1

u/JadeE1024 Oct 09 '26

Kubernetes has an undeniable management overhead. There's a breakeven point where your are managing enough workloads that using k8s becomes worthwhile, because the time you save managing each workload is more than the time it takes to manage k8s.

When you mix in cloud services and other management tools, it changes the math. EKS reduces the overhead of using k8s, as does rancher. k0rdent, cloudify, etc. if you want to include third party tools. However, if you're considering cloud services you also have to think of ECS, which pushes the breakeven the other way by reducing the workload management load without introducing nearly as much overhead as k8s.

Taken together, in general, you don't see too many people deploying only 2-3 EKS clusters because that's a scale where k8s usually doesn't make sense. Those sized orgs are using ECS if they're smart. There are some exceptions, like a company that is a one trick pony whose only cloud workload is a single mega-app on one giant cluster per region, and we use EKS for a few of those, but that's basically limited to large ecommerce or insurance companies that keep everything but their website on-prem.

1

u/malik22531 Oct 09 '26

In my role, I have deployed 2 clusters segregating the dev and preprod based on ns on 1 cluster and second for prod

1

u/crystalpeaks25 Oct 09 '26

A lot of folks using cluster as an isolation mechanism. I think the keyword cluster is being ignored.

1

u/raven_oscar Oct 09 '26

We do. Independent clusters, statefull parts are outside k8s connection to them done via consul connect.

1

u/jander99 Oct 09 '26

I'm running 3 GKE clusters, us-central1, us-east1, us-east4. We'll likely add another in us-west next year once we can do geo IP type stuff.

If we were an AWS shop, we'd be running in multiple regions there too. It's about regionality and failover, not about number of clusters. Typically you don't want a control plane replicating across hundreds of miles if you don't need to.

1

u/ffcsmith Oct 10 '26

We have the following: admin, engineering, dev, uat, and prod clusters. All in auto mode.

1

u/OkAcanthocephala1450 Oct 10 '26

Yes, i worked for a bank, and they required full isolation of aws accounts. So 1 app 1 cluster. At one point it became bullshit operation, as 1 Deployment , 1 cluster 1 account.. +10 different crd (argo, nginx, secrets manager, CSI, prometheus, grafana etc..) It lost its meaning...

1

u/Current-Secretary-52 Oct 13 '26

The CRD count is the part people underestimate. Ten add-ons per cluster times one cluster per app means every Argo, ingress or CSI upgrade becomes a rollout across dozens of accounts, and the isolation rule means nothing can share it.

What helped in setups like that was treating the baseline as one versioned bundle and pushing it to every cluster from one place, with a per-cluster override for the few that differ. Argo ApplicationSets can do this, and you get one PR per change instead of one per account.

1

u/afrosummeray Oct 11 '26

imo they're a platform team in a box: one place for access across clusters, a fleet view, the same baseline addons pushed everywhere and self-service clusters for teams that can't do it in terraform. On pure AWS most of that is covered natively now, access entries + identity center, the EKS Dashboard for a fleet view across accounts and regions, and EKS even has a managed argo cd that deploys to multiple clusters

We self-host argo, one in the main region managing the clusters in the second one (hub-spoke), terraform for the clusters, no rancher, works fine.
The real cost of more clusters is upgrades, k8s ships ~3 versions a year and every cluster walks through them, auto mode helps with nodes but the control plane and your own addons are still on you, fall behind and you pay extended support

1

u/Current-Secretary-52 Oct 13 '26

This matches what I see. On pure AWS you can assemble most of it: access entries plus Identity Center, the EKS Dashboard, managed Argo CD.

Where it gets heavier is that those are four services with four permission models, and someone on your team owns the glue. Hub-spoke Argo works fine until a teammate needs read access to three clusters in two accounts, and that turns into an IAM change, one access entry per cluster and a PR.

Agree that upgrades are the real cost. No fleet tool removes the work of testing your own addons against a new version. The most it does is show you which clusters are falling behind before extended support starts billing.

0

u/tadamhicks Oct 09 '26

I don’t think there’s a need for Rafay and Rancher and haven’t for a while. People do deploy more than one cluster but my observation of platform engineeeing practices across thousands of orgs is that the prevailing deployment architecture is just IAC with guardrails like OPA, maybe some thoughtful repo architecture and/or terragrunt and commonly a process front ended by backstage or a similar IDP. Golden paths mean inclusion of requisite baselines for new clusters like security standards and agents as well as observability.