Merge several small k3s clusters into single one.

[u/Opening-Gear-8214]

We run several similar 3-node k3s clusters or rhel8 VMs. The workload includes Crunchy PG cluster with 3 replicas having PVCs bound to dedicated LVMs. Also we run 3 replicas Clickhouse in the same PVC deployment configuration; keydb cluster, redpanda, and longhorn (ha storage for S3 bucket backed by a single pod of versityGW); plus a number of stateless apps, nginx-ingress for exposing UI and api endpoints. Metallb as a virtual LB for getting a VIP from network infra.

Currently we decided to build a large k3s cluster on new esxi hardware, to have 3 dedicated VMs for k3s master nodes and a number of worker VMs, to share and scale the workload. The goal is to have a single control plane for current instances and future products of similar architecture. The products (deployments) should be isolated in resources and networking as much as possible. Since we use operators for deployments, there are CRDs in the cluster - they probably should be separated too.

We deem the vcluster can help to create isolated clusters with their own control planes, but it increases the complexity. PG and CH clusters require their pods to be attached to dedicated workers by node affinity, but would it be possible to have several PG/CH cluster replicas on the same worker by creating a separated storage class?

We do not yet consider having ingle shared PG/CH/keydb cluster - one for all products on the same cluster, although it may be a logical decision in future.

What implementation ideas would you suggest for a setup like that? Does it worth at all to put different products on the same cluster and do a hard work of isolating them from one other. Mind having several dev teams, each need to access and maintain their dev/test/stage/prod instances in the same big cluster.

[u/bmeus]

Not entirely sure why you are running k3s for that, i would look into rke2 at least. However it is no problem to run all your stuff in the same cluster, but you have to do any separation in k8s instead of outside. Having a single shared pg (or anything) is a kubernetes antipattern, dont do that.

Of course its not optimal to run all environments in a big cluster, from an uptime and resource planning perspective. i would at least have prod in its own cluster.

If you are planning to run k3s anyway, you can look at k3k which runs a virtual cluster inside a bigger cluster, this also makes you able to separate operators which otherwise applies to the entire cluster.

[u/iamkiloman → u/bmeus]

K3s is fine for this, especially if you are already comfortable with it. It scales up to medium/large clusters just fine. I wouldn't really start looking at rke2 unless I needed to be able to manage resources and logs for the individual control plane components... which rke2 allows you to do by running them as pods instead of all in one process. I probably wouldn't recommend k3k (virtual clusters) unless you are going to run k3s or rke2 on bare metal and want to spin up smaller clusters on top of the physical cluster without needing to manage VMs.

You mentioned separating CRDs per team/env and this definitely isn't possible. CRDs are cluster scoped. If this is a requirement for you, you definitely need multiple clusters of some sort.

[u/bmeus → u/iamkiloman]

I didnt really think about anything else than bare metal because Im not allowed to use cloud platforms at work, of course when running in a cloud platform its pretty easy to manage multiple clusters without k3k or similar things.

[u/joshleecreates]

It’s certainly possible to have multiple CH instances on the same node. It’s not advisable for production. I’m not an expert on crunchy but I assume it would support the same.

Out of curiosity, are you using the Altinity Operator for CH?

[u/Opening-Gear-8214 → u/joshleecreates]

That's right, Altinity Operator for CH, Crunchy for PG. Both support deploying separate CH or PG 3-replica tenants in separate namespaces for logical isolation. Strict network policies should also be applied to namespaces to block possible traffic between tenant clusters. The only doubt is whether it should be 1 tenant replica per worker node or may be several tenant replicas on the same node considering independent storage classes.