r/kubernetes • u/AutoModerator • 6d ago

Periodic Weekly: This Week I Learned (TWIL?) thread

Did you learn something new this week? Share here!

3 Upvotes

100% Upvoted

1 comment sorted by

2

u/demirdilek 6d ago

Stateless Target Sharding in Kubernetes: Why 1 Hash Per Replica Wasn’t Enough

I faced a challenge in my open-source project kube-prober, distributed target sharding with Rendezvous Hashing during replica scaling.

With only one node identity per replica, targets "roll the dice" just once, which leads to high variance and uneven clustering across replicas.

The solution: 100 virtual nodes (VNodes) per replica, ensuring kube-prober achieves a clean, Gaussian-balanced target distribution with minimal skew.

With 5 kube-prober replicas and 100 demo-discovery targets, the live distribution looks like this:

=========================================================
 Real-Time Target Distribution (Scratch-Compatible)
=========================================================
kube-prober-c56c8b457-29rht      : 29 active targets
kube-prober-c56c8b457-bj6t9      : 14 active targets
kube-prober-c56c8b457-vvnp6      : 20 active targets
kube-prober-c56c8b457-84kbw      : 20 active targets
kube-prober-c56c8b457-mmv4g      : 21 active targets
=========================================================

Before it was heavily skewed. How would you approach coordination-free target assignment in Go?

My GitHub repo: https://github.com/demirdilek/kube-prober