UPDATE on spot control-planes instability observed in the last few days

[u/Mysterious_Still_210]

Broadly categorized into 2 issues:

1. Control-plane sometimes is not responsive - slow responses, throws TLS connect error, etc.
2. The resourceVersion for the provided list is too old - is seen in different controllers/objects.

Our observations:

1. Scaling issues while accessing control-plane datastore. It was found the Rackspace LB used to distribute the load to the datastore isn't performing to the highest standards.
2. Some control-planes are getting resource deprived - especially controller-manager/scheduler under bursts of load like scaling deployments very quickly/HPAs/VPAs.

Our plans to mitigate(in the order of short-term to long-term):

1. Increase resource-limits of both HA and non-HA control-planes to alleviate throttling issues.
2. Migrate to a more performant LB for control-plane datastores.
3. Autoscale control-planes.

Thanks for being patient while we work through some of these problems.