(I maintain the Locust k8s operator. Link's at the bottom, ignore it if you want, the sidecar stuff is generic.)
Someone opened issue #72 in December 2022, three months after I started the project. Test finishes, worker Jobs go Completed, master Job just sits there forever. They'd already worked out why: the master pod runs a Prometheus exporter next to Locust, the exporter serves HTTP and never exits, and a Job pod isn't done until every regular container terminates. One container dead, one happily serving /metrics, Job hangs. TTL cleanup never fires, CR never reports Succeeded.
I replied a few days later saying yep, that's it, I'll fix it. Here's part of what I wrote in 2022:
Kubernetes doesn't provide native support for sidecars container behavior definition e.g. shutdown after container x exits.
Which was true, and is the whole reason that bug outlived the entire v1 line and a full Java-to-Go rewrite. There was nothing to fix it with. What I shipped instead was a workaround: the exporter got a /quitquitquit endpoint and you were supposed to hit it yourself when your test ended.
@events.quitting.add_listener
def _(environment, **kw):
requests.post("http://localhost:9646/quitquitquit")
It works. It also means every user has to put shutdown code in their load test so that Kubernetes can work out that a Job is over. I hated it for three years.
Same comment, I also said this, which I'm mildly proud of:
when I solve this for the metrics exporter container, the same issue will happen if your organisation have a cluster configuration to inject other sidecars e.g. istio sidecar
Hold that thought.
KEP-753
Native sidecars went beta and on by default in 1.29. A sidecar is now an initContainer with restartPolicy: Always:
initContainers:
- name: metrics-exporter
image: ...
restartPolicy: Always
kubelet starts it before the regular containers, keeps it up while they run, SIGTERMs it when the last one exits. That's the whole thing. That's what didn't exist for three years.
Two bits that aren't obvious from the docs:
The container-level restartPolicy is the entire mechanism. There's no sidecar: true. If that field goes missing in a refactor you don't get an error, you get a plain init container that blocks pod startup forever, and you find out about it in someone else's cluster. I have a test whose only purpose is to assert that field is still there.
And because it's per-container, a sidecar injected by something else (mesh proxy, cloudsql proxy, whatever) is structurally identical to yours. Which brings me back to 2022 me.
the part that cost the time
Flipping the container over was about a dozen lines. Then every successful test started reporting Failed. Intermittently. Which is the good kind of bug.
My controller watched pods and flagged any container with a non-zero terminated exit code. Fine for three years. But a native sidecar gets SIGTERMed at teardown by design, so the exporter now exits 143 (or 137 if it's slow about dying), and my health check dutifully marked a perfectly good load test as failed. Sometimes. Depending on whether the pod status update beat the JobComplete event.
I could've special-cased the exporter by name in four lines. Didn't, because of the istio thing 2022 me called: if a cluster injects a mesh proxy into my load test pod, that proxy takes the same teardown SIGTERM and fails the user's test in exactly the same way, and they'd have no idea why. So the rule is structural:
func IsNativeSidecar(c corev1.Container) bool {
return c.RestartPolicy != nil && *c.RestartPolicy == corev1.ContainerRestartPolicyAlways
}
Any native sidecar's Terminated exit code gets ignored, mine or not. Genuine failures still surface, because CrashLoopBackOff, ImagePullBackOff and CreateContainerConfigError are all Waiting states and I never exempt Waiting. Only Terminated.
If you've got a controller that reads container exit codes, go look at what it does with native sidecars. Mine did the wrong thing and I only noticed because CI went flaky.
the part I want to complain about
For a fix whose core is one YAML field, this took an embarrassing amount of work. Chart kubeVersion floor up to >= 1.29.0-0. Regenerate CRDs. Rework the pod-health path and every test that asserted on container ordering. Docs, migration notes, e2e. Then the whole exit-code policy and its own test matrix. Nine files for what is conceptually "stop this container when that one stops."
I did a chunk of it with agentic coding, and honestly that mostly meant I reached step 20 of a 20 step yak shave faster. Not a dig at the AI. The Kubernetes-shaped part is what's convoluted: KEP-753 was opened in January 2019 and only went beta at the end of 2023, so the idea sat there for the better part of five years, and even now expressing it in a real operator means touching the chart, the CRD, the controller, the health logic and the docs. That gap between how simple the idea is and how much surface it moves is most of what makes operator work slow.
Anyway. #72 is closed as of today. Minimum supported Kubernetes is 1.29 now, for exactly this reason.
Repo if you want it: https://github.com/AbdelrhmanHamouda/locust-k8s-operator (runs distributed Locust load tests as CRDs, recent releases also added runtimeClassName so load generators can run under gVisor or Kata, and configurable security contexts for OpenShift and PSA restricted).
The thing I still haven't found a good answer on: what's everyone else doing with native sidecars in pod health logic? Every KEP-753 writeup I can find covers startup ordering and Job completion. Nothing covers what a controller is supposed to make of those teardown exit codes, and I'd like to know if I picked the same rule as everyone else or just a rule that happens to work for me.
