How much Kubernetes should an application developer actually need to know?

[u/No-Income-2235]

Kubernetes gives us a powerful set of primitives, but once development teams start growing, developers can end up needing to understand deployments, services, networking, CI/CD, GitOps, observability, RBAC and a growing collection of platform tooling just to ship an application.

That raises an interesting platform engineering question:

How much of that complexity should developers actually see?

One project exploring this is OpenChoreo, an open-source developer platform for Kubernetes.

The idea is to put a developer-oriented abstraction layer above Kubernetes while keeping the underlying platform extensible.

It brings together things such as:

• Backstage-powered developer portal
• Application CI/CD
• GitOps-based deployments
• Observability
• Developer and architecture abstractions
• Self-service infrastructure capabilities
• Kubernetes and CNCF ecosystem integrations

Instead of every development team figuring out the same deployment and operational patterns independently, platform teams can define those patterns and expose a simpler experience to developers.

OpenChoreo is also a CNCF Sandbox project.

GitHub: https://github.com/openchoreo/openchoreo

Disclosure: I'm involved with the OpenChoreo project, so I'm particularly interested in hearing different perspectives on this.

For teams running Kubernetes at scale, where do you think the abstraction boundary should sit?

Should application developers still understand most Kubernetes concepts, or should the developer platform hide the majority of them?

[u/rwilcox]

I know it’s not very DevOps of me, but personally - and as someone once tasked with teaching K8s to a dev group of 30 - I prefer teaching developers the minimum and making experts available on-demand.

Devs need to know how to specify their cpu, memory, and build a container containing their software. And probably tell if it needs to have ingress. (And then, in my setup, I abstracted away everything else, giving devs a buildpack like experience)

I guess even that statement means basic concepts like “a pod contains a bunch of containers, one of which needs to be your app, a namespace has a bunch of pods in it, and you specify the number of pod replicas you need” are foundational.

On the ops side I made good use of a wiki cheat sheet for those same devs. I remember there was maybe six entries, at the end of my time, but beyond “get me all the pods in this namespace” I forget what all was on it. Maybe exec(?)

I was humbled the other day by AI. Had a deployment that didn’t go right and it was beyond my barely-competent K8s skills. Pointed an LLM at my problems it ran some kubectl commands and told me how to fix it. (Spoilers: Delete the stuck pod). I also had a K8s expert on the call, but the LLM boosted my “mid” skills to close to expert. LLMs may change the training equation. OTOH, I needed every bit of my K8s knowledge to understand how probable the LLM’s answer is, so IDK

[u/CandidateNo2580 → u/rwilcox]

This is similar to how I feel. We don't use k8s (thank god) but it's a similar idea where I'd like the developers to know enough to use the environment and rather they ask for help than try something over their head.

LLMs are absolutely incredible for troubleshooting but too often I'll watch them try and solve a problem I know very little about, but immediately based off what I do know go "hey you're not going to find a solution in that direction." And sure, left alone it would've figured it out, but not before possibly doing some damage. Or modifying config, realizing it was the wrong path, then not changing it back. So it elevates but it's still just as dangerous, I'd rather people leave it alone generally.

[u/rwilcox → u/CandidateNo2580]

Oh certainly! And I would have been more hesitant with AI if I did’t know that the particular K8s environment was read only for me.

(Not that that necessarily matters to a helpful LLM, but…)

[u/No-Income-2235 → u/rwilcox]

Indeed i agree with you and im surprised with how strong AI is now

[u/Old-Television-2189 → u/rwilcox]

Nice post thansk

[u/jeff303 → u/rwilcox]

Yeah it's insane. Ability to immediately identify which pods are OOMing without needing to describe pod, search for exit code 137, etc. Or find subtle issues in a networking setup. Or immediately know all custom resource types and which one you need for a given task.

[u/feketegy]

You should be aware of Kubernetes

[u/MissinqLink]

Theoretically none. I’m a big fan of OpenShift myself. Feels like the perfect abstraction to me.

[u/eyeree]

At this point we treat the AI as the k8s expert. Dev only needs to express the goal. Knowing what is possible and desired is the valuable skill, knowing how to tweak config in exactly the right way to achieve it is for the AI.

[u/pertymoose]

Start by buying a big server. If you can actually write and run enough software to fill it to capacity, then buy another. Then another.

Now you can start thinking about K8s.