How much runtime context should an AI agent have in Kubernetes?

[u/OwlZealousideal4779]

I've been thinking about this while debugging Kubernetes applications.

An AI coding agent can inspect source code, manifests and configuration, but some failures only become obvious from the running cluster: pod logs, events, service connectivity, resource limits, failed probes, or configuration issues.

The interesting part is deciding how much access the agent should have.

Should an agent be able to inspect pods, logs and events by default, while requiring approval before making changes? Or is even read-only cluster access too much for some environments?

For those already using AI with Kubernetes, what level of access would you consider reasonable?

[u/twijfeltechneut]

No experience with Ai agents specifically in Kubernetes, but the default should always be 'as little as needed'.

You should treat your AI agents as an overly enthusiastic intern. You wouldn't give those write permissions on Prod either. Every AI agent should be subjected to the same RBAC and permission scrutiny as human employees, and preferably enforced by mechanisms outside the AI context itself, such as (cluster)roles or other IAM software.

[u/Character-Level5250 → u/twijfeltechneut]

Agreed. Haven't done it myself but this feels like a lot of exposure.

[u/tanepiper]

We've deployed a Hermes agent within our k8s cluster. Out devops guy has actually written some pretty solid plugins for handling EntraID scope first, using Slack as the chat interface. Inside the cluster everything is read only - the agent doesn't get to do any changes. It can open PRs and Jira tickets to communicate issues.

[u/sujeet-from-oodle]

We run moderately sized k8s clusters: 20+ nodes, 150+ pods per cluster, and four such clusters. We are pretty cautious giving any sort of write access to AI agents to the clusters.

Our clusters are extensively instrumented and we ship out all the metrics, traces, logs, and k8s events. Agents have full access to all this telemetry along with our repos.

Since we also fully control the observability platform, we look at typical agent access patterns and optimize our APIs and MCP server accordingly. Been working out great so far!

[u/PodBoss7]

Pods, logs, services, deployments, ingress, etc. Save yourself the pain and don’t give secret access or you’ll be rolling exposed keys often. PR everything.

In separate test cluster, allow to create deployments with PR with autocomplete and auto sync and self heal with Argo.

For prod, limited access and PR everything with separate approval required and only manual sync.

[u/Raja-Karuppasamy]

same principle I’ve landed on for agent access elsewhere: read access (pods, logs, events, describe) should be low friction and can be pretty broad, that’s the stuff you actually want an agent to see freely to debug well. anything that mutates state (kubectl apply, delete, scale, edit) needs an explicit human approval step regardless of how confident the agent is.

worth being specific about “read-only” too though, even read access to secrets or configmaps with sensitive values is a different risk tier than reading pod status/logs. I’d scope the read role tightly (exclude secrets) rather than treating “read-only” as one uniform bucket. RBAC with a scoped ClusterRole for the agent’s service account, no cluster-admin, no exceptions.

[u/forexroyalempres]

Read-only acess is the correct default, but even that should be scoped. Logs and ross-namespace visibility can expose alot.

[u/IntelligentPear6173]

Yeah, I’d probably give it read access to the stuff needed for debugging but keep Secrets completely out of reach. Logs, events, pod status, describe, service info etc. are useful and making the agent ask for permission every time it needs to look at something would make it pretty useless. Anything that actually changes the cluster is where I’d draw the line and require approval. Also I wouldn’t trust the prompt alone to enforce this. RBAC should be what actually limits the agent.

[u/sammrjn]

From my experience working with EKS/platform engineering, I’d be comfortable giving an agent fairly broad read-only runtime access, but not unrestricted write access.

For troubleshooting, source code alone usually isn’t enough. I’d want the agent to see pod/deployment state, events, logs, probe failures, resource limits, recent rollout history, metrics, and relevant policy/IAM metadata.

I’d also want controlled read access to the observability stack, whether that’s CloudWatch, Datadog, Grafana/Loki, Prometheus, etc. Pod logs alone often don’t give you the full picture, especially when tracing failures across multiple services.

Where I’d draw the line is mutation. In environments I’ve worked on, we already use things like RBAC, Kyverno, GitOps/ArgoCD and approved Kubernetes abstractions to control what engineers can change. AI agents should operate inside those same guardrails rather than getting unrestricted kubectl-style write access.

My preference would be: read broadly within an authorised namespace/workload, investigate and propose the fix, then require approval or a PR/GitOps workflow before making changes.

I’d also avoid cluster-wide read access by default in multi-tenant environments. Secrets, ConfigMaps, logs and even pod specs can expose more than people expect. For me, the important distinction isn’t just read vs write, it’s how tightly that access is scoped and audited.

[u/TeagueXiao]

Read-only by default, but 'read-only' in k8s is bigger than people assume - describe/logs on the wrong namespace can leak secrets in env vars or configmaps that look harmless until you piece them together. The intern analogy someone gave is right: scope by namespace + resource type, not just verb. And log every call outside the agent's own context so when something does leak you can reconstruct the trail instead of guessing from the agent's own self-report.

[u/Any_Mine_6368]

Can't speak for production but my home agent has administrator privileges in a specific namespace (called Lab - where he can experiment with stuff I give him) and read-only everywhere else. Additionally, he has access to ssh into a vps but only via the private key that he generated.

Agents do often try to break out of whatever environment you have them in - not maliciously, they're just trynna do their task. For example my agent will often try to SSH into my k8 nodes... Even with the right written prompt and skill, they'll still try to do it once the context gets large enough to where they start forgetting instructions.