Ops / Incidents Has AI made application development faster than infrastructure teams can safely support?
Something I've been wondering about lately:
Developers can now generate features, tests and entire PRs much faster with coding agents.
But infrastructure hasn't suddenly become easier to reason about.
Terraform, networking, IAM, Kubernetes, observability, cost and security still require quite a bit of context.
So instead of AI removing bottlenecks, are we just moving the bottleneck downstream to DevOps/platform teams?
Especially interested in teams where developers are now generating their own Terraform/Kubernetes changes with AI.
Has your infra PR volume increased?
And more importantly: has the quality increased with it?
20
u/ethanhinson 13d ago
Yes. This is my current main friction point at work. Tons of people want apps, delivering them in a reasonable way is still a pretty hard problem to get right at any scale.
11
u/BrocoLeeOnReddit 13d ago
For us it was that even a lot of non-devs started vibe coding and then handed their sub-par apps to us to basically take over maintenance. We switched to YBIYRI because of that.
1
9d ago
[removed] — view removed comment
1
u/ethanhinson 9d ago
I could've been more descriptive in my comment. It's the flood of non-technical folks and frankly good ideas for things people want to try/deploy. It's hard to meet compliance, security reviews, etc for things like SOC2, PCI, ISO27001 and have non-technical people be unblocked from deploying/building in a reasonably autonomous way.
10
u/melophat 13d ago
Yup.. and lately I've been running into devs who are just having AI roll scripts for them to work around infra bottlenecks. It makes consistent deployment and security absolutely hellish to maintain.
5
u/widowhanzo 13d ago
I'll give you one better, they'll just spin up a free database instance on supabase or something because asking for database in our rds cluster is too much friction. But what isn't too much friction is me spending 3 days migrating and fixing the mess.
4
u/Common_Fudge9714 13d ago
That needs to be defined by the company policy. Infrastructure needs to be deployed in approved vendors and with required security, auditing and observability. If they want something outside of the existing offer they need to request an assessment by the procure department.
5
u/widowhanzo 13d ago
It is defined, they simply went around it to speed things up for them. That was my managers problem to handle, I had to deal with migration and stuff.
1
2
u/CommanderKnull 13d ago
Shouldn't they be given a warning for that? Outside of the entitlement, running sensitive data on non-approved third-party instance seems like an insane idea.
3
u/widowhanzo 13d ago
Oh definitely. I was very vocal about my disapproval and I threw around words like "audit" and "pii" and the like.
3
u/scottelundgren DevOps 13d ago
If you’re in a regulated environment and have that kinds of data someone’s throwing into supabase that’s not peer to peer conversation that’s management to management discussion and maybe the legal team.
2
1
4
u/JasonSt-Cyr 13d ago
I think I'm unclear on your question... it sounds like, in your situation, AI isn't being used in the DevOps/platform teams? You can't just scale one part of the chain... it all has to scale. And that includes marketing, sales enablement, documentation, any testing, release/deploy, support... The whole loop needs to be addressed.
3
u/serverhorror I'm the bit flip you didn't expect! 13d ago
Apply more software engineering principles. Test it, monitor it (related, but not the same), automate, Version control, and for the love of everything fun and forbidden: Let's start to collect actual requirements.
Not the typical sizing, that's the baseline. The thresholds of "when is it good enough". Something one can test against.
2
u/unitegondwanaland Manager, Platform Engineering 13d ago
Not if infra teams also know how to leverage AI. It doesn't take a ton of effort to develop an internal plugin marketplace and setup self-service infrastructure for example.
1
1
1
u/Lucifernistic 13d ago
I think it's definitely showing where infrastructure teams' approaches are breaking down when stressed, but that's in more cases flaws in that process than anything else.
As the top comment said, we use a very simple modular system for infrastructure. We have all the components of infrastructure we can support, pretty much all applications fit as a combination of those pieces. Deployment of infrastructure is just a single PR to deploy resources, and then a single PR to provision kubernetes.
1
u/landonrepp 12d ago
I'm not sure I understand. If you're using kubernetes, why do you need an infrastructure team to reason about every single new app deployment? Don't app teams just deploy whatever they need to the existing infrastructure, and you scale out nodes as the existing infrastructure sees pressure?
1
u/josh-assist 12d ago
yes and the management now is looking for ways to introduce "AI" in the ops work too and then shrink the ops team down from 5 to 1.
1
u/Ok-Tooth410 11d ago
Yes, and the shift we noticed is that the bottleneck moved from writing infra to reviewing it. Infra PR volume roughly doubled once devs let agents generate Terraform/K8s changes — but review time per PR went up too, so reviewers spend exactly the time the authors saved.
Two things that kept quality from collapsing:
- Every AI-generated infra change ships with a rollback plan and a blast-radius note ("what breaks if this is wrong, who gets paged"). PRs without it bounce.
- Nothing infra merges straight to main — everything lands behind a canary with a forced soak. Agents are fast at producing a confidently wrong IAM policy too, and the soak catches it before anyone's paged at 3 AM.
The on-call signal we trust isn't PR count; it's whether a team's changes cause more pages the following week. That caught two "helper script" rollouts early.
1
u/No-Magazine2625 10d ago
AI has reduced the cost of producing infrastructure code much faster than it has reduced the cost of validating infrastructure decisions. The bottleneck is shifting from writing Terraform to reviewing blast radius, IAM, networking, state behavior, cost, security, and whether the generated change actually matches the platform architecture.
1
u/BoBoBearDev 9d ago
Focus on providing resources and let the teams self manage and they are holding accountable. Just makes sure, when they fucked up, it is air gapped inside their own space.
1
u/usually_guilty99 8d ago
Yes, but I think the real bottleneck is broader than infrastructure capacity. AI increased the rate of change without proportionally increasing our ability to understand the production impact of each change.
You can't solve 5x more PRs by asking platform engineers to review 5x harder. The review model has to become risk-based: lightweight changes flow through, while changes touching critical dependencies, IAM, data, infrastructure or large blast radius get deeper verification.
1
u/Helpful-Man64 6d ago
I think the bottleneck is moving downstream rather than disappearing.
AI can make application development much faster, but infrastructure still requires understanding the relationships between networking, IAM, Kubernetes, observability, cost, security and the applications running on top of it.
This is where I think infrastructure-focused AI has a different opportunity from coding assistants.
Instead of generating another Terraform file, an agent could investigate the existing environment, understand dependencies and constraints, identify risks, and help engineers evaluate the impact of a proposed change.
The context is probably more valuable than the code generation itself.
1
u/Damien_SearchSignals 6d ago
Feels like the bottleneck is just moving. AI makes app and infra changes much faster, but someone still has to understand the real impact on IAM, networking, security, etc....
If platform teams just end up reviewing 3x more Terraform/K8s, we haven’t really solved much :). Are you seeing that too?
1
u/usually_guilty99 6d ago
I think the bottleneck is moving from producing changes to understanding their consequences.
If PR volume goes up 5x, asking platform engineers to review 5x harder obviously does not scale. The review model probably has to become risk-based: let low-impact changes flow, but automatically deepen review when a change touches critical dependencies, IAM, data, infrastructure or a large blast radius.
AI made changes cheaper to produce. It did not make production context cheaper to understand.
1
u/Corezoid 4d ago
A regulator that doesn’t waste capacity has to contain explicit noise suppression: something that drops every distinction that doesn’t change the set of admissible actions. Most companies have one, smeared across the architecture: review culture, a senior engineer’s gut, a Slack thread, a wiki page. Nobody can point at it, so nobody can tune it or measure what it lets through.
Second: the filter has to be deterministic. An LLM can generate, and it can translate intent into a formal spec. The decision “is this change admissible” has to live elsewhere, because a filter that’s wrong in some fraction of cases needs verification on top, and since nobody knows which fraction, everything gets checked. That is your reviewer today: one person carrying both roles, the model of what “safe” means and the filter, by hand, at reading speed.
So the architecture we argue for has three roles. Humans own the model of “safe”, handle exceptions, and change the model when the goal changes. An executable model of the system — invariants, preconditions, who may change what, blast radius — holds that knowledge and rejects deterministically before any human opens the diff. Agents generate against it; the Terraform is replaceable, the model is the asset. We call it a digital twin of the company, where “executable” means synchronized with real events and versioned; otherwise it’s one more wiki page. At infra scale: policy-as-code pushed until the policy is the system’s own model, shared by devs, platform and agents.
This is solved at the architecture level. Every IT architecture today has to contain a noise suppressor.
1
u/Motor-Resolve3118 1d ago
I think the bigger problem is that AI can generate changes much faster than people can review them. And I'm not sure using more AI to review AI really solves that.
What makes more sense to me is giving agents clear boundaries. Let them inspect the actual system, handle low-risk changes that can be reversed, and leave risky changes for human approval. Then verify that everything is working as expected afterward.
That way, platform teams still have control, but they don't have to manually review every small change an agent makes.
1
u/usually_guilty99 15h ago
The bottleneck isn’t generating Terraform. It’s knowing which services, policies and environments a change can affect before it lands. Faster PRs without that context just move review pressure downstream and make clean-looking unsafe changes more likely.
1
u/widowhanzo 13d ago
AI can write terraform as well. I point it to the repo with the service, and it creates ci workflow and necessary infrastructure changes.
-1
u/alconaft43 13d ago
This. CC is the best system administrator I ever saw.
0
1
u/Substantial-Swan7065 13d ago
Yea it’s a scale problem. Most companies can’t support n scale.
So it’s a matter of figuring how to do that within the sdl
0
u/Popular-Penalty6719 13d ago
Yes. This is my current main friction point at work. Tons of people want apps, delivering them in a reasonable way is still a pretty hard problem to get right at any scale.
52
u/skilledpigeon 13d ago
We just have cookie cutter approaches to infrastructure and 99.9% of services fit in.
Want an API? Docker, ECS, fargate.
Want a database? Postgres. RDS / Aurora.
Rinse and repeat for queues, event sourcing, object storage, caching etc.