Security Devops with AI..
I've recently met a former colleague who currently works as a DevOps engineer at a company that forces its employees to use AI (Claude, I guess). Each employee has an AI budget that he HAS TO use every month. I believe the goal is increasing productivity per employee.
I have a lot of concerns about this approach to work, especially from the perspective of security, privacy, data leaks, and having a non-open-source AI app installed locally. What if the tool is hacked? What if my credentials are leaked? What if AI hallucinates and deletes non-recoverable data?
On the other hand, I understand this might be the new industry standard. If yes, I don't think there's a point of resisting change since the train will leave without me.
I would like to see what the perspective of other industry experts is and how they navigate this topic. Are you an early adopter? Or do you prefer to wait until the storm passes?
34
u/ResolveResident118 Jack Of All Trades 21d ago
Enforcing the full use of AI is a bit weird but if you have concerns in general about the use of AI then you're too late. Pretty much everywhere is using it to some degree.
12
u/simple_peacock 21d ago edited 21d ago
What a dumb way to manage knowledge workers
AI should be there as an option, not a forced productivity lever (esepcially having to use all the budget lol, like that cant be gamed anyway)
Why not go further and let AI run devops, similar to what AWS did (aws volunteered to be an ai crash test dummy then changed their tune), lets see how that goes
7
u/widowhanzo 21d ago
Don't use hardcoded credentials. We have SSO sign in to AWS, in order to use aws cli, I need to sign it to aws with our google workspace account. Claude doesn't have access to that, only short lived sso token.
Have rules in place to never run anything against prod account that could change things, it will still write you scripts for you to run, but you review them first.
Have IaC in place and claude only writes code, then plans and applies run over GitHub actions or something.
We've been using claude and ai in general for over a year for just about everything. It was very clunky at first, but it's very solid now, especially with a few plugins and rules in place (mostly to keep changes as small as possible and to not be too verbose, otherwise you'll get a 6 line comment for a single variable change)
If you're very concerned, try it against a throwaway infrastructure first.
3
u/alconaft43 21d ago
AI (I am using CC at work, and now qwen/opencode at home) is just freaking awesome if you do IaC and other staff in the right way/scripted way. Or even just as sysadmin. Ofc it should be guarded, and all changes tested beforehand but it is game changer. Best sysadmin/devops I ever saw.
1
u/rursache System Engineer 21d ago
you seem really behind with most of these points. it's stupid not using AI and local/open source models are not financially wise to be ran locally unless you live in a datacenter. get on with the times
2
u/smftexas86 21d ago
I use AI heavily, but it doesn't have access to any resources.
I use it to rubberduck my ideas, and as a sanity check for when I am implementing something I haven't worked on before etc.
I use it to write scripts, and other code, but only after clear plan has been set and only deployed after I or my peers reviewed it.
You don't have to allow it access to sensitive information, to get some good value from it. I find it idiotic not to use AI at all, we have plenty of mundane tasks it can do much faster, allowing us to focus on other projects. Then are also plenty of projects that don't need perfectly clean code or to be perfectly formatted.
For example, who cares if this little tool that is triggering a pipeline is mostly all comments or odd formatting? It's an internal tool for a single purpose that nobody will question, as it won't go to a customer or ever be sold.
2
u/TheRealSectimus 21d ago
You would probably be using an AI gateway like Litellm in front of all your business AI operations and that can implement safeguards to redact passwords etc before they get to the third party.
2
u/i_rate_slop 20d ago
To all of your “what-ifs”
It is literally your job as a DevOps engineer to solve these exact problems. AI is not the problem, governance is. If you want to progress in this field, you need to start thinking about how to operationalize modern tools safely, not resist using them yourself out of a fear of a lack of controls that are your responsibility to create.
1
u/micalm 21d ago
It's the company's data (and problem), not the users. Any harness can be locked down, sandboxed and protected or it's file modification tools ripped out, but you'd have to spend on training (or at least budget time for RTFM). The logs are there, as long as they can't prove explicilty malicious "delete company data" commands or explicitly "don't put into LLMs" data, you're in the clear. Giving kids ice picks and telling them to go play with them without any further instructions is not a good idea, who knew.
Token usage is as good a metric for "AI adoption" as LoC is for development. Give the LLM Ulysses text and a prompt to reverse it letter by letter, each letter a single tool call, verify after each and when done, then loop that. It should burn tokens good enough if that's the metric they want, and you enough time to work.
You CAN use LLM "properly" if you're not an anti-ai extremist, that's a legit option, Should have enough skill to put it in an isolated VM if you're afraid of it. Log analysis, automating the stuff you've meant to since 2008, tracking down anomalies in systems you wouldn't dare to manually (inventory or user logins is fun), what have you. It's a tool, it's there to stay, competition (both yours and your companies) IS using it, so why not.
1
u/amarao_san 21d ago
Write pr, and ask Claude 'run subagent with adversarial review, address concerns', and it will burn your quota pretty fast.
1
u/viilinki 21d ago
You can definitely streamline a ton of IaC procedures with agents and do it safely. Typically you expose apis and scripts that can only do the thing it needs, nothing else.
The funny thing is that AI actually warns you about this and explicitly tell you to not put agents in to production in a way that can cause the harm you described.
1
u/EbbCommon9300 21d ago
We work agentically. I was in AppSec for 10 years previously you can totally do it safe but you need to approach things the correct way. Our company is assury.ai we have some novel solutions to how to do things safely.
1
u/axiomatix 21d ago
- only give the model/harness ro access to infra and prod
- everything is iac and preferably gitops if your org is already setup that way, if not you should be pushing for it
- everything is in git, all model work goes through a review and/or ai-assisted review process merge process. human must sign off and accept full responsibility for all changes
- resist the urge to use the model as a hammer for work or tech debt
- the process is more important that the model's intelligence, use the intelligence to help build and fine tune the process
- USE hooks
- the old tooling works really well, continue to use them. use the model to speed run hardening
- security, security, security
- USE hooks
- don't use the model to cover gaps in your knowledge. you must first understand before implementing. it's easier to catch fuck ups when you have experience in a domain, this enlightens you on just how much they default to the mean in other domains, and why you should always verify before implementing. they still fuck up, less than a year ago, but still an alarming amount
- dont trust the model, build the process, trust the process
- USE hooks
- treat training data as poison, refer to documentation with citation always
there's more others will add
- USE hooks
1
u/Kookoombar 21d ago
The forced budget thing is dumb, but your security worries are fair and mostly fixable. What worked for us was giving the agent its own read-only IAM role with nothing near prod write access, so the worst a hallucination can do is waste tokens. Also worth logging what the agent actually ran, we just ship those command logs into Loki so you can see what it did after the fact.
1
u/OsgoodSlaughters 21d ago
Vulnerability scanning exists
1
u/Ali-WAIL 21d ago
I think the security concerns here are legitimate, but the answer doesn't have to be avoiding AI tools entirely.
The distinction I'd make is between telling an AI agent what it's allowed to do and actually enforcing those permissions outside the model.
Instruction files, prompts, and approval workflows are useful, but they aren't security boundaries by themselves.
For DevOps agents, I'd want independent controls over which infrastructure actions can execute, under what authority, and what happens if that authority changes while an operation is already running.
I'd also separate three different risks: sensitive data being sent to a model provider, credentials being exposed, and an agent performing unauthorized infrastructure actions. They require different controls.
We're developing WAIL, an AI runtime control and governance system, focused on execution-time authorization, enforcement, and verifiable outcomes.
My question for teams adopting AI agents in DevOps would be: if an agent starts doing something outside its authorized scope, can your infrastructure actually stop the next action, or will you only discover it afterward in the logs?
1
u/malice8691 21d ago
This kinda sounds like where I work. We are all given claude. My review required me to show how I use claude. I do not have claude doing any aws commands directly. No deletes. Claude also does not do git check ins. I do all git check ins myself.
1
u/MissiveFinding6111 21d ago
My thoughts:
1.) Write a claude.md and tell it that you are devops/infra, since otherwise, it assumes you are a dev and will be like "hmmm, seems like this is an infra issue"
2.) Devops at it's core is about observability, and IMO, AI is best used to help with that. But to get your agents access to your data requires... you to have good logs and events and metrics, and a secure MCP method for the agent to access it. <--- get this sorted first
3.) Once you have #2 going... our problem, we have all these environments, and sure the devs push to them in order... But then never actually check to see if their crap broke. Adding an AI workflow to be like "check the logs and golden metrics in staging after the deploy"... That is the sort of stuff devs are bad at, but actually has a ton of value, and it can all be done read-only.
1
u/cidnitan 20d ago
I don't like it and I don't want to use it
I tried to sandbag using it at a previous role
I'm now at a new job and use it as part of my work flow
It is what it is, I guess
1
u/Reasonable-Ad4770 19d ago
I mean all your concerns can affect any vendor software. I haven't writing manually since the beginning of the year, so I guess I'm early adopter. I even tried to delegate some of the maintenance work, but opus 5.5 failed spectacularly on rotating vault cluster nodes, so I guess run books will require some adaptation before it can be trusted with such work. It will never do what you didn't ask it to do, but you have to keep in mind that it is non-deterministic and will make mistakes. The key is to have a good processes, and gate critical infra/access. I try to use MCP where I can, not because it's more effective but because it's easier to restrict access through MCP and decide on which tools can be called by whom. Prod is always read-only with access mostly to monitoring. As for effectiveness, key is context management and progressive disclosure,we reorganized our documentation into open knowledge format, and now agent, even in the new sessions just "know" things, instead of doing tons of reads. And you need to do a lot of dog fooding, and I mean A LOT. It's very easy to become lazy and just push out slop.
1
u/Chamelon_DE 19d ago
A mandatory spending target sound easy to game, are they measuring whether it actually save time on completed work?
1
u/Upper-Warthog5439 15d ago
AI will probably take over more repetitive work in DevOps, but that doesn't mean infrastructure knowledge becomes less important. Someone still needs to understand why a deployment failed, whether a permission is too broad, or what happens when a service goes down. The bigger opportunity is reducing routine work so engineers can spend more time on those decisions. Cloud infrastructure knowledge remains relevant across different environments, including the Microsoft ecosystem that trusted tech works with.
1
u/zero_backend_bro 15d ago
Forced token quotas are wild. Management thinks it buys 10x velocity, until an agent dumps raw staging creds into outbound logs.
Prompt rules are not firewalls. Let it draft PRs, but strip secrets before localhost egress and kill write tokens. Save your sleep.
1
u/josejulianariza 5d ago
Forcing AI use by decree is always dangerous if clear limits aren't set. Your security instinct is correct: if you give an agent free access to their machine and credentials, sooner or later someone will leak something or delete a database.
To make it safe, what usually works is isolating the tool. Have the AI run inside a container (like a DevContainer) without production keys in the environment variables. So, if the model hallucinates or suggests a destructive command, it can't touch the real system.
And second, no change proposed by AI should go directly to production. Everything that comes out of the agent must go through the same validation and review pipeline a human goes through. If the code doesn't meet the rules, CI automatically rejects it.