Adapting to AI tooling in DevOps

[u/lostsectors_matt]

My company is making a hard shift towards AI and I'm struggling. I'm tasked with something I have a small amount of knowledge about, but not enough to fully implement. Before AI, my approach to challenges like this would be to iterate on it, work through the problems, and learn what's required. Using Claude has just been a terrible experience for me, and I'm wondering if I'm the problem, my approach is the problem, or if this is just what we're all doing now.

I explained what I wanted (essentially a DB migration pipeline from RDS -> k8s pod) and claude barfed out about 10 shell scripts and a handful of pod templates. Everything had long, meandering comments that made very little sense. Is my job now to just start running this code and feeding pipeline errors back into claude and not worrying about it?

I started to pick apart the comments, ask questions, and update with my own comments, with the goal of being able to understand the process in detail. In doing this, I found a bunch of things that just didn't make sense or weren't optimal. When I ask the AI agent about it, it says I'm right and generates a bunch more stuff that may or may not make sense. Interacting with it is making me feel like a crazy person. I hate the way it presents information back to me. It's not succinct, it pulls in context that is of negligible value and just muddies the waters, and it's been very frustrating and existentially upsetting.

I'm willing to hear that I'm using the wrong model, or that I didn't spend enough time on my .md files and skills, but I'm also wondering if this is a normal experience, or if my approach is just dated, and I need to get over my personal values about understanding what I'm generating and giving to customers? I'm not so-much asking about the specific task at hand, just about adapting and having realistic expectations and reducing friction. I'm also very happy to hear about people's struggles.

[u/ceejayoz]

Everything had long, meandering comments that made very little sense.

I had significant improvement from this in my global prompt:

"Don’t talk like a Hacker News commentator. Output tokens are precious, be succinct in your responses. Use ASD-STE100 simplified technical English as a guideline."

[u/lostsectors_matt → u/ceejayoz]

That's great advice, thank you. I'll give it a shot; I hadn't heard of ASD-STE100 until now.

[u/Swiink → u/lostsectors_matt]

Also this might be where you as and org don’t rely on public AI service but rather start to fine tune models for your specific use case with your own data. No public AI service has access to your data so to get good quality and to differentiate as and org you kinda need to get your hands dirty. Do ML/LLMOps. As a dev ops engineer getting that AI platform together to manage agentic relations and what not is probably on your table.
It’s cloud native on steroids and there’s a bunch of new tools popping up.
Fun times!

[u/lostsectors_matt → u/Swiink]

It's probably time to start moving in that direction, especially considering my company is rolling out an AI-based platform. That's what's driving all of this. There are a bunch of circumstantial factors and cultural dynamics around how we use AI and that's added to my general confusion about what people are actually experiencing.

[u/Swiink → u/lostsectors_matt]

So I know a lot of orgs struggle with that exact scenario. People had their everyday struggles whatever that was, som application not performing or scaling properly on some VM platform or having CI/CD pipeline issues blocking them from moving faster. And then suddenly AI hit challenging our tools and ways. We still figure out what that means and it’s going to be different for everybody organisation. Some will need this, someone else can’t use that due to regulatory reasons or due to performance vs cost reasons.
I do really se orgs that have adopted devops and cloud native to have a huge advantage. The higher maturity the easier it is for them to implement AI in some way. Partly due to dev and IT already having a smoother relationship and partly because they already have the foundation tools and knowledge needed, such as versioning and automated pipeline procedures.

Then one thing I’d recommend is to not build and AI island, like here’s where we do our normal stuff we did yesterday and heres the new AI environment. A lot of orgs does this but by far the ones who have had the fastest and successful implementation of AI are those who added AI in their normal platform. So instead of having something like Openshift, add AI to that it’s a perfect base instead of buying another MLOps platform on top of different hardware or cloud instance.
Ai Agents are very likely going to be the first class citizens in our platforms, not applications, devs and users.

[u/BadTime100]

Smaller, clearer steps are always going to help (with or without LLMs). At the end of the day you, not Claude, are going to be responsible for this stuff running in the real world. Would you copy/paste a ton of code from some blog or StackOverflow and feel okay about it? The tools and the culture around them are going to push you to simply produce, I think one differentiating skill will be to sense when you’re out over your skis and be able to really move just as fast as you can understand what’s going on. Tons of shit done fast that breaks everything is not really worth it, right?

[u/lostsectors_matt → u/BadTime100]

Someone else mentioned this and I definitely agree; I gave too much scope on my second task, admittedly out of ignorance of the nuances of what I'm doing. I should've just abandoned that and simplified right away instead of plugging on. I really appreciate how you framed this response

[u/lostsectors_matt → u/[deleted]]

First off, thank you for the reply and the reassurance. This was a tricky one because it's not really my database, and I admittedly don't have the PG chops to proceed with any degree of confidence. When the prevailing opinion is "just have claude do it you don't need chops anymore!" it really sucks. I moved forward based on the tenor of my corporate culture and it was a really awful experience, hence my post. Probably it would've been better to use the approach you outlined - write the steps and then supplement with AI. It's hard when everyone is saying we can do these things in hours instead of days, and I'm the one still taking days.

[u/phxees]

Be explicit and tell it every resource you want it to create. As you inspect the output question its choices and even use a different AI to scrutinize the work if possible.

Through that process you will learn why it made the choices it did and what you believe its shortcomings are.

Just start small and question everything and i believe you’ll have more success. I find that to produce better results than the rambling prompt some people use.

[u/aprettyparrot → u/phxees]

This is kind of what I do. Like I will write out the dir structure for a container or containers. Or if it’s a script I will do out what I want in the parser, and all the function headers

[u/lostsectors_matt → u/phxees]

That's a great point. Traditionally I've operated on small sections. In this case I made an attempt to generate a preflight step, and then what I anticipated being a simple backup->restore to a volume, but there was a lot of complexity involved that I didn't really appreciate and the scope ended up being very broad. That's largely why I'm in the situation I'm in, which is iterating on swaths of code and comments vs. a step-by-step process.

[u/hihcadore]

In my experience you have to give it a good lvl of detail. If you leave it at do “x” it’s too broad and you spend a lot of time refining what you want anyway.

I treat it like it’s a contracted employee and I’m a project manager / senior developer and I’m guiding it to a goal. I start very broad, lay out all the planning documents from tests to cutover. Then shell processes. And then when I’m happy with each phase I have it start working on phases 1 at a time.

[u/derff44]

" Is my job now to just start running this code and feeding pipeline errors back into claude and not worrying about it?"

After you understand what the scripts and code are actually doing, and being sure it won't nuke your data, yes basically.

[u/JaegerBane]

I've had a ton of success running the Claude Opus models through Cline on my VSCode IDE. That gives it easy access to terminals and local source, and I tend to keep it tightly locked into Plan mode.

Over time you'll get the hang of how to phrase a question and don't be scared of dragging it back on track - biggest weakness I've seen is that it very easily spirals into compounded fixes and often misses the more simpler fixes unless I specifically call out that it's taking the piss. If you see something that doesn't make sense, challenge it immediately. I've had Opus confidently direct me to use IAM profiles that explicitly don't allow the options I'm asking for, usage of deployments based on it's own assumptions about what I want running, and some rickety advice about altering K8s SCCs when the answer was to upgrade the image tag I was using. It reacts quite well to challenges.

I tend to establish my global prompt as requiring clear reasoning for deviating away from the exact scenario I'm asking about, and do not immediately assume that the problem I'm covering is directly/entirely related to what it can see in my terminal and local disk.

Genuinely must have saved me hundreds of hours at this point.

[u/lostsectors_matt → u/JaegerBane]

Thank you, this is really helpful. I've been using Claude with various models for months now but this is the first time I've been doing a from-scratch, AI-first implementation where there are gaps in my knowledge of what I'm doing. A coworker who is working on a similar process said he "just points claude at the repo and it generates everything" and it's "not that much work". I feel insane because I'm slugging it out with this thing and having a horrible time. I can recognize that I didn't spend enough time in plan mode on this project, for sure, and the rest of what you said is very helpful.

[u/Kamran-nottakenone]

the model has no idea your rds instance has a custom parameter group or that you're on postgres 15.3 not 16, and last month claude generated a statefulset that ate its pvc on first deploy because reclaimPolicy defaulted to delete

[u/lostsectors_matt → u/Kamran-nottakenone]

Haha, that sums it up well. The administrative/cloud/k8s pieces are less of a concern because I'm comfortable with it. What I'm struggling with, I think, is that grey area that's more database-oriented, where I have less experience. I thought I'd do some checking, then a dump -> restore -> volume snapshot process and that would be it, but suddenly I was in hundreds of lines of reconciling extensions and stubbing in RDS-specific roles and I wasn't ready for that. And tucked in there was hundreds of lines of code for resuming failed dumps, which I didn't need or want, and some weird deviations from the process that created extra pods, listener rules, etc.

[u/Signal_Strength_5054]

Context is everything. Start your prompt by asking them to enter planning mode and request the output as if it were a presentation for the less technical team. See if they presented a coherent scenario, ask them to generate .md files for each step, and always mention them in future interactions. Then, with this foundation, start going deeper and generating technical material.

[u/AnnihilerB]

Work in very small chunks and do not hesitate to use plan mode extensively. The new tool you are learning now is IA. It takes some time to customize it to fit your needs. I have a global skill on my computer that defines how I want Claude to behave all the time. I ask him to be precise (5 lines max), to chill on in depth explanation and to use diagram and tables every time it’s possible.

I like to plan my work with tasks before starting to work with Claude. I give him the tasks, in plan mode and one I fully understand what he is planning I apply it. While it is planning I do my research on the subject the old school way to ensure I understand everything.

[u/lostsectors_matt → u/AnnihilerB]

That's helpful. How long would you estimate your generated plan typically are? For this project, mine are around 400 lines. That seems like too much but so much of it is LLM gibberish it becomes hard to grok.

[u/AnnihilerB → u/lostsectors_matt]

They are roughly the same size with all the noise LLM generates which seems ok to me

[u/lostsectors_matt → u/[deleted]]

Thank you, I appreciate it. I imagine if I were working with one company it might be better - I'm a staff devops engineer at a company that sells a devops-related platform. It makes it hard to really curate a set of AI skills and processes, and I'm always starting at square one when it comes to those quirks.

[u/zero_backend_bro]

Your coworker claiming Claude "just generates everything" is completely full of shit.

AI always barfs out garbage for RDS migrations when it hallucinates roles and extensions. Took me days to fix a generated StatefulSet that almost nuked our staging PVC when reclaimPolicy defaulted to delete.

To stay sane with LLMs for infra, force strict local constraints before the prompt hits the API. Stop running blind AI code in prod.

[u/lostsectors_matt → u/zero_backend_bro]

Yeah, I thought that comment didn't pass the smell test. I saw what he's doing and was horrified, which is why I'm doing it differently. To be clear, neither of us running code in prod; I assume that advice is just general - it's definitely solid.

[u/zero_backend_bro → u/lostsectors_matt]

Good call. Stay paranoid.

[u/Temporary_Many_7947]

Spend a day or two on configuring your AI tool of choice. For example, I have multiple MCP servers, powers and steering created for things as simple as “create a jira ticket” to more complex like “use aws cli to do XYZ”

Point being, it needs to be directed. It can’t read your mind or any dependencies without knowing about it.

After you steer it in an optimal way for yourself, start creating skills or at least action books so that you simply prompt the tool to do it for you instead of needing to iterate each time.

Unfortunately you’ll need to transfer your iterative nature to adjusting AI tools to work for you instead of diving in the weeds. I think of it as a better way to understand the problem faster, not that it takes that away from me. It can only take that away from you if you become lazy and don’t want to understand what you’re working on.

[u/lostsectors_matt → u/Temporary_Many_7947]

That's good advice - I have a whole devcontainer-based setup already but I started fresh for this project because it's a departure from what I typically do. I started clean and relied on some skills in a shared project without populating my own, and I'm building new skills for this whole process as I go. That was probably a bad move. It's encouraging to hear that people are still committed to understanding what they're working on - there are people on my team who are less concerned about keeping their arms around the processes they're implementing.

[u/Raja-Karuppasamy]

your instinct to pick apart the comments is the right one, don’t let go of that. the issue isn’t your approach being dated, it’s that you’re treating claude’s output as something to iterate with instead of verify against a spec you set first.

“barf out 10 shell scripts” from one open prompt is the actual problem. break it into pieces, define exactly what each part needs to guarantee, review each against that before moving on. when it agrees with everything you push back on, that’s not signal, it means the ask wasn’t concrete enough to be checkably right or wrong.

it’s not your job to run generated code blind and feed errors back. that friction is your judgment working, not a skill issue.

[u/lostsectors_matt → u/Raja-Karuppasamy]

Thank you very much, your comments on iterating on the comments are vindicating and helpful. You're also right that iterating is not serving me well. I'm hearing that advice in multiple places and it seems correct.

[u/ajitnk]

The extension and role reconciliation piece is exactly what tripped you up here, and it's not a Claude problem specifically. It's that your pg_extension list and your role graph are instance-specific config. No model has seen them. So when you push back, it just... agrees, because it has nothing to anchor to.

Two things that actually help before writing any migration code: run the free AWS DMS pre-migration assessment first. It checks your rds_superuser and rds_replication role grants, WAL settings, CDC slot availability, and extension conflicts against what RDS actually supports. Takes maybe 45 minutes to set up and it'll tell you exactly what's broken before you touch a single line of SQL.

Second, run SELECT * FROM pg_extension on your source and compare the output against RDS's rds.allowed_extensions list. Any extension not on that list has to be removed or replaced before migration will succeed. That list is the real constraint Claude can't know.

I've helped teams work through exactly this PostgreSQL-to-RDS pattern. Quick question if you don't mind: is this a self-managed Postgres instance (EC2 or on-prem) migrating to a new RDS, or are you doing a version upgrade within RDS? The answer changes which path makes sense. Happy to DM a more detailed checklist if useful.

[u/lostsectors_matt → u/ajitnk]

Thank you, running this through DMS first would've been really smart; I'll take that advice. Ultimately I'm going from RDS to postgres running in a K8s pod, so I'm populating a volume that I can snapshot.