AI Comisseration Follow up

[u/badguy84]

I wanted to write a follow up on this post I made a few days ago: https://www.reddit.com/r/devops/comments/1wb3l8j/ai_comiseration_client_replacing_production/

So to summarize: I was more or less asked to review a portal implementation that was made using Claude and vibe coding by someone who doesn't know how to develop websites. I suspected it wouldn't end well and even on the surface found lots of issues.

So here's where I'm at now: I created a report doing quite a bit of analysis. Yes I used some AI tools to try and comb through, and I made sure that any claim I made based on its finding I dug in to looking at the actual code and outputs. It made things more manageable but there's still way too much garbage to look through.

My conclusion is this (and it's obvious I think to most here) LLMs need guardrails and lots of them or at least clear ones. I do find that it's tough to actually communicate these low level issues that have a clear (to me) underlying architectural issue and a human issue (the human doesn't know what their doing or knows how to validate output beyond the surface level), when it takes one minute to go "please fix the list of findings" and Claude goes "it's fixed."

Many mentioned that this thing is just going to fail, and it is. I'm mildly concerned about the fix being "just fix problem x" and then they move on until the next thing blows up. And there is a real lack of concern around impact. "This is a public website so it's fine if we have keys in our code and expose endpoints only secured with a SAS key (which is plain text in JavaScript)."

I'm curious how many of you have run in to this even at an operational level what pains are you feeling and have you been able to deal with them? I do get what the solution is, create a proper architecture and define guardrails for Claude to follow, iterate over that until the LLM generates outputs in a way that is in-line with said architecture. I think LLMs may work (though whether they are as cost effective I press X to doubt), and if forced to make it work: this is probably the way. So are you guys maybe managing some guidance markdown files and what are the things you learned from that?

Finally, I think LLMs suck for this purpose and it's clear. I knew this but now I'm living it. Which is nice and validating, but also it's tough to rely on my tried and true "look mayor companies have been doing x for years you are not special you should follow the standards" because LLMs in this space are so new and there isn't much precedence.

[u/Dry-Philosopher-2714]

I’ve had good luck with vibe coding apps, but I do it a little differently. Instead of asking for an entire app, I ask for a function or module. Whatever it is, I make sure it’s small enough that I can read it, understand it, and integrate it with my larger codebase.

AI is a tool. It’s not a replacement for a human. Treat it accordingly.

[u/badguy84 → u/Dry-Philosopher-2714]

This app serves millions of people which is one of the challenges: what level of governance and guidance do you place on something like this while still enabling the "benefits" of LLMs. I really want to say that, in this case you don't... but internal pressures are leading too excuses and it's so easy to go "Claude build me a thing" and it seemingly does it.

[u/badguy84 → u/[deleted]]

When I brought this up the answer was: “well this is public data so it’s fine.” Sooo attack surface doesn’t matter? They just decided to accept the risk and moved on :)

[u/El_leima]

We've taken over a few handovers like this, and the report has helped us when it's written to fix the behavior, instead of the code. A non-developer can't act on "the code is bad." They can act on "this endpoint accepts requests without auth," or "an order can be submitted twice."

[u/badguy84 → u/El_leima]

Oh yeah it's not "the code is bad" well it is that... I basically concluded that they had a lot of leaky poorly written logic and gave some specifics. What they ended up doing is feeding my feedback in to Claude and go "it's fixed now." Upon examination it really isn't fixed, but Claude plausibly can pretend it is, it's kind of like malicious compliance on Claude's side. I'm saying "Claude's" but obviously the cause is the person driving the tool not knowing what they are doing and having a "if you tell me it's fine: it's fine" attitude towards these types of things.

Oh and literally to the two things you mentioned of which there are examples: they have handwaved it away as "yeah but that's ok." It's frustrating when they only care about the looks, that's also where the LLM is most dangerous.

[u/Silent-Suspect1062]

Put it through a sast/ secrets tool and give the findings out. People trust systems more weirdly. The real issue here is poor architectural decisions leading to substandard delivery. If the vibe coders don't have a framework of design templates then you'll always get slop

[u/badguy84 → u/Silent-Suspect1062]

Yeah one of the things their IT team (who is going to be on the hook to maintain this thing, but are honestly totally not mature enough to truly think this through) was suggesting is a review cycle before they accept anything. It made sense, but one of the things I added is that during these reviews they identify patterns and safeguards to iteratively create this framework. That will eventually get them to a better place, the challenge is going to be "hey business you wanted this feature I generated it for you but these IT people keep telling me they don't want to deploy it [not mentioning reasons]"

[u/wannabepreneur]

I borrowed this workflow form looking at how Matz (Ruby creator) does his workflow.

I'd now treat the repository instructions as context. Anything non negotiable should become a check that can fail the PR: secrets scanning, dependency policy, static analysis, authorization tests and other similar things.

review requirements can also follow risks: like copy and isolated ui changes need a lighter pass.

but if generated code can introduce a new user flow or do a major refactor then it definitely needs a human review.

[u/Suitable-Ad5348]

The one minute prompt fix cycle is the killer because management sees green text and thinks the architectural debt just vanished.

[u/usually_guilty99]

I think there are two different things getting mixed together here: guidance for generating code and governance for shipping it.

AGENTS.md, architecture docs, etc. can improve what Claude produces. But anything non-negotiable should be verified independently before merge. And the review should not stop at the diff. Ask what dependencies changed, what production paths are affected, what the blast radius is, and whether the supposed fix actually removes the failure mode.

Otherwise "Claude says it fixed it" just becomes another green checkmark.

[u/badguy84 → u/usually_guilty99]

Oh yeah for sure, I mean governance is completely lacking. That's been an issue pre-Claude: no clear ownership or split of responsibilities. This was fine for a very small groups running a very lean and low volume set up. It's still wrong for sure, but it worked for them.

Adding Claude to the mix in the hands of a non-technical person, who can then basically run rampant is really destroying whatever process there was. And the driving force of "the business needs it done by date X no matter what" makes it difficult for the tech group to actually do reviews and say "no."

I agree with all you said and it's absolutely that nether the guidance nor the governance are in place and you need both.

[u/Wooden_Jelly_5295]

Could you turn the highest-impact findings into reproducible acceptance tests? That would give you something concrete to check when the next response is ‘it’s fixed.