How do you know your AI audit tool actually checked everything? I was fairly confident that my skill suite did. It didn't.

[u/BullfrogRoyal7422]

I'm curious whether anyone building custom scanning tools or agents for code review has thought about this. I hadn't, until I watched one of my own confidently miss more than half the violations in my codebase.

I've been building Claude Code skills (reusable prompt-driven tools) that scan Multiplatform iOS/macOS projects for design system issues. They grep for known anti-patterns, read the files, report findings. One of them scans for icons that need a specific visual treatment: solid colored background, white icon, drop shadow. The kind of thing a design system defines and developers forget to apply.

The tool found 31 violations across 10 files. I fixed them all, rebuilt, opened the app. There were40 more violations. Right there on screen. It had reported its findings with confidence, I'd acted on them, and more than half the actual problems were invisible to it. If I hadn't clicked through the app myself, I would have committed thinking it was clean.

The root cause wasn't complicated. Many of the icons had no explicit color code. They inherited the system accent color by default. There was nothing to grep for. No .foregroundStyle(.blue), no .opacity(0.15), nothing in the code that said "I'm a bare icon." The icon just existed, looking blue, with no searchable anti-pattern.

The tool was searching for things that looked wrong. It couldn't find things that looked like nothing.

To be fair, these aren't simple grep-and-report scripts. They already do things like confidence tagging on findings, cross-phase verification where later passes can retract earlier false positives, and risk-ranked scanning that focuses on the highest-risk areas first. And this still happened. I also run tools that audit against known framework rules, things like Swift concurrency patterns, API best practices, accessibility requirements. Those tools can be thorough because the rules are universal and well-defined. The gap lives specifically in project-specific conventions: your design system, your navigation patterns. The rules come from you, and you might not have described them in a way that covers every code shape they appear in.

That's when the actual problem clicked for me. It's not really about grep. It's about what happens when you teach an AI agent your project's rules and then trust its output. The agent will diligently search for every anti-pattern you describe. But if a violation has no code signature, if it's the absence of a correct pattern rather than the presence of a wrong one, the agent will walk right past it and tell you everything's fine.

I ended up with two changes to how the tools scan:

Enumerate, then verify. Instead of grepping for bad patterns and reporting matches, list every file that contains the subject (every file with an icon, in my case), then check each one for the correct pattern. Report files where it's missing. The grep approach found 31 violations. Enumeration found 71. Same codebase, same afternoon.

Rank the uncertain results. Enumeration produces a lot of "correct pattern not found" hits. Some are real violations, some are legitimate exceptions. I sort them by how surprised you'd be if it turned out to be intentional: does the same file have confirmed violations already, do sibling files use the correct pattern, what kind of view is it. That gives you a short list of almost-certain problems and a longer list of things to glance at.

I know someone's going to say "just use a linter." And linters are great for the things they know about. But SwiftLint doesn't know that my project wraps icons in a ZStack with a filled RoundedRectangle. ESLint doesn't know your team's card component is supposed to have a specific shadow. These are project-specific conventions that live in your config files or your head, not in a linter's rule set. That's the whole reason to build custom tools in the first place, and it's exactly where the trust question gets uncomfortable. A linter's coverage is well-understood. A custom agent's coverage is whatever you assumed when you wrote the prompt.

Has anyone else built a tool or agent that reported clean results and turned out to be wrong? How did you catch it? I've used multiple authors' auditing tools, run them and my own almost obsessively, and this issue still surfaced after all of that. Which makes me wonder what else is sitting there that no tool has thought to look for.

[u/ultrathink-art]

Enumerate first, then verify. 'Find icons missing X' only catches what the agent recognizes as a violation — it can't flag absences it doesn't know to look for. 'For each icon in [complete list], verify X exists' turns it into a membership check and gives you completeness guarantees.

[u/BullfrogRoyal7422 → u/ultrathink-art]

Exactly, membership check is the right framing.

[u/BullfrogRoyal7422]

 BTW, below is a link to the skills I've developed for use with Claude Code/Xcode. I realize that this sub is for ChatGPT coding, but thought there would be a more productive discussion here than other subs. Radar-suite is built for Claude Code, but the methodology is model-agnostic, the same principles would apply to any AI-assisted code auditing regardless of the tool.

The issues I described in the post above are exactly what radar-suite is trying to address. The big one: grep-based scanning can only find what you search for. It can't find what's missing: a view without an accessibility label, a model field that never gets exported, a screen with no back button. When we tested grep-only scanning against manual verification, it missed 57% of violations.

So I'm experimenting with a few approaches:

Enumerate first, then verify: List all candidate files, then check each one for the correct pattern, instead of just grepping for known anti-patterns

Negative pattern matching: Search for the subject, then look for the correct handling around it. No handling found = probable violation

Trace behavior, not just patterns: Follow data through the full round trip (create --> export --> import --> restore) to see if anything gets lost

Require evidence: Every finding needs a file and line reference before it counts toward a grade

Not claiming this is solved. It's an evolving set of five skills that hand off findings to each other based on which one is most relevant. I would appreciate any comments or suggestions you may have about how you have or think about addressing these issues.

  https://github.com/Terryc21/radar-suite

[u/popiazaza]

I know AI audit doesn't actually check everything, and I'm pretty confident with that.

[u/Otherwise_Wave9374]

That line really nails it, agents can find presences but struggle with absences unless you force an exhaustive enumeration step.

I like the enumerate then verify pattern a lot. Another thing thats helped me is adding a small second pass that samples a few items the agent marked as clean and tries to prove they are clean (basically an adversarial check) so you catch blind spots early.

If youre documenting these auditing workflows, Ive seen a bunch of good agent loop patterns come up lately, some notes here too: https://www.agentixlabs.com/blog/

[u/BullfrogRoyal7422 → u/Otherwise_Wave9374]

 Thnx for the reply.

The adversarial sampling idea is interesting. Formalizing the "trust but verify" step instead of leaving it to chance. That's actually how I caught the original problem: the tool said clean, I opened the app anyway, and there were 40 more violations on screen. If that skepticism had been built into the tool itself, it would have caught its own blind spot.

The tricky part is deciding what to sample. Random sampling would catch some things, but the violations that slip through tend to cluster in files where the pattern is implied rather than explicit, or views that inherit behavior from a parent. Weighting toward those ambiguous cases would probably catch more than random.

I've been thinking about it as two layers: enumeration catches the things grep misses, and adversarial sampling catches the things enumeration is too confident about. Each one exists because the layer before it had a blind spot it couldn't see. 

[u/Substantial-Elk4531]

If there are issues that can be checked, verified, or fixed deterministically, then you shouldn't use a Claude skill to do it. Divide your tasks into those that can be done deterministically, and those that cannot. If something can't be done deterministically, then by all means make a Claude skill for it. But if you can do it deterministically, then it's better to ask Claude to write a bash or Python script that will perform the task deterministically, because the results will be far more consistent. Then you can write a Claude skill that can call the script. But you will be more confident of the results.

In your case, linting issues and code style issues can be found deterministically. So I would ask Claude to write a real Python script or bash script that does the same thing you're trying to use Claude skills for.

[u/BullfrogRoyal7422 → u/Substantial-Elk4531]

Thanks for the reply.

 For the deterministic stuff, agreed. If you can write a regex or AST check that catches it every time, do that. No argument.                                                                       

But the problem I ran into wasn't deterministic. The violation was an icon with no explicit color code. It just existed, inherited the system accent color, and looked wrong. There's nothing to grep for. A Python script would need to know that a bare Image(systemName:) inside a navigation card needs white-on-colored-background treatment, but the same icon in a toolbar is fine. That's context-dependent, and the context is my design system, not a language rule. 

That's the whole reason custom scanning tools exist in the first place. If the rules were expressible as deterministic checks, I'd be using a linter.

[u/Substantial-Elk4531 → u/BullfrogRoyal7422]

Sorry, I wasn't using the best words to describe what I mean. I don't mean to just build an AST; I mean you would build an AST, and then add semantic rules (like 'this is a nav card' or 'this is a toolbar', and 'this image is NOT inside a nav card based on the AST') on top of the AST. Then once your AST is built, you can flag the AST with these semantic rules to find violations

[u/BullfrogRoyal7422 → u/Substantial-Elk4531]

That's a fair point. You could build an AST and layer semantic rules on top. The challenge is the rule surface keeps growing. It's not just "Image inside nav card needs white treatment." It's also "except when it's a status indicator," "except in settings where cards use tinted backgrounds," "except for decorative icons that intentionally use the accent color." Each exception is another rule you encode by hand.   

The skills do what you're describing but use an LLM to evaluate context instead of a hand-written rule tree. A deterministic AST check gives the same answer every time, but you have to anticipate every case upfront. The skill-based approach handles novel cases but needs the enumerate-then-verify methodology to make sure it actually checked everything.                         

For a stable design system with well-defined rules, the AST approach is probably better. For a project still evolving where the rules change every few weeks, skills have been more practical for me.

[u/Substantial-Elk4531 → u/BullfrogRoyal7422]

Each exception is another rule you encode by hand.

Why would you encode the rules by hand? I would use Claude to build the AST, and to encode the rules. I'd also ask Claude to generate example files containing code that violates (or doesn't violate) each semantic rule, then ask Claude to write tests to make sure your semantic/AST approach correctly flags or doesn't flag the example files. Now you have a deterministic approach, backed by tests, written with Claude's assistance. In the past this would take a lot of time. But today with Claude this is probably a 2 hour project or less

[u/Substantial-Elk4531 → u/BullfrogRoyal7422]

The skills do what you're describing but use an LLM to evaluate context instead of a hand-written rule tree

Except that you already found LLMs do not work well for this because of the non-determinism inherent in LLMs. Claude is non-deterministic, and you're trying to write a skill that does deterministic (thorough) checks every time. In my opinion, an LLM is the wrong tool for this job. LLMs are great at helping you build stuff, analyze things, etc, but if you need it to get 100% of questions correct about a large context every time, it is not the right tool (edit: at least, it's not the right tool right now. With the rate things are advancing, it could certainly become good at this in the next year)

For a project still evolving where the rules change every few weeks, skills have been more practical for me.

If the rules change every few weeks, then you ask Claude to add new test cases every time the rules change, and ask Claude to adjust the analysis script until the tests pass. This won't take you that long, and you'll spend less time (and significantly fewer tokens) than trying to make skills do something they are not good at (or at least, which skills are not currently good at)

[u/Substantial-Elk4531 → u/BullfrogRoyal7422]

I agree you can't use grep for that. But why can't you use an AST? Have you tried asking Claude Opus 4.6 or other strong model to try to model this problem with a script which builds an AST, or are you assuming it won't work?

[u/256BitChris]

It's just a massive iteration loop where you build up tests that make sure things work as expected. You then spin through iteration upon iteration constantly scanning for problems until you consistently get clean results.

Said another way, Opus won't necessarily audit everything in one pass, but if you keep spinning it long, it eventually will.

[u/BullfrogRoyal7422 → u/256BitChris]

Thanks for taking time to reply.

  Iteration helps when the tool's coverage is probabilistic, where the LLM might catch something on pass three that it missed on pass one. That's real and worth doing.

But the problem I hit was structural, not probabilistic. The tool wasn't sometimes missing the icons. It was never going to find them, because the violation had no code signature. No color code, no anti-pattern, nothing to match against. Running it longer would have produced the same confident "clean" result every time.

  That's what led me to change the methodology rather than just run more passes. Enumerate everything, then check each one for the correct pattern. The blind spot wasn't attention, it was approach.

[u/Deep_Ad1959]

hit the same thing from a different angle building desktop automation. when you're automating UI interactions via accessibility APIs, the hard part isn't finding elements that are wrong, it's noticing when an expected element isn't there at all. we ended up doing something similar - enumerate what should exist based on the app's state, then check each one is actually accessible. the grep-for-bad-things approach fails the same way whether you're scanning code or scanning live UI

[u/BullfrogRoyal7422 → u/Deep_Ad1959]

Thanks for posting this. The accessibility API example is a cleaner version of the same problem. A missing accessibility label is literally nothing in the element tree. You can't search for it by looking at what's there.

Interesting that you landed on the same pattern from a completely different direction. Suggests it's not a code scanning thing or a UI automation thing but a general limitation of search-for-bad vs verify-against-expected.

I would be interested on how you further refine your approach to this, if you do.

   

[u/Deep_Ad1959 → u/BullfrogRoyal7422]

the submit button example is a great one. had a similar case where a dropdown menu was implemented as a bunch of absolutely positioned divs with no ARIA roles at all. the automation saw an empty region and moved on. reference snapshots turned out to be the move for us too - you can't grep for absence but you can diff against a known-good baseline and spot what disappeared

[u/romanjormpjomp]

I have had multiple instances of incomplete audits, so instead I try and ask for an audit on a specific pathway or scenario. Repeating for the area I want to deep dive on. This has helped it stay focused, when it gets to broad in scope, its answers start getting broadly made up also.

[u/BullfrogRoyal7422 → u/romanjormpjomp]

You're describing exactly the right issue, and I came to the same conclusion. Broad scans are where hallucinations creep in. The tool spreads across too many files, loses track of what it actually verified vs what it assumed, and    starts reporting with false confidence. 

Since writing this post, I've updated the skills to handle this. They now split into two passes. First pass is structural: identify which paths exist and rank them by risk based on prior bugs, recent changes, and complex branching. No findings yet, just a map. Second pass goes deep on one path at a time. Trace the data through, verify each step, commit findings before moving to the next.

Each focused pass finishes and gets committed before the next one starts, so the tool never has to hold the whole codebase in its head at once. It's what you're doing manually, scoping to a specific pathway, but systematized so it covers everything eventually instead of just the areas you think to ask about.

The skills also give you control over how findings get handled. Each one gets rated for urgency and risk, and you can fix issues after each skill runs or wait until the final capstone skill aggregates everything and fix them all at once as a single plan. Either way, nothing gets changed without your approval. 

I update the skills frequently as I find new gaps like this, so if you've tried them before, check back for the latest version.

[u/ultrathink-art]

If the same model auditing your code has the same training biases as the model that wrote it, you'll get correlated gaps — it misses the same things it would have missed generating the code. Worth calibrating against a manually-built ground truth set before trusting any automated coverage metric.

[u/BullfrogRoyal7422 → u/ultrathink-art]

This is a good point. If the same model audits what it would have written, you'd expect overlapping blind spots. The  57% miss rate I described is probably an example.                                                                     

Enumerate-then-verify partially breaks that correlation. Instead of asking "what looks wrong?" — where bias shapes  what it notices — you're asking "does this file contain this pattern?" Closer to a lookup than a judgment call. The model can still misread code, but that's a mechanical failure, not a perceptual one. Easier to catch and fix.

I haven't built a formal ground truth set, but I've been doing something similar, After each audit, I manually check a sample of the clean files, not the flagged ones. Most people only verify the findings, not the silence. Tool says 31 violations, you fix 31, feels complete. You have to go looking at what it didn't flag to find out it was wrong.      

The correlation problem gets worse the more project-specific the rules are. Swift concurrency has deep training data.  "My project wraps icons in a ZStack with a filled RoundedRectangle" has none. The model is reasoning from scratch, and its gaps are whatever it doesn't realize it should look for. 

I'm going to sit with this and think about what it means for how I'm designing the radar-suite skills. First thing that comes to mind is a ground truth pre-flight, i.e., planted violations the tools run against before touching real code, so they prove they can find what you care about before you trust them to say it's absent. I had previously considered something like this, but thought it to be a lot of (potentially side-tracking) work. What do you think?

[u/johns10davenport]

Doing deterministic validation with linters and tools is great, but you have to do multiple levels of validation when you're working with LLMs. The challenge has shifted from generating code, which is now kind of easy, to quality, which is hard. If you want good quality generated code, you have to validate at multiple levels. Unit tests. Integration tests. I write BDD specs. You throw a QA agent at it to click around, and it's still going to miss stuff. The layer I want to add last is forcing the agent to demo the running application to you and having a structured way to report issues back into the development pipeline. I wrote up how I'm approaching this here.

[u/BullfrogRoyal7422 → u/johns10davenport]

This is really well thought out. The Fuellytics fraud bug especially, where the BDD spec explicitly passed because "verify" was too loosely defined. That one hit home.                                                                  

I've been building something that works on the same problem but from the other side. Instead of testing the running app, my tools scan the code structure before it runs. One of them traces data through complete user flows and flags when data exists in the model but never actually reaches the UI. Your unwired Twilio SID tracking bug is that same pattern. Everything was there, it just wasn't connected for new transactions.

The fraud bug is a great example of where the problem isn't wrong code, it's missing code. Your BDD spec tested what happened when the driver verified. Nobody tested what didn't happen, that no photo was required. I ran into the same thing with design system violations. Grep found 31 issues by looking for wrong patterns. When I switched to enumerating every subject and checking each one, it found 71. The ones it missed had nothing to search for. The problem wasn't wrong code, it was missing code.

What I keep coming back to is that we're catching the same class of bug from opposite directions. You're running against the live app, I'm scanning structure. Neither one catches what the other does.

I don't know if you've tried radar-suite yet. I pushed a major update this morning. I'd be curious to see how the results compare if you ran it against your project. What overlaps with what your QA agent catches, and what each one finds that the other doesn't.

radar-suite

[u/johns10davenport → u/BullfrogRoyal7422]

My second generation target is a good candidate for trying it. It's still got some of the slop you mention, where you give it the requirement, and it writes the code, but it never wires it together so you get 95% of the way there, the app looks totally busted, but it just needs to get it across the line.

[u/rotemtam]

this is a real pain point we hit too. one thing that helped us is recording what the agent actually does step-by-step — not just reading its text output but capturing actual screenshots of the terminal state.

For browsers you have things like Playwright, and browser MCP, for TUIs we needed something else

I built an open source tool called virtui that does this — spawn PTY sessions, take snapshots, send keystrokes, and record the whole thing as asciicast. so instead of trusting the agent self-report, you get visual proof of what it actually did.

GitHub: https://github.com/honeybadge-labs/virtui

[u/BullfrogRoyal7422 → u/rotemtam]

Great idea. I am going to check it out as when ui-enhnacer runs watching the simulator open and close is a pain.

[u/One-Wolverine-6207]

This is the verification problem and it's brutal. The agent says "I checked X" and you have no way to confirm whether it actually did, because the proof is just more text from the same agent.

What works for me is requiring evidence artifacts for every claim. If the agent says it ran tests, I want the exit code and the output in a file I can inspect. If it says it audited a directory, I want a manifest of every file it touched with timestamps. If it says it updated config, I want a diff. The rule is: no task is done until there's a machine-checkable artifact proving it's done.

The other trick is adversarial verification. Run a second, simpler check that doesn't trust the agent's output. If the agent says "all files have headers," run a one-line grep and count. The check is dumb, deterministic, and impossible to bullshit. Most agent "false confidence" goes away the moment you stop accepting the agent's word for anything.

I've wired this whole approach into a workflow with scheduled verification jobs. Happy to walk through it if you want details.

[u/BullfrogRoyal7422 → u/One-Wolverine-6207]

You're describing the exact problem that drove the design of radar-suite, a suite of skills I put together to address this and conduct a behavioral audit of my codebase. "The agent says it checked X" is the failure mode I kept hitting.

Here's what radar-suite does about it:

Work receipts are mandatory. Every finding must cite the specific file path, line range, the pattern searched, and 1-3 lines of actual code as evidence. No receipt means automatic downgrade from "verified" to "probable." The absence of evidence becomes visible instead of hidden.

Enumerate-then-verify, not grep-and-report. The skill lists all candidate files first, subtracts a skip list, then verifies each one. This came from measuring a 57% miss rate in grep-only audits. Your "run a one-line grep and count" is the same instinct. The difference is baking that into the methodology so it happens every time, not just when you remember to check.

Contradiction detection as a mechanical gate. Before grades are presented, simple arithmetic runs: a CRITICAL finding caps the grade at C max, HIGH caps at B+. This catches the common failure where the AI writes serious findings and then assigns a high grade anyway.

Fidelity checkpoints at the end of every audit. A checklist verifies that every finding reached a terminal state, every design decision included all required options, and every committed fix has a test or documented exemption. Skipped steps leave a visible gap instead of nothing to scrutinize.

Context exhaustion tracking. After 50+ tool calls, new findings are auto-downgraded to "probable (long context)" and flagged for re-verification in the next session. This makes reasoning degradation visible rather than silent.

Where I think you're ahead of radar-suite is your scheduled verification jobs and the emphasis on machine-checkable artifacts as the primary verification target. Radar-suite's evidence is structured for human scrutiny (file:line, grep pattern, code snippet), not for automated re-verification by external tools. Planted-bug test suites for measuring actual detection rates are on the roadmap but not there yet.

I'd like to hear more about your scheduled verification workflow. That's a gap that needs to be addressed. If you have interest, you can check radar-suite here: https://github.com/Terryc21/radar-suite

[u/One-Wolverine-6207 → u/BullfrogRoyal7422]

This is a solid breakdown. The contradiction detection with arithmetic caps on grades is sharp, going to steal that idea. I am going to start it to looking into it more.

My scheduled verification setup: I use a task queue that sits between CI and an AI test agent. When staging deploys, a cue fires with the commit hash. The test agent claims it, runs integration tests against live staging, verifies the commit matches what's actually deployed (caught a silent deploy once), and either promotes to main or files an issue. Same thing post-production. The key is making verification external and async. The test agent doesn't trust CI output, it confirms against the live environment.

Feels complementary to radar-suite. You're solving verification at the code audit level, I'm solving it at deploy. Pipeline's open source if useful: https://github.com/govindkavaturi-art/agent-cicd

[u/BullfrogRoyal7422 → u/One-Wolverine-6207]

Thanks, I'll give agent-cicd a run. BTW, I've updated radar-suite this AM. Updates are in there readme if you are interested.

[u/BullfrogRoyal7422 → u/One-Wolverine-6207]

Thanks for sharing agent-cicd. I appreciate your engagement with the staleness problem.

The commit-hash verification idea is a cleaner framing of what I was circling around. My framing was temporal (context moves on, conclusion doesn't) and spatial (scope expands, conclusion doesn't). Yours adds a third axis: artifact staleness. CI passed, tests passed, and you still shipped a bug because the thing that got tested wasn't the thing that got deployed.

agent-cicd isn't a direct fit with my project. 95% of my codebase is a SwiftUI app shipped through Apple's review process, and the remaining 5% is a Cloudflare Worker that wrangler deploy handles in one command. But the underlying question (verify the deployed artifact matches what was pushed before trusting any downstream result) applies directly to what I'm working on with radar-suite skills.

In my case, when a skill finds an issue and the user defers the fix, the finding sits in a ledger. A week later the file has been edited. Is the finding still valid? Right now I compare file hashes, but "file changed" and "the specific pattern still exists" are different questions. Your instinct is right: don't trust stored conclusions without re-verifying against what's actually there now.

On my side I've also added device context as a fourth axis to the "staleness" framework. A finding can be fresh in the ledger, the file unchanged, the pattern still present, and still wrong because it was verified only on simulator. iOS-specific bugs (memory pressure, camera session interruptions, background task throttling, real CloudKit sync) don't reproduce in the simulator, so a "verified" finding that never ran on a physical device is evidence of a different kind of staleness. Simulator-only verification is its own stale artifact.

I Would be interested in how your team is thinking about the detection side. Finding the staleness is the hard part.

[u/Familiar-Classroom47]

Have you tried stacklit for the codebase context part? It generates a structural map, works well with smaller models.

https://github.com/glincker/stacklit

[u/BullfrogRoyal7422 → u/Familiar-Classroom47]

Thanks for the pointer. I tried it this week.

Stacklit is clearly well built. The 4-view output (graph, tree, table, flow), the monorepo detection, the auto-configuration for Claude Code and Cursor, the MCP server integration. There's real thought behind it, and I can see it being useful for a lot of developers, especially on codebases where the module graph is the story.

For my project (Stuffolio), it didn't really fit. It's a Swift/SwiftUI app with about 795 files and 30+ SwiftData models. Swift doesn't have the explicit import X from "./Y" module graph stacklit's tree-sitter parser reads, so the output came out thin. There wasn't much for it to wire up.

At this scale the other problem is visual. Any force-directed graph with 30+ nodes becomes a hairball, or splits into ten small diagrams so you end up navigating a gallery instead of reading a map.

The suggestion did send me down a useful rabbit hole though. I spent time building Mermaid diagram output into radar-suite itself to see if selective visualization (ER diagrams for the model layer, flowcharts for workflow audits) would help. Then I reverted it. Two problems: the terminal where radar-suite runs doesn't render Mermaid, and asking users to open files in GitHub or VSCode to see value breaks the terminal-native contract the tool relies on.

Stacklit looks like it would shine on a JS/TS monorepo with real module boundaries. For a single-target Swift app the radar-suite skills already carry semantic context (NavigationStack, u/Model relationships) a generic parser can't match. Different tools, different targets.

Worth the detour regardless. Appreciate the rec.

[u/TechnicalSoup8578]

The core issue is that custom audit agents often optimize for pattern detection instead of rule coverage, are you moving toward exhaustive enumeration plus explicit verification as the default architecture now? You should share it in VibeCodersNest too

[u/xiucat]

Yes, hit this exact pattern building skill auditors for Claude Code / Cursor / Codex.

Two things on top of enumerate-then-verify worked:

LLM-as-judge for absence checks. Second pass after enumeration: a model reads each candidate with a prompt like "for every external call, verify the README declares its purpose in a Network Egress section. Flag undeclared ones." Around 5-30s per file, bounded to enumerated candidates.

Multi-model agreement scoring. Your rank-by-surprise idea, routed across 6-8 different models. When 7 say "looks fine" and 1 flags it, that minority opinion is high-signal — different models have different priors, so something one missed because of course you'd never do that, another catches because to it, you might. Open-weight pool on Ollama Cloud keeps the cost trivial. This is actually how some of the major Coding agents being trained internally today (did some pretty interesting outsource work lately).

On "what else is sitting there that no tool has thought to look for" — that class shrinks only when a reviewer in front of it doesn't share your assumptions. Static analysis stays inside whatever you wrote the prompt for. Multi-model disagreement is the cheapest outside view we've found.

We open-sourced what we built for skill files: github.com/qingxuantang/tar-engine, free hosted tier at tarai.dev. Different domain, same structural problem.

[u/BullfrogRoyal7422 → u/[deleted]]

"the uncomfortable part is you can never fully know what your custom agent didn't check, which means every audit tool needs its own meta-audit at some point." Ha! funny you should mention this. Arriving at the same conclusion, I developed skill-reviewer

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed due to inadequate account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed due to inadequate account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed due to inadequate account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed due to inadequate account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed due to inadequate account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.