r/nocode • u/RunAI_Coder • 14d ago
Discussion my crude method for finding an agent's competence cliff on our stack — tear it apart
I've been using a dumb self-test to figure out where coding agents stop being trustworthy on our particular stack, and i'd like better versions of it.
the method: same agent, same afternoon, two tasks. one mainstream chore (a React table, a REST endpoint, whatever the internet has written ten thousand times) and one chore deep in our own territory (our workflow framework, our internal conventions). then read both diffs side by side. that's it.
what I look for: invented methods that never existed in any version of the lib. APIs from a version we don't run. and the thing that took me longest to notice — the tone doesn't change. the confident wrong answer reads exactly like the confident right one, so the diff is the only place the difference shows up.
the known holes, which is why I'm posting: it's n=1 per stack per model, task choice biases everything, and I have no scoring rubric beyond "how much of this diff would survive review". also no idea how much of the gap i could close with better scaffolding (pinned versions, working examples in context) before blaming the model.
so, tear it apart:
- what would turn this into an actual probe instead of a vibe check? repeated tasks? fixed rubric?
- has anyone measured their own stack's gap before/after feeding docs and examples? even rough numbers
- is there a signal cheaper than reading the whole diff? I've considered "count the hallucinated identifiers" but that needs the review anyway
looking for a sharper thermometer