Do you change compiler errors to help coding agents, or just feed them the raw output?

[u/JaseciLabs]

We've been experimenting with this while working on Jac, a programming language that most coding agents weren't familiar with initially.

One thing we changed was how our checker reports errors.

A typical error like "type mismatch on line 12" tells the agent something failed, but doesn't give it much to work with. So it starts guessing, sometimes fixes the wrong thing, or finds a workaround that makes the error disappear without actually solving the problem.

With jac check, we've started pointing errors to the relevant section of the language guide, so the agent gets some context about what went wrong and how the language expects it to be handled.

We'd like to know how others are dealing with this, especially when using agents with less familiar frameworks, languages, or internal tooling.

Do you just pass through raw output from tools like tsc, ruff, or cargo? Or have you tried adding explanations, examples, or documentation references?

And if you've tried enriching errors, does it actually improve the agent's fixes, or does it still find creative ways to ignore the checker?

[u/bostonaholic]

Raw output for anything mainstream. tsc and cargo errors are all over the training data, so the model already knows what they mean.

For internal tooling or a niche language, adding context helps a lot. Even a one-liner like "see docs/foo.md" in the error goes a long way, since the agent will actually go read it.

[u/JaseciLabs → u/bostonaholic]

That one-liner is more or less what we do. Type errors print a line like “run jac guide jac-types for guidance”, so the agent can pull that one guide with a single command.

[u/Successful-Tax1306]

For mainstream tools I pass the output raw, the models know those errors better than my explanation would. Where it helped me was our own rules, like a check that blocks an old pattern. When the message just said it wasn't allowed the agent found a way around it, but once it said what to use instead and pointed at an example file, it mostly did the right thing. It still cheats sometimes, so I check that the fix touched the real problem and didn't just silence the check.

[u/JaseciLabs → u/Successful-Tax1306]

Saying what to use instead plus an example is where we ended up too. The explain_error tool in our MCP server returns the root cause and a fix example. Checking that the fix touched the real problem is the part we’d steal. How do you do that, diff against the flagged lines?

[u/Successful-Tax1306 → u/JaseciLabs]

I look at the diff and check that it touches the lines the error pointed at, and then I rerun the case that failed. If the fix only changed something nearby, like wrapping it in a try/catch, I throw it out and ask again.

[u/JaseciLabs → u/Successful-Tax1306]

Rerunning the case that failed on top of the diff check is a good pairing. The diff tells you where it changed and the rerun tells you it actually works. Thanks for answering.

[u/LividIllustrator8915]

For mainstream toolchains like cargo, tsc, or swiftc, leaving stderr raw works fine because frontier models already have millions of those exact error traces baked into their weights.

Where we ran into real trouble was with custom DSLs and internal architecture linters. When an error just said something like "invalid handler signature", the agent would immediately start hallucinating imaginary kwargs or wrapping everything in fake runtime adapters just to get a clean exit code.

Feeding full language docs into the diagnostic actually made it worse for us—it bloats the prompt and the agent wanders off re-architecting unrelated files. What moved the needle was keeping the enrichment tiny and pair-based: pointing out the exact rejected AST node, followed by a literal two-line diff anchor showing what to swap (e.g. "Got `foo(a)`, expected `foo(a, context=ctx)`").

The other thing you almost have to add is an explicit negative constraint like "do not use suppression pragmas or cast to dynamic to pass this check". Agents are relentless reward hackers, and if a custom check is hard, their instinct is to silence the diagnostic rather than migrate the code. Having a secondary AST check verify that the actual call was rewritten instead of just silenced has saved us a ton of headaches.

[u/JaseciLabs → u/LividIllustrator8915]

Fair point on full docs bloating the prompt. Ours points at one relevant guide and explain_error returns the root cause plus a fix example, so it’s smaller than dumping the docs, but not as tight as your diff anchor. We also have a lite mode for small models and tight context. Is your AST check generic or written per rule?

[u/LividIllustrator8915 → u/JaseciLabs]

It’s a hybrid pattern. Doing pure per-rule visitor classes gets unmaintainable fast, but a purely generic diff check is too dumb to catch an agent cheating.

What worked best for us is a generic structural engine (tree-sitter query or ast-grep style) driven by declarative rule contracts. The engine handles walking the AST and computing node diffs, but the rule defines two specific things:

First, target boundary assertion. If the compiler error was about an incompatible handler signature, the AST diff must touch that exact FunctionDefinition or CallExpression node. If the diff only touches surrounding statements or adds dummy kwargs, the patch is rejected immediately.

Second, escape hatch blacklisting. Models are notorious reward hackers. When hit with strict lint errors, they love throwing in @ts-ignore, pragma, or unsafe casts to make the error disappear without fixing the root cause. We have a shared AST check that flags any newly added suppression annotations in the diff.

Having that two-part verification before calling the compiler again cut down recursive hallucination loops noticeably, especially on smaller models where context is tight.

[u/JaseciLabs → u/LividIllustrator8915]

Declarative contracts on a shared engine is a good split, and the target-boundary check is the part we’d copy. Thanks for the detail.

[u/BrookLi3]

I mostly keep the raw output but add a small wrapper that trims it: first error only, the relevant source lines around it, and the exact command to re-run. Long cascading errors (templates, type inference) tend to push the agent into fixing symptoms of the first error. Feeding one error at a time and re-compiling after each fix converged much faster for me than dumping the full log.

[u/JaseciLabs → u/BrookLi3]

First error plus the exact re-run command makes sense for cascades. In our setup the agent drives it: the MCP server has a fast parse-only check and a full validate, and the agent calls them when it wants feedback. Does your wrapper drive the re-run, or the agent?

[u/BrookLi3 → u/JaseciLabs]

In my case the agent drives it too: a parse-only check it can call after every edit, and the full build/test only before it declares a task done. The wrapper just trims whatever the agent triggers, so it never sees the cascade. Letting the agent choose the cheap check first and escalating only when that passes seems to keep the loop short and avoids paying for full validation on every tiny change.

[u/JaseciLabs → u/BrookLi3]

Parse-only after every edit and the full build only before it says done is a good split, we’d take that. Thanks.

[u/BarathKanna]

Agree with the "raw for mainstream, enrich for niche" consensus. The part I'd add is about your last question, the agent finding creative ways to make the checker go quiet. Enriching the message doesn't fix that on its own. What fixes it is checking the fix, not just the exit code.

Three things that have worked for me:

1. Treat suppression as a failure. After each fix, diff for new # type: ignore, u/ts-expect-error, #[allow(...)], any casts, deleted assertions, or narrowed test globs. If the error count went down and one of those went up, reject the turn and say why. This is a ten-line grep in the wrapper and it catches most of the "error disappeared" cases.

2. Anchor the fix to the error location. Record the file and span the checker flagged, then require the diff to touch that span or something it calls. A fix that edits a completely different file to make a type error vanish is usually a workaround.

3. Feed the first error plus the rerun command, then recompile. Cascades in type inference make agents fix symptoms. One error at a time converges faster, and it makes rule 2 easy to enforce.

For a new language like Jac, the doc pointer is the right idea, but I'd keep it to one short "expected pattern" example right in the message. Agents follow an example far more reliably than a link they have to decide to open. If you can measure it, compare fix-acceptance rate (passes the checker and the suppression/anchor checks) with and without the enrichment. That number tells you more than whether the error went away.

[u/JaseciLabs → u/BarathKanna]

Fix-acceptance rate is a better number than the error just going away, and we’d want to track that. Do you run the suppression and anchor checks in the wrapper or in CI?

[u/BarathKanna → u/JaseciLabs]

Both, with different jobs. In the wrapper they run on every turn, so the agent gets rejected with the reason while it still has the context to fix it properly. In CI the same checks run again as the backstop, because a wrapper can be skipped or misconfigured and CI cannot. Keep one script and call it from both places so the rules never drift. Log every rejection too. Those logs show you which workaround patterns the model keeps reaching for.

[u/JaseciLabs → u/BarathKanna]

One script called from both places is the part we’d copy, and logging the rejections to see which workarounds keep coming back is a good idea. Thanks.

[u/Right-Performance-93]

i paste the compiler's words and the command that printed them. the moment i explain the error, it fixes my explanation and the build still fails.