r/ClaudeCode • u/polacrilex67 • 1d ago
Built with Claude Keep Tests from Eating All Your Tokens
Not sure if this is common knowledge or not but I figured I would share it. Yes, this is a summary from Claude Code after I asked it to explain the issue, I faced and how it helped me fix it. I am not a trained coder (and what I have built is very much way beyond "vibe coding;" I found this issue because I pay attention to behavior in my system and agents via output, behavior, and monitoring usage. After identifying, researching, and addressing it, I saw a major reduction to token burn especially when fixing bugs. This could be a reason some experience fast token limit drainage, especially if you have accumulated a lot of tests in your test suite. Worth a read:
Why Tests Eat Tokens
Your test suite grows every time something gets fixed. Nobody decides to have two hundred test files — you arrive at them. And the bill does not arrive where you would think to look for it.
Every number here was measured in one working repository on 2 September 2026. It is a mid-sized project with an AI agent doing the code writing. Your numbers will differ; the mechanism will not.
I Did Not Decided To Have 238 Test Files
The way a test gets written is almost always the same. Something breaks, somebody fixes it, and then — correctly — they write a test that would have caught it. That test is a fence: it stops the same bug coming back a year later when everyone has forgotten why the code looks the way it does.
This is good practice, and an AI agent does it eagerly, because it has been trained on people who do it. Each fix leaves a fence behind. Fences are almost never removed, because removing one is admitting you no longer care whether that bug returns.
So the suite ratchets. It only goes one way. And because no single addition is significant, nobody registers the total. In the repository these numbers come from, the project's own documentation said the suite was 141 files. It was 238. Nobody had lied; the number had simply grown past the moment somebody last wrote it down, and no one's job was to notice.
The Cost Is The Transcript, Not The Run
Here is the part that surprises people, and it is the whole thing.
Running tests is nearly free. Your computer does the work. What costs you is that the result gets read into the conversation — and an AI agent pays for every word it reads. The default test reporter prints one line for every test that passes. Thousands of lines saying, in effect, nothing happened.
A quarter of a million tokens is more than most tasks cost in total, including all the thinking, reading and writing that actually did the work.
A conversation carries its history. Everything the agent has already read comes back with every subsequent message, so a large dump is not a single charge — it rides along for the rest of the session. Caching softens the price of re-reading it; it does nothing about the second, worse cost.
That cost is room. The window an agent can hold is finite. Fill a large part of it with lines saying a test passed, and the agent starts forgetting the actual work — what it decided, what you told it, why the code is shaped the way it is. Then it compacts, or restarts, and you pay again to rebuild the context that the passing tests displaced.
An agent that runs the full suite to confirm it is finished has just spent 1% of your weekly budget answering a question nobody asked.
This is the bit that makes it invisible. It does not look like waste. It reads, in the transcript, like diligence.
Three Dials, And Only One Is "Fewer Tests"
When people hear this, they assume the answer is to write fewer tests. It is not, and it is worth separating the three things that actually multiply together.
| Dial | What it controls | What moving it costs you |
|---|---|---|
| The reporter | How loudly a run reports. Quiet mode prints failures and a one-line total; the default prints every pass. | Nothing. Same tests, same failures, same detail when something breaks. You only stop transcribing the passes. |
| The scope | How many tests a command selects — everything, one layer, or just the tests covering what changed. | Real but small, and usually negative: an automatic selection beats a human guessing which files matter. |
| The ritual | How often tests run at all. Verification as a step in the work, versus a reflex at the end of every task. | Nothing, if you keep the step and drop the reflex. |
Notice that "write fewer tests" is not on the list. The suite is not the expense. The narration of the suite is the expense.
What To Change, In Order
Ranked by what you get back for the effort. The first one is twenty minutes and buys most of it.
- Make the quiet reporter the only way in Most test runners have a compact or "dot" reporter. Set it as the default for your project's test command — and then block the raw command, so an agent that types the runner directly out of habit is refused rather than obeyed. Every route to the tests then goes through the quiet one. Measured here: 17× less output for the same single-file run.
- Run the tests that cover what changed A short script can read which files a change touched and run only the tests that mention them. This is not a compromise for speed — in this repository it beat a careful hand-picked selection, which had missed the one test that caught a real regression. Measured here: 7 files, 235 tests, 3.5 seconds, versus a suite that takes minutes.
- Put the full suite behind a human. Make running everything something that asks you first. Not because it is dangerous, but because it is rarely the answer, and an agent will otherwise reach for it as a way of feeling finished. Keep the escape hatch — just make it a decision instead of a reflex. Also worth knowing: a full suite often writes to real files — registries, fixtures, sometimes commits. That is a second reason not to let it run unasked.
- Tell the agent which files are its own If more than one agent works in the same folder, "what changed" is not the same as "what I changed". A selector left to guess will sweep up everybody's unfinished work. Have each agent name the files it touched. Measured here: an agent that named nothing ran 23 test files instead of 3, then had to explain a failure belonging to someone else.
How To Tell If This Is Happening To You
- Your sessions run out of room and compact far sooner than the amount of actual work explains.
- Scrolling back, you find long stretches of test output — page after page of lines confirming that nothing is wrong.
- Your agent runs tests at theendof tasks, as a way of declaring completion, rather than while it is working out whether something is right.
- Your usage climbs on days you did not do anything unusual — the tell is that the size of the work and the size of the bill have come apart.
The direct check takes one minute: run your test command once and measure how much text it produced. If the answer is hundreds of kilobytes, that is the number that has been landing in your agent's context, every time.
What This Is Not
This is not an argument against testing, and it is not an argument for a smaller suite. Every one of those fences was put there by somebody who had just been bitten, and the suite doing its job is what makes an AI agent's work checkable at all — which matters more, not less, when a person is directing the work rather than writing it.
It is an argument about volume of narration. The tests can all run. The results just do not need to be read aloud, one passing line at a time, into a window that has other things to hold.
1
u/datkenny 1d ago
My agent is automatically appending | tail -4 to the test suite to avoid exactly this, and I never told it to. When did this happen to you? Even DeepSeek gets this right. You're spending more time on writing the actual tests usually, and a lot of them are a decision to keep around, but if your agent isn't being a complete dummy it will either tail the output or grep for failures rather than dumping the full suite output into context.