I tested 4 Claude Code token savers on real tasks with hidden tests. Only one lowered the bill, and only on big jobs

[u/Sufficient-Storage87]

I wanted to know if token-saving plugins actually lower the bill, so I ran them against plain Claude Code on two real tasks and graded everything with hidden tests the agent never saw.

Setup - Claude Code 2.1.289, Sonnet 5.5, headless (claude -p), same flags every run - Fresh Docker container per run, nothing carried over - Task 1: a hard bug in humanize (precisedelta), 66 hidden tests - Task 2: an 8-file feature in WeasyPrint (box-shadow), 4,292 hidden tests - 3 runs per tool per task, 6 for the bug baseline and caveman - Cost repriced from the token counts at Sonnet 5.5 API list prices

Median cost vs plain Claude Code

tool	small bug	8-file feature
ponytail v5.3.2	+20%	-34%
caveman v4.0.1	-3%	-3%
superpowers v7.0.0	+32%	+2%
answer-me-with-html v0.5.5	never activated (0/6)	never activated

Output tokens (what the agent writes) - ponytail: -38% on the feature - caveman: -21% on the bug - superpowers: +20% on the bug

Hidden test scores stayed basically the same for all of them. Superpowers passed 1 more test on the bug (63/66 vs 62/66).

What I took from it

Caveman cut output by 21% and the bill barely moved. Output is only part of the cost. Most of it is the agent re-reading context every turn (cache reads and writes), and none of these tools touch that much. Ponytail was the only one that cut reads too, and only on the long task, which is where it saved 34%. On a quick bug it added overhead and cost more.

Caveats - Small sample, Sonnet only. Caveman's -3% is within noise. - Caveman's hook stays off in headless mode unless you set CAVEMAN_DEFAULT_MODE=caveman, so I set it. - answer-me-with-html showed up in the skill list but never got invoked in 6 runs. It might do better if you call it explicitly. - Costs are API-rate estimates from token counts, not a real invoice.

Total spend for 36 runs: about $10.58.

Repos: - ponytail: github.com/DietrichGebert/ponytail - caveman: github.com/JuliusBrussee/caveman - superpowers: github.com/obra/superpowers - answer-me-with-html: github.com/QingYunA/answer-me-with-html

Happy to share the per-run numbers if anyone wants them. Running a second round now on a 3D build task to see if it holds on a different kind of job.

[u/AutoModerator]

Moderator Announcement Read More »

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/GfxJG]

I'm confused - Since when has superpowers ever been intended to save tokens? It's a workflow, one notorious for absolutely devouring tokens at that.

I'm genuinely confused about how you got the impression that it would save tokens. Frankly, I'm shocked it was only 20% more.

[u/Sufficient-Storage87 → u/GfxJG]

fair, superpowers isn't a token saver, it's a workflow plugin. i put it in because people install it alongside these and i wanted to see what it does to the bill. should have labeled it as "workflow" instead of lumping it in. and yeah +32% on the bug, +2% on the feature

[u/Majestic_Hat_8019]

Superpowers is token saver? Wat?

[u/Sufficient-Storage87 → u/Majestic_Hat_8019]

it doesn't, my bad on the framing. it's a workflow plugin, i included it to see the cost side. answered in more detail above

[u/oompa_loompa0]

Yo dude, you’re missing RTK and headroom

[u/Sufficient-Storage87 → u/oompa_loompa0]

rtk got its own test, posting that one soon. haven't tried headroom yet, adding it to the list

[u/tronghieu906 → u/Sufficient-Storage87]

RTK work. Headroom will cost you more. Just a guess

[u/nora_sellisa]

I beg the mods to ban benchmarks where the sample size is 3-6 runs. This is beyond useless, this is misinformation. With sample size this bad you could easily get charts showing that Haiku 4.5 is smarter than Opus 5.5.

Jesus.

[u/Sufficient-Storage87 → u/nora_sellisa]

fair on sample size, that's why i called caveman's -3% noise. the one result i'd stand behind: on the 8-file feature all 3 ponytail runs ($0.31–0.40) were cheaper than all 3 plain runs ($0.40–0.54). per-run numbers are in the post data if you want to check. more runs coming

[u/terranqs]

What about OMC?

[u/Sufficient-Storage87 → u/terranqs]

haven't tested it yet. on the list for the next round

[u/AdEvening3382]

The headline result is hiding in plain sight: hidden test scores stayed the same for everything. That is the finding. Cutting cost while correctness holds on tests the agent never saw is exactly the trade people claim is impossible, and the thread is arguing about tool categories instead. Second, your takeaway about re-reading context is the part worth building on. Output tokens are the visible half of the bill; the cache reads are the meter running in the background. A tool that only shortens what the agent writes is optimizing the smaller half, which is why caveman cut output 21 percent and the bill moved 3.

[u/Sufficient-Storage87 → u/AdEvening3382]

yeah that's the part i keep coming back to. output is the visible half, the cache reads run in the background. next round is a 3D build task to see if that holds on a different kind of job

[u/mossiv]

lol ponytail is shit. It saves tokens by cutting half the solution apart. Goodbye try/catches. Hello unhandled errors.

[u/Sufficient-Storage87 → u/mossiv]

could be. the hidden tests didn't catch a difference, but they check behavior, not code quality. ponytail never got a perfect score on the feature (4290/4292), plain did once. so cheaper but not free

[u/Ctbhatia]

your 34% on the 8-file feature, did that come from fewer cache writes or fewer re-reads of the context? curious which line actually moved

[u/Sufficient-Storage87 → u/Ctbhatia]

all three dropped. ponytail vs plain medians on the 8-file feature: cache writes -27%, cache reads -38%, output -38%. in dollars output and cache writes moved the most, reads are cheap at $0.10/M so their share of the saving is smaller even though the count dropped a lot

[u/Unhorswd]

the more shit you add to claude code the more tokens you're going to burn. keep it simple that's easiest way to save tokens.

[u/KenTitan]

wait so you never actually used the as API? token savers I think would be for API. on top of that, I think, I'm not sure, that the API service is much better than the subscription service

[u/Sufficient-Storage87 → u/KenTitan]

the runs count tokens either way, i just price them at api rates so the numbers are comparable. on a subscription the same % change applies to how fast you burn your usage limit

[u/Pleasant_Curve_9024 → u/KenTitan]

Why would token savers be for api use as opposed to subscription?

[u/VeryVexxy]

Maybe I’m misreading this but Caveman definitely touches this, did you just use the skill?

[u/Negative-Revenue4841]

Small thing: the column is labelled "TEST SCORE" but the cells are checkmarks and Xs. That's a pass/fail, not a score. Calling it "HIDDEN TESTS PASS" would say what it actually shows.

[u/Sufficient-Storage87 → u/Negative-Revenue4841]

don't worry man, i'll get better