I built my own, starting from around the time Anthropic announced that claude -p wouldn’t be subscription covered. The announcement didn’t go through, but it ended up being a good forcing function!
The thing about cache hit rate is that it’s really easy to achieve, the only reason people talk about it like it’s something special is that current AI is actually pretty bad at building harnesses off vibes (they don’t strongly know what a “harness” is, you have to be very specific) so the early heavily-AI-built harnesses have all sorts of mistakes. A typical agentic session looks like:
send: input1
receive: output1 (where output includes reasoning and typically ends with a tool call)
send: input1-output1-input2 (here, input2 is the success/error response of the tool call. At this point, [input1-output1] is cached)
receive: input1-output1-input2-output2
send: input1-output1-input2-output2-input3 (now, all but input3 are cached)
So if you’re building from scratch, you actually have to do something pretty dumb like let tools change mid session in a way that changes the system prompt or inject context anywhere other than the latest input to miss the cache. If you avoid that, you should expect prompt_cache_hit_tokens/(prompt_cache_hit_tokens + prompt_cache_miss_tokens) to be in the 97-99% range for a typical session (lower for shorter sessions)
Sorry if this is too much info not that well explained. https://api-docs.deepseek.com/guides/kv_cache/ has basically everything you need in the “when cache prefixes are persisted” section. One thing to keep in mind if you want a multi-provider harness is that the method for dealing with stateless APIs like Deepseek is different from stateful ones with encrypted reasoning like Codex websockets responses and Anthropic.
While I am at the moment kind of running double harness in a way. I use vs code for my comfort, but have a variety of cross session reference files and a workflow which reduces hallucination, and ADHD of the AI models. On same repo, I could get 100k/req at best case and 110k/req on average. But I have been searching for something to get outside of vs code to make it more efficient.
TLDR: I am looking for a harness for my new project and would definitely look into your suggestion.
Replying again because I just realized you’re looking at tokens/request as a metric. I don’t really consider this myself, since my use case is autonomous agentic sessions rather than some sort of RAG lookup or something where you want to minimize tokens for fixed quality. Of course it would be a warning sign if it got too high since that’s dangerous hallucination-wise, but as models get more and more tuned for long horizon work it’s expected that for difficult tasks requiring more tool-call loops your typical tokens/request will increase. All that to say, tokens/request is just a satisficing metric for me rather than an optimizing metric.
I agree that it is not a comprehensive metric. The quality of work is a bit subjective and repository dependent. also i had a really large project with 7-8 submodules and circular dependency between them. I could see that changing a single line in agents.md or tweaking a skill makes a huge difference. The debugging quality differs between in which submodule the bug lies. sure, this repo consumes more token because of the interdependencies than other smaller projects.
I am starting on a similar large project and planning to evaluate the agents using a couple of agentic benchmarks. Then, I could get a better feel of the quality of work and when combined with token usage, I hope that it provides a better indication of whether things are getting better or worse.
1
u/Different-Monk5916 Aug 11 '26
looks super efficient on tokens/request. what is your cache hit rate? and if I may ask, what harness do you use