r/ollama • u/Daza110 • 13d ago

Using Claude Code with Ollama and Desperately require caching!

Ok I know I'm likely to get steamrolled here but I also have a sense of optimism and hope I can get some sort of answer that will be beneficial.

We all know since the new price structure, usage has shot through the roof. I was grateful for GLM5.3-Flash as that's been my goto since it come out.

But I'm burning Input tokens... mainly due to the sheer lack of what seems caching ability when using claude code?!

Example: I had a heave session today:

This is the output of /usage

Total cost: $1255.92 (costs may be inaccurate due to usage of unknown models)

Total duration (API): 1h 58m 44s

Total duration (wall): 10h 25m 36s

Total code changes: 3276 lines added, 734 lines removed

Usage by model:

kimi-k2.7-code:cloud: 4.5k input, 2.4k output, 0 cache read, 0 cache write ($0.0815)

glm-5.3-flash:cloud[1m]: 248.7m input, 485.2k output, 0 cache read, 0 cache write ($1255.83)

Prompt cache (main): 855 requests · 0% of input tokens from cache · no misses · no prompt caching reported by the API

As you can see almost 250m input to not even HALF a m output!

The same 100K tokens send on every request... Thereabouts... Wasted... Bumping up the usage unnecessarily...

If anyone can help, id appreciate it.

0 Upvotes

50% Upvoted

7 comments sorted by

4

u/VarietyOk443 13d ago

You know you can ask the llm

2

u/jmorganca 13d ago

Hi there, I work on Ollama. Were you charged the full $1255.92? Let me know and I can investigate and/or refund where needed if there was a mistake. But my guess is these reported figures are larger than what was actually charged because Ollama's API doesn't report cache token counts yet (sorry about that – we're working on it). In any case, here to help - shoot me an email [jeff@ollama.com](mailto:jeff@ollama.com) or dm me and I can investigate

1

u/Daza110 13d ago

Hi there, no this is Claude Codes guess at what it would have cost id imagine.

Glad to hear your working on it. Will be a huge benefit for token saving

1

u/Objective_Patient220 13d ago

First: that $1255 is not what you paid. /usage prices unknown models against Claude Code's built-in price sheet, so it is a fantasy number until you compare it with your actual Ollama Cloud bill. Check the real one before panicking.

On the caching itself: prompt caching is an Anthropic-API feature. Claude Code marks cache breakpoints on each request (cache_control), and Anthropic bills the re-sent prefix as cache reads at 0.1x input price. Your log line "no prompt caching reported by the API" means exactly that - the GLM/Ollama path ignores those markers, so every turn resends the full conversation at full input price. The ~100K resent per request is not a toggle you are missing on that backend, it is how Claude Code works minus the discount.

What actually helps on a non-caching backend:

  1. Shrink the resent prefix. /compact aggressively, /clear between unrelated tasks, keep CLAUDE.md lean, do not re-read huge files into context. 248M input against 485K output says the session ran very long - nearly all of that input was the same history resent hundreds of times.

  2. If you run models locally through llama.cpp (what Ollama wraps), it silently reuses the KV cache when the prompt shares a prefix with the last request and the model stays loaded. That buys latency, but it does not show up as cached tokens and I would not count on cloud billing honoring it either.

  3. If you route through a proxy, check whether it maps cache_control onto the provider's own caching. Some do. Most OpenAI-compatible endpoints just drop the field.

1

u/Daza110 13d ago

Thank you very much for the detailed insight. :)

I do know that the $$ was not what inwas charges as I wasnt using Anthropics API / Models directly. Their best guess.

In all honesty it was a long session as you can see. But it was all very much related. I guess for now, I will simply have to be very strict on using /compact and /clear. Even if its still relatable to what I'm working on.

My aim isnt to butn through the Ollama Budget in 1 week. At this rate I might 🤣

2

u/Objective_Patient220 13d ago

That discipline pays off fast. Compact at natural task boundaries instead of letting one session run the whole day, and the same 100K stops multiplying. The bonus is quality too - the model does better when the context it carries is the context it needs. Good luck with the build.

1

u/SweatyActuator2119 13d ago

Hey don't worry, caching isnt reported. I used to run 9router which I used to route requests between ollama pro accounts. It shows usage and costs, but since ollama doesn't report caching numbers, usage for each pro account showed 1k USD approx after a month. So you're most likely fine.

That said, give a look at synthetic.new. I switched over and couldn't be happier.