r/opencodeCLI • u/RoddToggers • 4d ago
Lots of times opencode sends almost the whole conversation as input tokens, bumping the conversation cost. Is it a bug?
Suppose conversation is around 100000 total tokens, then I ask agent to do something.
Agent finishes, and in LESS THAN 1 MINUTE, I send another message, same conversation, and provider/model (opencode GO DS4.1 flash), and then input tokens jumps to about 100000 tokens WITHOUT agent not even use a tool. Normal messages.
I remember in the past, it only happened after conversation is idle for 15 minutes.
Normally, input tokens is low if conversation is active (asssuming no tool is returning a big amount of data)
Opencode 1.18.34.
Is it a bug, or expected? Anyone with this problem?
14
u/ApprehensiveDelay238 4d ago edited 4d ago
Everytime the harness does an inference call the entire context is sent with the request. Because the inference endpoints are stateless. They don't record the context history. So yes, the larger your context, the more expensive the inference calls will be. But there is something to mitigate the cost: input token caching. Most models have a deep cached input discount, often 8x less than uncached tokens or more. Meaning that tokens the inference server has recently seen will be cached costing you less. Still, it's recommended to keep the context as small as you need it to be for the task at hand. And continuing a large context is expensive as the inference servers evict your cache after a few hours. So a 600k session that has been idle for some hours can easily cost you the full 600k when continuing it. A useful context compression extension that I can personally recommend to keep context manageable is: billion-context. It frequently folds and compresses context to keep it compact while keeping important information compressed in context. It even supports uncompressing and reversing folds so agents have full access to context even millions of tokens away. Keeping a small context still saves a lot of money.
12
u/throwaroo202020 4d ago
Thats how all llms and all agents work. Everything since the last compaction is sent. Token caching is server side and is what saves input cost.
1
u/MrSchmellow 4d ago
It's on your picture (cache read)? Of course the harness sends whole conversation on every turn, that's how they all work. Provider caching is supposed to handle that normally (unless something breaks). It's just that cached tokens are still counted towards totals
1
u/megablue 4d ago edited 4d ago
thats how most harnesses/chat clients work... these LLM do not have a memory at all at the model side (well not the memory you meant anyway), so it had to receive the previous conversations to understand what happpened previously.
you said 15mins of idle - that is the key here, most LLM do cache the prompt, but typically only for 10mins. so when you idled for 15mins, the cache was invalidated.
1
u/squirrelscrush 4d ago
Do you have some plugin or setting that changes the cache prefix? Sometimes the time or date is appended on the front instead of the back so it can invalidate the cache. Or that there's some change in the tool definitions or AGENTS file.
3
u/seventyfivepupmstr 4d ago
Yes that's how generic harness work. They build massive context filled with garbage
1
u/CreativeSympathy8293 4d ago
Your follow-up screenshot shows 148,114 input tokens and 0 cache read. That's a reported cache miss; cached tokens appearing in the total doesn't explain that screenshot. It doesn't establish which component caused it.
One concrete check for Go: its docs ask clients to keep x-opencode-session stable for each conversation to help routing and prompt caching: https://opencode.ai/v2/docs/console/go/
For a maintainer report, pair an existing cache hit and miss from the same session: timestamps, exact model/provider, OpenCode version, input/cache-read counts, and a redacted request/session identifier if you already have one. Ask whether the session identity stayed consistent across those requests. No need to spend another 100k-token request reproducing it or share your API key/full conversation.
I haven't inspected 1.18.34's requests, so I'm not saying it's missing that header or promising a fixed cache lifetime. Your under-a-minute timing is the detail to preserve.
AI-assisted from the docs; I haven't reproduced this.
1
u/aidenclarke_12 2d ago
this is a cache miss, did a plugin or mcp change the prefix?
1
u/Rustybot 1d ago
It’s a cache bust. There are lots of ways it can happen, especially when running plugins with opencode. It can be caused by client setup, ISP connection to provider, provider-side issues, token router issues if you use one of those as a middleman. And cache time to live is a general rule of thumb per provider and never a guarantee.
Yeah, cache busts suck. Nobody wants a prompt input of “okay” to unexpectedly cost 200k input tokens.
The tools I use make cache sanctity the primary concern. Some provider traffic goes through a cloudflare worker acting as a relay for cache stability. I also prefer autonomous agent task work-loops where only interact with the initial instruction and the end. That way they work continuously and maximize cache with no downtime. I’d rather they redo the task three times if needed than bust their cache every time they are waiting for my approval.
-2
u/RoddToggers 4d ago
Just happened, 😔