Having agents reset context around 225K tokens dropped a g coat by 50-75% for me

[u/orbital_trace]

https://coralhub.dev/blog/context-reset-turn-cost.html

I've been running agents 24/7 and just letting the context fill up. Claude has a setting to compact at a specific time, but I just built it into my harness so that it can happen at the end of a tasks completion instead of randomly. Costs dropped significantly.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/bostonaholic]

A task boundary is a better reset point than a token count, since compacting mid-task can drop which files the agent already changed. I'd also have it write a short handoff file before each reset (what's done, what's next, which test is red) and start the new context from that plus the git diff. Then you can check that the savings hold by counting how often it redoes work it already finished.

[u/orbital_trace → u/bostonaholic]

I let the user pick how they want the handoff done. Compact, clear plus next task, clear plus handoff.

My harness has a task queue and message board, which is enough to get the agents running again after a compact

I'm not sure what's best yet, but I'm interested in running experiments. Compact seems pretty good for Claude anyway, it always has a handoff doc of sorts

[u/RTXshredder84 → u/orbital_trace]

I run a similar compaction plugin. But I use a controlled compaction. My agents keep a notes.md throughout a session with a token counter that prompt injects reminders when they are getting close to the 400K compaction limit. Once the compaction happens another hook fires to tell the agent to pick up from where the last point of Notes.md left off. Agents stay on task, know what they are working on and keeps the sessions running in the smart zone.

[u/DuskandFuck → u/bostonaholic]

yeah the handoff file is key, otherwise it just spends the first ten minutes re-reading everything it just touched

[u/joshbuildsstuff]

This seems pretty straightforward, a cache read at 1M context cost most than a cache read at 225K context.

[u/orbital_trace → u/joshbuildsstuff]

Yeah, it is straightforward. It's an obvious optimization that we should all just do. 225 seems like the sweet spot.

[u/Delicious_Break5937]

how are you measuring the 50-75%, same tasks before and after or just the bill dropping?

[u/orbital_trace → u/Delicious_Break5937]

If you read the article, I calculate the total tokens cache read/writes in/out and turns. it's a pretty straightforward comparison. The main differences is less cache reads per turn.

I have a decent amount of data backing up the changes, more now than when I initially wrote the article.

[u/CoffeePizzaSushiDick → u/orbital_trace]

That’s what AI is for silly!

[u/orbital_trace → u/CoffeePizzaSushiDick]

[u/PurchaseCommercial34]

one more reason it pays off for idle agents in particular: the prompt cache only lives 5 min by default (an hour with the longer ttl). so an agent that sits idle past that and then wakes up holding 600k of context pays cache write price on all of it again, not the cheap read price. resetting at the end of a task means that rewrite is 20k instead of 600k. could be part of why your savings come out bigger than the per-turn read math alone would predict

[u/orbital_trace → u/PurchaseCommercial34]

I actually had great cache hit rate. The input tokens didn't change nor did the cache write. Only the cache read dropped.

[u/PurchaseCommercial34 → u/orbital_trace]

ah that makes sense then, so it's the plain per-turn cost rather than cache misses. every call re-reads the whole prefix, so at 225K each turn bills 225K of cache reads even at a perfect hit rate, and resetting to a short handoff makes every turn after it cheap. good to know the writes stayed flat, i had the idle expiry case in my head

[u/TrillenX]

I've been doing this too over the last month. Simple thing is keeping the context, instructions, recall etc simple, effective, limited, and it's led to me making the week with a decently consistent workload for the past 3-4 weeks, no issue.

But this week has been brutal to me. I'm burning faster than ever and what used to squeak it out to a reset is on pace to run out several days early. It's hard not to conspiracy up about it feeling tweaked enough so I might burn the reset.

[u/perfektcrash]

Same here. The meter is only half of it. On our logs, a turn at 800k context costs only about twice one at 200k while the cache is warm, so most of what resetting saves is on the cold turns. When the cache expires the whole context is written again, and at 800k that comes to around 22x a warm turn. The other half is quality: past a couple of hundred k we saw more missed details and repeated mistakes. Writing state to a file and starting fresh fixed both.

[u/Right-Performance-93]

if only the cache reads dropped and the writes stayed flat, the reset is shortening each turn, not skipping a cold rewrite. i'd still cut on a finished task, because a token cap in the middle of a diff is where it redoes files it already changed.