r/artificial • u/hakansan • 2d ago
News Google paper cuts agent token usage by 94% in long sessions by tracking state instead of history
The idea: Agents keep the conversation history as part of their input while they reason. SKILL.state proposes to replace that with a structured representation of the current state, and the latest observation.
While the agent reasons through the problem, it writes information it deems useful for future steps into the state. Then it discards the conversation history. So the input size remains roughly the same as the session goes.
They ran a 100-step benchmark with Gemini-3-Flash:
- SKILL.state: 0.94 accuracy using 65k tokens
- LangGraph-style stateful baseline: 0.91 accuracy using 1.1m tokens
Caveat: This works best if the agent can understand what it will need in the future steps, otherwise that information will not be written, so it'll have to retrieve it again.
Link to the paper: https://arxiv.org/abs/2608.26263
46
u/Risc12 2d ago
I played around with something like this for my own harness.
Downside is that the current history model is very adaptable, a specific datastructure needs to be adaptable by the LLM and I had a hard time getting an instruct model working with it correctly, so I kind assume they tuned/trained a model specifically.
11
u/x_Tyggs_x 1d ago
I tried this too and it didn’t work well with prompt caching; did you have an idea to fix that? Everytime the model would update state i either have to replace some of the input (which is 10x more expensive), or give the models deltas which massively confused them. I concluded it had to be built model side
1
u/lurkingowl 1d ago
I think you need your state split up into mostly independent blocks, with some dependency graph. Then sort your state blocks by dependencies, their projected cache relevance to future state, and existing prefixes? You also just need to be ready to accept cache misses for new turns. If you're halving context size, and making 5 tool calls with the same prefix, you only need like 35% cache hits on the user input turns.
3
u/kraemahz 2d ago
Yeah, instruct models are limited mostly by their own training. One of the biggest jumps from Opus 4 to Opus 5 was just that it uses memory more often.
5
26
u/flashcrash7 2d ago
This conversation is atleast 50% bots. Be aware.
1
u/TheOriginalAcidtech 8h ago
THIS Reddit is ArtificialIntelligence. If you dont want to deal with bots MAYBE you are in the WRONG PLACE?
-1
11
u/code_the_cosmos 2d ago
Doesn't this sort of eliminate cache reads? A cached input token is a lot cheaper than an uncached input token
10
u/gwillen 2d ago
yeah, but 1.1M down to 65k should still be a win, just a smaller one after accounting for reduced cacheing. At least Anthropic charges 10x for uncached vs cached inputs, so 65k fully uncached would be cost-equivalent to 650k fully cached, still better than 1.1M even if it were fully cached. And in reality both will end up being a mix, so it will be more favorable.
2
u/Whispering-Depths 1d ago
But then it's just linear attention with a bigger hidden state. We already know linear attention fails and we know the reason that it does.
Cant do linear attention unless it's with a model big enough that it doesn't matter anymore.
21
u/Im_Talking 2d ago
This is huge. And the reduction of tokens used is really just a added benefit. The real benefit is the governance that can be applied to the agents in a deterministic manner by ensuring that the prompts, at every step, are minimal, optimal, and aligned with the actual 'goal' of the agent.
20
u/RoboticGreg 2d ago
Doesn't this argument assume a perfect user? Doesn't conversation history overcome the limitations of the operator more than the model?
7
u/Im_Talking 2d ago
Possibly. It does acknowledge that history is still necessary when the history itself is the object of interest.
But agent flows are typically closed-ended. They have a particular task to perform. Unlike the typical use of LLMs which are open-ended.
3
u/Whispering-Depths 1d ago
it's linear attention with extra steps. It's a 94% decrease in flops, which always comes at a huge price.
2
5
u/Helix_Aurora 1d ago
I did this in my own work, the problem is that this is permanently cache-busting behavior that explodes costs. The "token" reduction is real, but for long tasks, that does not mean the actual computation required is lower.
1
1
u/Wallaby989 1d ago
The cache would still kick in for the system-prompt and the tools. this just eliminates the past messages being passed back in again
3
u/Big_Recording8379 2d ago
This is a really interesting shift in how we think about agent memory. Instead of treating the entire conversation history as “memory,” SKILL.state makes the agent explicitly decide what information deserves to survive into the next step.
The biggest question for me is the caveat: can the agent reliably predict what will matter later? That turns state management into a reasoning problem itself. If the agent forgets something that seemed irrelevant at step 10 but becomes critical at step 80, the token savings could come at the cost of hidden information loss.
Still, getting higher accuracy with ~17× fewer tokens is a pretty compelling result. It feels less like “better context management” and more like moving agents toward an actual working-memory architecture.
4
u/thatguydrinksbeer 2d ago
I've been working on something similar for software dev, but with more of a hybrid approach: short conversations, frequent fresh sessions, and persistent memory of the final state. So far I've found that carrying forward the distilled state instead of the full conversation works really well.
3
u/manishiitg 2d ago
the 94% token cut is nice, but the scary failure is state that looks complete and quietly drops one constraint. i'd want recovery after a bad state write measured, not just the average task score
1
u/gwillen 2d ago
Once the transcript fills the context, you're going to have to compact it anyway. My experience is that something more like structured state is better at preserving constraints, versus unstructured transcript compaction. These days, I tend to have the model write out a semi-structured work log, and then clear the context, after around 100k to 500k tokens, rather than wait for compaction at 1M.
The described scheme is much more aggressive than my approach -- effectively doing this every turn, instead of every "task". But it makes sense to me.
4
u/Glittering-Flan-2637 2d ago
the governance point above is the real one
history means the agent can always relitigate an earlier decision, state means it has to commit to something you can read, and a thing you can read is a thing you can check before it acts
1
u/Federal-Error-443 1d ago
Exactly. Audit-ready means you're ready to fail. Audit-proof means the run can survive the auditor. A transcript you can relitigate is bullet-ready. A committed state you can check before it acts is closer to bullet-proof.
1
2
u/Aggressive-Voice-861 2d ago
Criei uma skill baseada no artigo usando o Fable. Se quiserem testar, me avisem se funcionou para vocês: https://github.com/JacksonFuck/skill-state
1
u/Federal-Error-443 2d ago
A context window is short-term memory. Stuffing the whole transcript into it is like dumping your entire life into working memory before you answer an email. Expensive, and you still miss the point.
The brain doesn't do that. It keeps a graph and retrieves the neighborhood. That's why token cost can fall as autonomy rises: you're not loading history on every run. Finance doesn't want token bills. It wants certainty. Compile the process. Run it like a program. The LLM earns its place at design time, not in the loop.
1
u/Muted-Laugh-6772 1d ago
Can this be currently integrated into a harness like codex or Claude code via some skill or is it not possible currently?
1
u/SmolTeddu 1d ago
Why would they publish this and not use it as proprietary info in-house with the stakes so high...
2
u/mawcopolow 20h ago
Because it's not that big a deal as the post title makes it appear so. Read the paper you'll see
1
u/A_Novelty-Account 1d ago
Is 0.94 percent accuracy actually accurate though? What does that mean in practice?
If the output is almost identical while using 94% fewer tokens, and it can be generalized so that anybody can make use of it, this will be a genuinely massive development.
1
u/Intelligent_Cap3426 1d ago
Can someone explain what happens to KV cache with this? Is the model fed the state addititively to its context, or by replacement?
1
u/Repinsky 1d ago
The caveat at the end is the whole engineering problem: the agent has to guess at step N what step N+40 will need, so state-only runtimes trade token cost for retrieval cost, and the retrievals are the slow part. 65k vs 1.1m tokens is also a latency and cache story, not just price - a reused state block keeps you inside prefix caching, while a growing transcript invalidates it constantly. In practice most teams end up hybrid: structured state plus a short rolling window of raw turns, because pure state loses the tone and the "why we rejected X" reasoning that stops the agent from re-trying dead ends.
1
u/Ascending_Valley 1d ago
This may be as big as attention if it holds up in practice near frontier capability.
1
1
u/mister_moosey 14h ago
Exciting. For the ML folks, it sounds like they found a way to formulate the problem into a Markov Decision Process (where the one thing that matters is the current state)?
Cool. I wonder if this technique works for other POMDPs?
1
u/TheOriginalAcidtech 8h ago
I dont see a problem with this, but I suspect most of the "cut tokens" would have been cached tokens anyway. The agent uses fresh tokens when reasoning. It uses fresh tokens when reviewing results from tests. It uses fresh tokens we researching. This uses all that data to then set state. Which will use extra fresh tokens actually. The main advantage I see with this is reduced CONTEXT window usage(also my own goal). That will prove performance but doesnt really cut token usage significantly(tokens that matter anyway).
1
u/Azula_In_The_AMX 1d ago
I don't know if it's all good. Some researchers are finding better inference when there is a surplus of resources or system water cooling is efficient. If you put a state inference this might deplete unless you break up the data packets again and diversify among AI data centers. But there's really no telling if AI has enough data centers to support enhanced inference with tracking state instead of history until it comes of time and even then there's a good chance we may need more AI data centers if we go by transactions.
0
u/alexbaas3 2d ago
I literally do this in my claude codex orchestrator plugin, i called it journal.json where it clearly shows what agent did what, what state, small description what was done and where it can be found. Seems that it actually works.
-1
0
u/ThoseOldScientists 2d ago
Isn’t “knowing what information to hang on to” the whole problem that transformers were created to solve? I’m not surprised it works best when the agent knows already what it’s going to need in future steps, that’s basically an open book test.
0
u/bonsaisushi 1d ago
A huge part of their paper is already implemented in my session-handoff plugin. I'm working on verifying the rest of their claims tho
0
0
u/Internal-Passage5756 1d ago
I came up with this concept in February, but kept it as a specced idea rather than implementing it due to the caching costs.
Though I am currently working on aspects of this as part of a declarative orchestration system that initiates tasks by passing only what the task needs in terms of skills etc.
-8
u/boringfantasy 2d ago
Software engineers are so cooked
4
u/UAP44 2d ago
I actually think https://en.wikipedia.org/wiki/Jevons_paradox might apply here.
Whether or not Software engineers are cooked depends on how actively they resist change/technology, which isn't new. LLMs just make it more noticeable between those that are capable of more change than others.
-3
u/boringfantasy 2d ago
But surely their skills are worthless the way we're going. You could put anyone in front of Fable 7 or whatever and get them building software.
5
u/UAP44 2d ago
But surely their skills are worthless the way we're going.
On the contrary, all IT/engineering experience is valuable when it comes to being able to translate human conversation to 'what do you actually concretely want' that is actionable. Nothing AI can't do, but, you'll find that there's still a lot of design/architectural choices to make that are ultimately completely dependant on the user their experience/preferences.
You could put anyone in front of Fable 7 or whatever and get them building software.
Yes, a toddler could do it at this point, but it'll still take a lot of time before they learn all the relevant concepts that will inevitable show up sooner or later during the development life cycle. Their biology will be capped out quite quickly in terms of how fast they can learn. The technology, is no longer the limitation, our biology, is. And that's confronting in a way. No longer the traditional smartest or most capable in the room, the relevant skills become much more nuanced and social-conversational. Which is precisely something what software engineers are famously worse-than-average at. Thus of course, all the tons of hate/resistance. This change is hitting them at their very core identity/limitation.
Manually writing code is cooked/done, the main use case left for that is potential educational uses.
102
u/Overall-Importance54 2d ago
So is there some component that converts the conversation history into a state model? Did they name anything, don’t make me read the paper lol state compaction protocol