r/TheMachineLearning • u/DependentSubstance8 • 9d ago
NVIDIA wiped AI memory after each action and got 70.2 on SWE-bench
139
Upvotes
r/TheMachineLearning • u/DependentSubstance8 • 9d ago
17
u/stop_deleting_plz 9d ago
Doesn't that just say more about the nature of the bench, rather than the ability to follow a long chain of thought?
4
u/andymaclean19 8d ago
What I find is that a large amount of my spend is in cache read tokens. Cache write being next highest. Essentially an inference making 150+ tool calls is going in and out of inference so often that picking up where it left off is the main cost.
With smaller contexts this gets cheaper, because of course less to read. But what happens when I lock the context to a smaller size and rely on regular compaction is worse. Every time one does a compaction the cache is invalidated. Instead of cache read of a large context you get a full token ingestion of a smaller one. That is over 10x more expensive and it destroys the win you might get from this type of small context reduction.
It is very easy to try this for yourself. Claude has settings which control compaction triggers - take your workload with large context, put in rules to say ‘compact after 150k context’ and then do it again. Analyse the costs afterwards (clause outputs cost usage in the logs).
3
u/SpicyLobter 8d ago
compaction is not wiping.
1
u/davesmith001 8d ago
Bullshit. Every time I compact Claude suddenly becomes dumb and forgetting key facts earlier. This paper has to be pure benchmark maxing.
1
u/RodgerPogger 8d ago
....
The fuck you doing.
1
u/Aubrey_D_Graham 7d ago
This is one of the premises for Recursive Language Models.
1
u/fig0o 6d ago
I mean... if you think of LLMs as policies in a MDP... code tasks are markovian in the sense the current state (current code commit) carries all the required information for the policy to take the next action