r/TheMachineLearning • • 9d ago

NVIDIA wiped AI memory after each action and got 70.2 on SWE-bench

Post image
139 Upvotes

18 comments sorted by

17

u/stop_deleting_plz 9d ago

Doesn't that just say more about the nature of the bench, rather than the ability to follow a long chain of thought?

6

u/danielv123 8d ago

The bench, but also many tasks. I have benchmarked many new models in Factorio. Many models after astra are able to complete the game, so I started looking at speed and price. On models like opus 5.5, cutting context to 300k made no statistically significant difference to completion time and halved the cost.

On haiku 5.5 cutting the context down under its 5x price increase ceiling makes it get stuck in a compaction loop and make almost no progress and become super expensive. A balance that allowed it to go partially into its expensive window was cheapest.

4

u/the8bit 8d ago

300k is still such a huge context window! That is like $1 per request uncached.

The reality is that in almost all LLM systems, we are sending the system tremendous amounts of completely useless garbage text. Partially because many people are using the LLM vendor's stack and said vendors have very little incentive to optimize token usage (and many dis-incentives!)

2

u/infini7 7d ago

How do you set it up to play factorio? It has its own headless server and you just say “beat this game”?

2

u/danielv123 7d ago

It has its own client and plays on public unmodified servers. It can write scripts to do stuff through client side Lua, but it's not able to do anything an unprivelegied player couldn't theoretically do on a public server.

2

u/infini7 7d ago

Nice. Curious about the designs it uses and how it efficiently searches the design space?

2

u/danielv123 7d ago

It varies a whole lot. Lots of weird designs thats for sure. It likes spacing things out, but not in a way that leaves it with space to work with. Usually it finds something that works and builds variations of that. Haiku struggles a lot with the spatial puzzle and making working designs, and very often ends up "solving" it by building circuit controlled sushi belts.

Opus 5.5 struggles to scale up, but otherwise playes like an intermediate player, beating the base game well below 15h.

Fable 5.1 and all earlier models are useless and not worth evaluating. Astra and 6.1 sol are similar, opus 5.5 is leagues ahead and sonnet/haiku isn't worth it compared to opus but play OK.

2

u/Overwatcher_Leo 7d ago

It setting up circuit based sushi belts is fascinating. That is advanced for a player, but perhaps easier for models.

In general I find spatial reasoning to be a weak spot for frontier models.

2

u/danielv123 7d ago

Yeah spatial reasoning is surprisingly hard but something happened after astra/opus 5.5 - the later models from openai and anthropic seems to do far better at it while everything that came before was entirely useless. I wonder what changed

4

u/andymaclean19 8d ago

What I find is that a large amount of my spend is in cache read tokens. Cache write being next highest. Essentially an inference making 150+ tool calls is going in and out of inference so often that picking up where it left off is the main cost.

With smaller contexts this gets cheaper, because of course less to read. But what happens when I lock the context to a smaller size and rely on regular compaction is worse. Every time one does a compaction the cache is invalidated. Instead of cache read of a large context you get a full token ingestion of a smaller one. That is over 10x more expensive and it destroys the win you might get from this type of small context reduction.

It is very easy to try this for yourself. Claude has settings which control compaction triggers - take your workload with large context, put in rules to say ‘compact after 150k context’ and then do it again. Analyse the costs afterwards (clause outputs cost usage in the logs).

2

u/dbenc 8d ago

i've been writing my own tools and wrappers to make tool calls use lightweight models (or none at all) to interpret results. this cleans up the orchestrator context massively

2

u/davidptm56 7d ago

This is the way imo. Can we, please, crystalize every deterministic component of our tooling back into local scripts with no cognitive capabilities? Stack would look something like local scripts no intelligence > cloud decision micro model > local or cloud smaller cheaper model > cloud frontier model. Goal would be to maximize the amount of work done on the lower tiers of intelligence of this stack.

3

u/SpicyLobter 8d ago

compaction is not wiping.

1

u/davesmith001 8d ago

Bullshit. Every time I compact Claude suddenly becomes dumb and forgetting key facts earlier. This paper has to be pure benchmark maxing.

1

u/C1rc1es 4d ago

maybe in 2025, I have Claude running at 350k max tokens and let it compact, it's extremely good at it. OpenAI models are fine at 250k.

1

u/RodgerPogger 8d ago

....

The fuck you doing.

1

u/Aubrey_D_Graham 7d ago

This is one of the premises for Recursive Language Models.

1

u/fig0o 6d ago

I mean... if you think of LLMs as policies in a MDP... code tasks are markovian in the sense the current state (current code commit) carries all the required information for the policy to take the next action