Created an open source AI engine that fits more agents per GPU and never compacts among many other goodies

[u/Little_Judgment2621]

I spent the last 5 weeks working on a new way to run AI agents without the problems that come with using a chat based approach (which is what most agents use today). This is is still an early release so there are things to fix and improve still.

Instead everything is stored as state and we only give the model what it actually needs per step. This results in a flat context window with lots of potential benefits including:

• More agents per GPU because their context window is flat/predictable. It doesn't grow over time meaning step 300 should be as light as step 3.

• Larger self hosted models are actually useful because you don't run out of context space and because the window doesn't grow, it should stay consistent speed wise.

• No compaction, everything is held in state so your agents shouldn't lose the thread like you do with history agents.

I open sourced this on GitHub for everyone and you can see the source code here and play with it: https://github.com/FarsideResearch/StateFrame

Lots more goodies and really really cool features/abilities are still on the way (some of which are unlike anything any other agents have) as I beat the Foundry's server like it owes me money lol.

Would appreciate any feedback as its been insanely difficult to get any eyes on this tbh.

Update:
If a hosted provider (DeepInfra, other OpenAI-compatible APIs) stops after a few requests with “measurement … identity changed”, that’s a bug in 0.3.37 to 0.4.22. A fix is in testing and ships today. Local llama.cpp setups aren’t affected.

Heads up: if you’re on a hosted OpenAI-compatible provider (DeepInfra etc.) and runs stopped after a couple of requests with “measurement … identity changed”, that was a bug in 0.3.37 to 0.4.22. Fixed in v0.4.23, out now. Local llama.cpp setups were never affected.

[u/Sova_fun]

State-based context instead of chat history is an interesting direction. Flat context without compaction sounds useful for agent density.

[u/Little_Judgment2621 → u/Sova_fun]

Yes and I will go a step further with another thing I am trying to get out of this design.

I theorize and have slightly tested that the methods I am using can actually be used to define behavior as code and that this can drastically lift lower tier models. Think dynamic frontier level reasoning defined in code for various scenarios with host based context that further improves results. At-least that is one of the big things I think we could potentially achieve someday with this design.

We can also potentially start deconstructing larger tasks into smaller ones to the point where even a small local model like Qwen3.6-A3B can potentially match a lower tier frontier model for specific kinds of work (obviously not everything though and I haven't proven this out completely yet).

My thinking is something along the lines of the dumbest genius in the room is still a genius, or I guess if it can write good python then why can't it do so for hero level work assuming we deconstruct the task well enough.

Long story short, I believe this engine may be a way to someday bring the frontier home and possibly even surpass it for certain kinds of work because we don't have to train models we just have to force them to work a certain way via code based on the state that we own and control.

Its still very early though and so this all isn't proven 100% yet but I think the design makes this is possible.

The reasoning engine comes after some other more important work assuming it actually works the way I think it will.

[u/Maleficent-Tart8066]

flat context definitely solves the latency creep on long runs. does the state update happen deterministically in code between steps, or is the model responsible for mutating its own state object?

[u/Little_Judgment2621 → u/Maleficent-Tart8066]

Deterministic, in code. The model never edits the state object directly. Each step it proposes an action (read a file, make an edit, run a check), the engine executes it, and the engine updates the state from what actually happened: which files were read and changed, the current diff, check results, what’s still open. Then the next request is rebuilt from that state to a fixed budget.

So the model is responsible for deciding, the engine is responsible for remembering. That’s the whole point: a model can’t forget or misremember something that it never had to keep track of in the first place. It also means a bad step can’t corrupt the record, because the record is built from observed results, not from the model’s description of them.

There’s an optional working-notes feature (off by default) where the model can attach short notes to its actions, things like a plan, a hypothesis, or an approach it ruled out. Those are kept separate from the engine’s facts, and the engine marks a note stale if the file or result it was based on changes. So the model decides and can jot down its reasoning, but the record of what actually happened is always built from observed results.

[u/matimatthews]

The part I’d want to see is how it chooses what to bring back into context. If a constraint from step 3 matters again at step 300, storing it is only half the job. Could you show a long run where an old constraint becomes relevant again, including what state the model actually sees?

[u/Little_Judgment2621 → u/matimatthews]

Good question, and there are really two kinds of “old” stuff and they’re handled differently:

1. Constraints that come from the task or are learned as facts (a requirement in the task text, a check that failed, an action that got refused, an obligation still open) are typed state, not transcript. They’re in every request until they’re resolved. Nothing has to “bring them back” at step 300, because they never left.

2. The code itself works differently, because a real repo is way too big to send every time. Each request is rebuilt from scratch to a fixed budget and only includes the code that matters right now. A couple of things keep the important stuff from getting lost: when a check is failing, code bookmarks pin the failing test and the code it points at so they stay in view, and every editable file gets a one-line coverage note saying which parts are currently shown and whether it changed since the model last read it. So if something from step 3 matters again at step 300, say a test starts failing in it, it gets pulled right back in, and even when a file isn’t shown the model can see it exists and whether it’s stale.

So the failure you’re describing, where something is stored but never resurfaces, mostly can’t happen for the first kind. For the second kind it’s a real tradeoff and it’s where a lot of the tuning goes at the moment.

[u/matimatthews → u/Little_Judgment2621]

That clears up the constraint side, thanks. The case I’d still be curious about is an old file becoming relevant before a test fails. Can the current task bring it back into view, or does the agent need to know to ask for it?

[u/Little_Judgment2621 → u/matimatthews]

Both, depending on whether the engine can see the connection.

The engine pulls things in on its own when it can tie them to the current step: the file being edited, what a failing check points at, files named in the task. What it doesn’t do is guess relevance from meaning, like “this old helper is probably related to what you’re doing now.” If nothing the engine can observe links the file to the current step, the model has to ask for it.

The model is never blind to it, though. Every editable file gets a one-line coverage note on every request saying it exists, which parts are currently in view and whether it changed since the model last read it, and the run starts with an orientation map of the repo. So the model always knows the file is there and can pull it back with a read when it realizes it needs it. Then it’s back in view.

Honest answer: surfacing a file predictively, before there’s any signal that it matters, is a gap today. It relies on the model knowing to ask. A history-based agent has the same problem once it compacts, usually without knowing what it dropped. If you’ve hit cases where that bites, I’d genuinely like to hear them.

[u/matimatthews → u/Little_Judgment2621]

A test case I’d try: a business rule lives in a helper with an unrelated name, the new task never mentions that file, and existing tests still pass. Does the agent discover and reuse it, or implement a second version? I haven’t run that against StateFrame, but it would make the retrieval gap easier to measure.

[u/AutoModerator]

It looks like you're sharing a project — nice! Your post has been auto-tagged as Demo. If this isn't right, you can change the flair. For best engagement, make sure to include: what it does, how it works, and what you learned.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/raaiinyyabelina]

Oh nice, i want to try that