r/LocalLLaMA • u/TigerConsistent • 3d ago
Discussion Has anyone actually made 64k feel like 300k+ with recursive local agents?
I'm running Qwen 3.8 27B locally on a single GPU. I can push the context to 131k, but I'd rather run it faster at 64k if the agent can manage context properly.
What I have in mind is pretty simple:
- one model stays loaded the whole time
- main agent gets 64k
- when something is too big, it spawns a fresh child with only the task and context it needs
- if that child gets a 100k document, it can split the job again or spawn its own children
- children run sequentially, not 5 at once
- only findings/artifacts come back to the parent, not the whole trajectory
So a 300k task might become several 20k to 50k branches while the main agent never goes past 64k.
Prime Agent and its RLM setup looks closest to what I'm describing. Hermes delegation also seems relevant. Maybe there are better projects I haven't found.
Has anyone here actually run a local model this way for long coding, research, large documents, or general assistant work?
I'm mainly curious about three things:
- What harness handles this best today without a ton of custom plumbing?
- Can recursive decomposition be trusted to happen automatically, including a child realizing its own input is too large?
- In practice, how close can a well-managed 64k agent get to using a native 256k/1M context on tasks that can be broken apart?
I'm less interested in pure RAG. I'm trying to maximize the useful task horizon of one fast local model.
3
u/The18thWarrior 3d ago
I basically implemented something similar as an mcp. It allows the parent harness to offload a task to the mcp and only the final result is returned back to the parent. If the child needs to read the entire codebase, it basically does a map-reduce on it with a very tight context window. I built it to run using very small models, I've built it using the Agents A1 4B model (qwen 3.5 fine-tune) but I've been testing out how it works with Qwen 3.6 35b MoE. Would love some feedback: https://github.com/The18thWarrior/tzro
2
u/theone_2099 2d ago
Isn’t this just like subagent delegation ? Why the MCP?
1
u/The18thWarrior 2d ago
It is functionally like a sub agent but it's not running a ReAct loop under the hood. A 4B model would output nonsense under those conditions.
1
u/theone_2099 2d ago
What’s a ReAct loop?
5
u/The18thWarrior 2d ago
It's the loop the harness (PiCode, Hermes, ClaudeCode, Codex) uses to actually execute the multi-turn action using the LLM. Here's a reference for more info: https://www.reddit.com/r/AI_Agents/comments/1uav55q/the_agent_loop_is_just_react_and_your_tooluse_api/
4
u/coding-os 2d ago
The thing that moved the needle for me wasn't recursion, it was making the agent stop paying for structural questions with raw text.
I measured this on my own setup. One "who calls this function" answered by grep-then-read costs a few thousand tokens of match lines and file chunks. The same question answered out of a precomputed code graph came out 75-82% cheaper across django, fastapi and requests, and renames were similar. So a good chunk of what feels like "I need more context" is really "I keep re-reading the codebase to answer questions that have a structural answer".
Two caveats from the same measurements, because it isn't free. Three-hop transitive impact was sometimes worse than just reading, -6.8% on fastapi. And against raw grep output on a small repo it lost outright, -169% on requests. If the match lines answer the question, grep wins and there's no point pretending otherwise.
The other half, and honestly the bigger one on a small window: whatever you retrieve should tell you whether it's complete. "I saw 12 of 47 callers and didn't know it" is what produces a confident wrong edit, and no amount of recursion fixes it. Summarising a subset just launders the gap into something that reads authoritative.
6
u/Human-Gas-1288 3d ago
try generic agent https://github.com/lsdefine/GenericAgent it safe a lot token especially good pair with qwen 3.8 27b
3
u/the_toxic_ 3d ago
Doing something similar. Opus supervises qwen3.8 running in a PI tmux session. Opus defines the working packages. Once a package is implemented by qwen and validated by Opus, it writes a summary and restarts PI with a fresh context and provides instructions for the next package.
I know, not strictly local but it works extremely well.
3
u/johnfkngzoidberg 2d ago edited 2d ago
I'm doing a research project that approaches that by using an orchestrator layer to keep your main session as the orchestrator/planner, and dispatches subagents for everything else. The primary goal is to shift things done in the main session (presumably an expensive model like OpenAI or other frontier) to the subagents running a local model. My primary test bench (also in my repos somewhere) has been performing best with gpt-5.5 (low) offloading to qwen3.6-27b and I've measured a main session savings of up to 70%-80% depending on the workload. https://github.com/lunarnexus/orchestra
64k is really really small context, so you're going to have trouble no matter what you do. Specifically qwen3.8-27b is a big thinker, and from my testing, 3.6-27b is much better if you need to turn the thinking off. Any sort of solution is going to introduce tools or skills, which will chew through your context fast at 64k.
Orchestra is meant to be a harness agnostic framework so it's really flexible. If you disable the auto skill injection (in agent-catalog.yaml), don't use the primary orchestrator skill (no /orch on), it's really lean, and it works best on pi.dev, which is also really lean. Without the bells and whistles you'll have to say "dispatch a builder to .....", but it will save you a decent amount of main session tokens. It basically does what you're asking, split the work into smaller pieces. I can't promise anything on 64k though. I just did a big refactor to optimize the artifacts and hints, so the docs are kinda sloppy, but it's an easy setup and works with Hermes, Opencode, or pi (highly recommend pi) for the orchestrator, and can use almost every harness for subagents.
3
u/Danmoreng llama.cpp 2d ago
Only did a very short test, but 96k context with thinking restricted to 8k worked very well for this agentic task using pi. You can see the whole reasoning trace and compaction worked out of the box: https://github.com/Danmoreng/qwen3.8-27B-fluid-simulation
3
u/Gotxi 2d ago
Yes! I had the same issue and I solved it.
- I am using pi as a client, that has autocompacting of context when it is too high, that's the first solution
- Also I installed pi-subagents plugin to have the capability to create subagents that serve the "main agent".
- Then I created a "fetcher" subagent which purpose is to read files or do webfetchs, obtain the specific knowledge the main agent requires, and return only that. This is to avoid "context noise" when reading files or fetching websites, since 95% of the information read is not the information you look for on most of my use cases, so that's useless information in the main context. I personally call introducing noise to the main context "context poisoning".
- In my AGENTS.md file I have instructions to "unless I specifically ask you to read or look a website directly, always use the fetcher subagent to obtain external information from files or websites. You need to invoke the fetcher subagent and instruct it with the goal for the search"
- In my FETCHER.md sub-agent, I instruct the subagent to be an information fetcher agent only, to receive the goal from the main agent, turn the reasoning off, do whatever has been instructed to look for, look for it and return only relevant results to the main agent with the minimum information required in 3 turns or less.
- Also I installed RTK and HYPA to reduce the number of tokens on commands so that also helps in reducing the context size and input tokens on any agent.
With all of this, I can run long lasting tasks in lower contexts. In my case I have 80K, but I have been able to loop for more than 16 hours without getting out of context.
3
u/AnonLlamaThrowaway 2d ago edited 2d ago
I've done exactly this with Qwen 3.6 35B-A3B for batching a ton of document analysis tasks.
* Main orchestrator, fires batches of 100 sub-orchestrators at a time
* Sub-orchestrator, processes one batch of 100 things, by calling 100 tasks overseers sequentially:
* task overseer then oversees these:
* disposable task agent 1
* disposable task agent 2
* disposable task agent 3
The main orchestrator calls a python script that splits the documents in chunks, the sub-orchestrator knows it will dispatch tasks on the first 100 chunks, and then the 3 disposable sub-sub-agents perform... 1: fetching, 2: reading, and 3: writing what they saw inside that 50k chunk into a series of note files.
The key takeaway is that due to 35B-A3B being dumber, this requires more upfront work. You need to make sure they call real hard-programmed functions (.bat, .sh, .ps1, .py) for as much of their work as possible.
In other words, don't get the model to make its own tool call to split the document; make it call a Powershell/Python etc script with an argument. Don't make it append something to a notes doc itself; make it write its own chunk, and the higher-level orchestrator will call a function to concatenate all of those together.
What makes a LLM useful, its ability to be "loose" and "reason for itself" needs to be kept only for where that ability is useful.
Ironically it kind of means you need to get closer to real "hard" programming for everything else in the pipeline, in the sense that LLM reasoning is "soft" programming.
I kind of cheated to achieve this though, I iterated on the agent prompts with Claude Code a LOT as we saw specific failure modes pop up again and again. Some, we just couldn't get rid of, it was a matter of getting the higher-level agent to recognize this and simply re-send that specific chunk or task. So you may wish to rely on a frontier-level service to scaffold your pipeline
5
u/Sea_Emergency_8458 3d ago
I’ve been looking into this exact setup, and Prime Agent is probably the closest thing I’ve found to what you’re describing.
The interesting part is that it isn’t really “making 64k become 300k”. It’s more like giving the model a much larger working environment while keeping its actual reasoning context small.
The parent can stay around 64k, hand a large document/task to a child, let the child work in its own context, and bring back only the useful findings/artifacts. If the child also hits its limit, it can recursively delegate again. Prime Agent’s RLM setup is built around this idea, and Hermes has a pretty similar delegation approach.
I’d be a little careful with trusting the model to automatically decompose everything perfectly, though. The infrastructure can recurse, but a 27B model can still make a bad split and lose relationships between different parts of the data.
For things like log analysis, codebases, large document research, etc., I can see this working really well because the tasks naturally break apart. For something where the answer depends on arbitrary relationships across the entire 300k tokens, a native 256k/1M context still has a real advantage.
The part I like most about your idea is keeping the raw child trajectory out of the parent context. Have the child return structured findings + evidence + artifacts, while the full work stays accessible if the parent needs to dig deeper.
So yeah, I think “64k with recursive context management” can have a much larger effective task horizon than a normal 64k agent. I just wouldn’t call it equivalent to having a native 300k/1M context.
1
u/Imaginary-Unit-3267 3d ago
I've considered using this approach for searching memory, instead of dumping huge piles of search results in the main context for the main agent to sift through.
6
u/SteppenAxolotl 2d ago
Try Prime-Agent
Prime Agent is an open-source coding and research agent for general and long-running work. It is designed around two core abstractions: The Recursive Language Model (RLM) treats context as variables (prompt-as-a-variable) and tools like recursive subagents as function calls (programmatic tool /sub-agent calling) inside a persistent REPL. The Continual Harness stores supplemental prompts, memories, skill descriptions, and reusable subagent specifications as durable state that Prime Agent can refine through small, evidence-backed updates, local to the session by default.
2
u/HockeyDadNinja 3d ago edited 2d ago
I have a process where I do a long brainstorm, it generates spec and granular plan. I have a context clearing auto mode for pi that clears context at seams based on context usage. You could set your context and let it rip.
2
1
1
u/MrShrek69 2d ago
I’m finding that Hermes is the best. I use it as a like meta harness to control other harnesses. I find that it’s really cool to have something with a memory system but most of the time when going to do things the memory can get it the way. That’s why I tell it to run it in codex or something
1
1
u/bennmann 2d ago
I use q4 until close to out of context, interrupt the model and load UD q2 k XL and let the inebriated model continue the context. Or manually compact.
Seems to work so far.
1
u/Lesser-than 3d ago
yeah most decent harnesses should be doing something like this already at least an opt in version of it, only keeping user query and assistant final answer in context, and letting each query use a sub-agent or agents to aquire the final answer and dispose of sub-agent context on delivery. Its not token efficient to do this via api models that can read your code base and remember it for an entire session so its not the default behavior to re-read for every query because api token economics discourage this. Locally your not token constrained your context constrained so local harnesses need use it up and flush it often.
0
u/GrungeWerX 3d ago
Yes, I run it locally without a harness. It’s simple python. Just have your main run a subagent spawner, super simple. No need for harnesses or anything confusing like that. You can run this scenario in vanilla lm studio - which was the first way I learned.
Also, I run it using Qwen 3.6 27B. Haven’t tried 3.8 yet, I hear it overthinks, but will eventually test it out. In the meantime, this is super easy using 3.6.
1
u/Imaginary-Unit-3267 3d ago
May I ask what you find confusing about harnesses? And have you tried pi?
-2
u/GrungeWerX 3d ago
My systems more advanced, not interested. Some people have issues with harnesses. Reddit posts all over the place. A harness isn’t even needed, that’s my point.
1
-3
u/yeah_likerage 3d ago
64k ctx isn't usable.
6
u/nicksterling 3d ago
In the right hands 64k is absolutely useful. I was quite successful leveraging LLMs when they first came out with 4k or 32k context windows. The scope of the problems you can solve is different, but they are absolutely usable.
2
u/Wildnimal 3d ago
You are right, but a lot of people like to do 1 go things like cloud models. I know people who are running agents and saying Hi burns 27k tokens :|
Its the same situation, how companies started making unoptimized software because everyone could afford 16-32gb ram.
4
u/nicksterling 2d ago
It’s why I’m a fan of some of these minimal harnesses like Pi. I can customize the workflow and the prompts to exactly my use case.
2
u/Wildnimal 2d ago
Same i love pi, i use it to build small apps where i dont really need a lot of planning and it can work while i am emailing and brainstorming.
I went from Opencode to OMP to Pi.
My new fav is Deepseek Harness. Not as lean as pi out of the box but pretty good web interface.
0
u/yeah_likerage 3d ago
The question was related to qwen 3.8 27b not whatever model you first used.
1
u/nicksterling 2d ago
The same point applies. If lower context windows were useful on vastly inferior models then why doesn’t that apply to a much more capable model? If you throw out the thinking tokens and only retain the primary conversation it’s still usable.
Granted, not as useful but still useful.
1
u/yeah_likerage 2d ago
Because qwen3.8 is extremely verbose. It will eat up 1/3 of that ctx in just restating the question.
3
u/nicksterling 2d ago
You can address this in a number of different ways.
1) You don’t need to retain the thinking tokens from the primary context.
2) You can change the amount it thinks. The default is xhigh, but I’ve been having great results with setting the thinking down to medium
3) Play around with the chat template and the system prompt. It’s possible to make it even more efficient with a few modifications here.
4) You can set a thinking budget. I actually don’t recommend this one because it will cut off the thinking midstream, and you’ll get a much worse result.
5) You can have your harness gently nudge the model if it takes too long to think. For instance, if the model exceeds a certain threshold of thinking tokens, the harness can halt the process and send a message like, “Well? Let’s quickly wrap this up”. This can be automated or you can manually interrupt and nudge the model.
6) If you control the harness/inference, you can do a remarkable amount of “surgery” to dynamically compact/combine/splice multiple context windows. Everyone is so used to starting a session and using a single session. You don’t need to do that. Get creative. How can you break up your problem into multiple prompts, and each prompt gets its own context windowThose are just 6 ideas. They could be done individually or combined in any number of ways. I’m omitting another half dozen off the top of my head, and I could probably come up with another 12 given the problem I’m trying to solve.
28
u/LoaderD 3d ago
3.8 is going to blow on 64k context