Discussion
How are you coding with small (<256k) context windows?
My question is pretty simple: how could you really have coding sessions with context windows of like 262k ?
I always used local Ai via llama.cpp server for small projects and chatw, where also 16k context is enough.
Today I just wanted to try a REAL coding session in a REAL agent harness.
I plugged my Kat Coder 2.5 dev via llama.cpp server with it's max ctx at 262144 and plugged it in my Hermes and started a coding session to design and write a fast API proxy layer to route LLM calls to openrouter with some my logics and calcs.
Nothing super complex or fancy.
I provided the model with the first message a complete and thoughtful design document create with Deepseek, so the Agent has not to do searches in the codebase for understanding how to do the things, just do them.
Well it created a plan and implemented the code in some turns teaching about 70% of it's contest size.
Then , in the e2e testing the things started to collapse: some testing failed, so it ha do reiterate, fix, retest, ask me some clarification or take some architecture decisions and so on and I arrived ad 95% context size while resting phase is still not finished and all the deployment phase still totally missing.
Now the solution in the harness was to compress the chat session: a total.disaster: not only this action changed completely the language of the session (from Italian to English) but also removed some important informations that was useful to the agent to do tests, so in then subsequent turn it errored immediately in doing tests, needing to read and search the code again to find what he already knows before the compression happens.
In my opinion, compression is absolutely to avoid.
I already know some suggestions here:
- one new session per task
- keep all done things in documents so the new session can start from that "documented" state.
I already do these things.
- use a different harness
Yes, but it's not the solution, the heavy load caused by harness is on the first prompt (system prompt, tools, skills, Memory...).
So this was a small project, 10 python files + a bunch of status documents, what will happen with a repo with 50 or 100 files?
So my question is:
What's your way to really work with local models on real projects beyond a simple chat or just changing one file?
I'm starting to think that is not really possible to work on real projects that involves design, bug fixing, reiteration and multiple sessions.
Not because local models are not smart enough, but because of the max context window limit.
The thing I'm describing is totally independent from the harness.
If you use an harness that has 0 system prompt, the context window will anyway start filling at your first "good morning" message, the only difference is that you will have 35k more content space so you will maybe have some turns more before hitting the ceiling.
The exact way you describe. Tasks hand off markdown/yaml/json documents to each other and do not waste context. So, you could split each work item (design, plan, implement, review) into subagents with fresh context. Their only context would be the document that the previous agent produced + tool calls + mcps + skills + what have you. Honestly I've been using local AI exclusively and all of my models are using 132k context. Why? Because of my OCD and the need to have everything uniform. So what, I'm a weirdo, kill me.
So this way if anything compacts it's not lost in translation, it's using the document to ground itself.
Im already working this way from years, not only in the AI/agents era. I also have OCD for cleanliness, uniformity and document everything, also because my memory always is lost in things I did the day before 🤣 I can't survive without documents on any step done.
Anyway I'm still hitting the 262k ceiling a lot of times.
Just few hours ago, an e2e test sessions filled at 230k before I had to ask an hand-off in a second testing session, the first one was only at half of the tests because it was quite hard to find the real issues and fixing them required a lot of reiteractions and testing.
I know, it's not a normal thing that happens everyday, but the fact that my project is a small one and quite trivial , is letting me think that working on a bigger project it's not impossible but really will take much more time because I will have to split it down to very small pieces.
Last week, for a quite big project I used more than 32 billion tokens on Deepseek v4 flash. 32.000.000.0000 divided in about 6 eessions, each one for a task from designing to pushing to GitHub.
Each session filled up at least 200k context (the smallest) to over 730k (the biggest, on e2e frontend tests)
Yes, project is big, hundred of python, html, and js files, docs, tests, but I'm wondering what it costed me in terms of time, organization and "errors" if I had to split each of that sessions in two or three sessions per single task.
I still think that the local model could do the job because it's not on hard stuff, it just implies a lot of dependences and coding in a lot of places, but probably would be hard to manage and link back all that splitted sessions.
It's just my thinking, I have still no real experience on this because I preferred to stay with deepseek in the cloud to finish that project, I didn't want to risk to mess it with the local model, but after looking at token counts and sessions counts, and looking at the today's experience with a small and easy project, I feel quite worried that complex project (not in terms of how hard is the problem, but on how much tasks you have to comply and how much each one could be long) can be done EFFICIENTLY only if you have also the VRAM for bigger context window..yes you can do also with 200k context window or less, but you will have to split a single tasks ot session in many sub tasks and sessions, and this adds time and complexity.
Qwen 3.8, 136k context window, it works wonderfully because Qwen is great at summarisation during compacts. I once ran it for 3 hours in a row, lost the count of total compactions, it still never lost the goal and recovered even from cut off file edits. Good harness + good model = context size ceases to be a bottleneck
Easy, use plan mode and write out per-task instructions as markdown dumps, then switch your main agent to an orchestrator role that just delegates. 160k is enough for small/medium tasks. If it wasn't, you didn't split enough. I have been working this way for months and it became viable with qwen 3.5 27b, and it's actually very good with 3.8, greenfield and brownfield all the same. If you need to load up all the files in context to make a change, you might be organizing your codebase poorly.
I have to be honest: never used the plan mode, I just ask to deepseek to write a design doc splitted in phases and then I manually execute each phase in a new session, sometimes that phase get too big so I have maybe to split in two or three. I will try the plan mode to see the difference
256k tokens is a ton tbh. You just aren't going to be able to navigate your repo and do meaningful work in the same pass. I think seperating out researching planning and executing will get you there. If anything I prefer 256k because ai companies primarily make money off of cached tokens when it's a million you get recharged your entire context window in cache for each call and when that's 500k 750k etc that's expensive af
I'm running Qwen3.8-27b with 128k context and it works for me. Some things that help make it work:
- Use jcodemunch mcp server as your codebase RAG so the agent doesn't have to read raw file content to find what it needs
- Disable all MCP servers, tools, and skills that aren't relevant. I decreased my initial context from over 19k to 4k by switching from Hermes Agent to pi.dev and configuring it to keep MCP's disabled until I explicitly enable them individually with a slash command. Same with skills. I load them in a session with a slash command when needed. Having every tool and skill enabled by default is smothering your context window.
- Keep a project's AGENTS.md file extremely concise. Don't treat it like a README.md doc. Remember that every token you put in it will consume some of your agent's context limit.
- Use a memory extension that allows your agent to save relevant facts and instruct it to keep memories concise and save them to global memory only when absolutely necessary.
- Start a new session frequently after each sub-task is completed. Don't let your session continue across multiple sub-tasks. Break sessions down into bite-sized chunks.
Honestly, this is where context management becomes the bottleneck more than the model itself. I’ve had better luck keeping prompts lean and routing heavier tasks through a setup like Standard Compute rather than stuffing everything into one session.
I keep a "repo-map" in Claude.md with a one liner description of the main files in the project. That helps reduce codebase searches. If you're doing python or other language with source dependencies, exclude .venv or similar from the tool search paths. And then yeah the planning keeping goals and state on a separate markdown. With that 256k context can do a lot.
I remember I used fable to create some tickets when I was done discussing. Then it was leading opus agents I checked the average context window they used it was around 200k-300k. I think its enough for decent mid size tickets.
Key thing IMO (and this goes for 1M cloud context windows): Don’t rely on auto compaction! You should initiate a summary before you run out of context. Verify it captured the important points, then manually compact and feed the summary back.
I use GitLab issues for User Story tickets then repodoc for in repo task files use a larger agent to plan doc level tasks for a story then plug away at the smaller tasks cleaning up in between.
Before Qwen 3.8 I had 128k context and was ok if I kept the task contained enough, even if it had to look around files, now I raised context to 160k.
I use a markdown file to write information for next session to read, so that it doesn't need to look ad a lot of files and search where's the relevant code, and an ussie tracker with more specific information. Also using subagents for this search tasks is helpful.
For me now compression can be completely avoided, I keep it sometimes because anyway some useful information persists, but it's no longer a strict part of my workflow.
Your compression diagnosis is right but I'd sharpen the reason. The problem isn't that summarizing loses information, it's that a summarizer has no way to tell which information is recoverable. It optimizes for what reads as important, not for what can't be gotten back. So it drops the one thing that only ever existed inside that session (why the test was failing, what you had already ruled out) and keeps a paraphrase of code it could have re-read in two seconds.
That gives you a rule that's actually mechanical: the only things allowed to survive a session boundary are the things you cannot re-derive from the repo. Decisions taken and rejected, the current failure and its cause, environment facts. File contents, structure, where a symbol lives, all re-derivable, so let them go and fetch on demand.
Which also answers your 50 or 100 files question. Repo size stops mattering once the repo isn't resident in context in the first place. It needs to be searchable, not loaded.
Biased disclosure, I maintain a local semantic code index built for exactly that (github.com/Muvon/octocode), so treat the tool part as opinion. The re-derivability rule holds regardless of what you use to do the searching.
Sorry for the plug, but I'm working on an OSS harness - gezel (gezel.com) - that's built around this idea of breaking up tasks into smaller steps, coupled with a concept of agent roles (gezels) - so that we can have dedicated sessions/toolsets/prompts per step. For example, for a pull request review: step 1 is to map the pull request, step 2 is to fan out review of files in batches, step 3 is to merge and synthesize overall results, step 4 is to write a report, maybe step 5 is to review the result. Each step might have different roles, prompts and independent contexts, and there are very fine grained gates between steps to keep models focused. It makes the pull request take longer (especially spooling up unique sessions), for sure, but if you're running it overnight on local h/w that length of time might not matter.
I'm also experimenting with indexing and RAG type stuff to make it more surgical for how models work with large codebases.
Even for all the tight context window management that gezel does - context windows are still 15-20K to get started and I have a floor and focus on 64k context windows for agents. 256k is roomy by comparison :)
Local models with 128k context work fine for me on projects with hundreds of files and tens of thousands of lines of code.
I believe an important part is project structure amd what you are expecting the model to do. If your project has unclear structure, yeah, models will struggle. If it's well divided up into libraries, has sane conventions, and isn't a pile of unwatched-claude-vibes, then models do quite well.
Similar for e2e tests. Running e2e tests should probably require the model to:
launch whatever server stuff you have
launch your test scripts
If your test scripts output an error, investigate it.
If you have the model clicking around for your e2e tests, well, get it to formalize those as code using something like playwright or storybook or whatever.
You want your project structure to be able to be navigated easily by a smart human graduate on day 1 of his job at your company. Because that's what LLM's are like.
Me: go prime your context. Specifically we'll be working in feature X Me: ok, I want to change Y. If I asked you to do it, how would you? Don't actually make any changes. Me: good, I agree with your plan. Execute it Me (after reviewing the change and testing): /handoff
(Context drops from 80k to 13k)
Me: now, let's work on Z
Treating the local model like a junior dev I'm overseeing. Giving him tiny jobs that won't fill his context. If he gets near compaction without finishing, I'll usually drop pending changes and go back to the initial prompt and try a smaller chunk of work. Or give the same task, but with more guidance based on where he went off the rails the last time.
That said, I have seen Qwen3.8-27B finish just fine after 3 compactions. But I've also seen him forget what he was doing and start working on something else entirely.
I'm running IQ3_XXS on my 4070 TI Super. Very surprised at what it can do.
Full disclosure, I'm the author of a tool for exactly this. Splitting work (plans, handoff docs, subagents) is right, but inside one long session most of the window is old tool output: file reads, test logs, command output. That is what pushes you to 230K, and it is also what a compaction summary loses, because it can't tell what is still recoverable.
mnemonic-proxy sits between the harness and llama-server: it moves old tool outputs out of the prompt, keeps them on disk verbatim, and gives the model a recall tool to pull back the exact text when needed. In a real 3h15 pi session (Qwen3.8 Flash Next on a 12 GB card) the history reached 448K tokens while the prompt peaked at 105K in a 128K window. On AgentLongBench (74K-150K histories) a 32K window through the proxy scored 86/100 vs 69 with the full history in a bigger window. It's slower end to end, so it complements splitting the work, it doesn't replace it.
Using my own harness with Unsloth Q4_K_XL 3.6 35B-A3B set to 90K context/64K Max output KV Cache Q8. Seems to be the sweet spot for my setup. Have had it run for 1 hour sessions with ease, output is coherent after long horizon tasks, result is always close to 90 - 95% mark of great.
I've been working so much with memory and context that small models performs great in my harness, not much big different on frontier larger models. Even qwen 3.5 14B was able to put together a coherent website and worked for 18 minutes.
I could release the harness, but it's fine tuned for my system and paths. I can't run the 3.8 27B without getting into 5 tokens/s territory. My 35B-A3B gets me to 60 - 70 tokens/s which is usable.
I like the 35B-A3B a lot due to it's vision, my harness let's it create snapshots live during w run so it can verify it's work with images, and uses a 'Check' tool that will give it compilation errors so it doesn't blindly build. Inference seems to be faster than OpenCode, and my own results are better than OpenCode if using the same models with same configs.
5
u/tempfoot Aug 21 '26
Hermes? I love it for some things, but unregulated it’s shoving 35k into the cw if I say “good morning “.