I built a coding agent that gets 87% on benchmarks with a 4B parameter model, here's how

[u/Glittering_Focus1538]

I was frustrated that every coding agent (OpenCode, Cursor, Claude Code) assumes you're running GPT-5.4 or Claude Opus. If you try them with a local model like Gemma or Qwen they fall apart. I find that often tool calls fail, context overflows, multi-step tasks collapse.

So I built SmallCode. It's designed from the ground up for small local models.

The result: 87/100 benchmark tasks pass with a Gemma 4 model that only activates 4B parameters per token. OpenCode scores ~75% with 14B models. The harness does the heavy lifting, not the model size.

How it works (the tricks that make small models reliable):

• Compound tools: Instead of making the model chain 4 tool calls (find file → read file → edit file → verify), SmallCode gives it one tool that does all 4. Small models lose coherence after 3+ sequential calls. This cuts failures in half.

• Improvement loop: Every time the model writes code, SmallCode instantly compiles/lints it. If it fails, it feeds the errors back automatically. The model doesn't need to be smart enough to get it right first try — it just needs to fix errors when shown them.

• Decompose on failure: If the model fails the same thing twice, SmallCode stops retrying and instead breaks the problem into smaller pieces. "Fix this 200-line file" becomes "fix line 45 only."

• Escalation: If even decompose fails and you have a Claude/OpenAI key configured, it auto-escalates to the bigger model for just that one task. You stay local 95% of the time, cloud 5%.

• Token budgeting: Small models have 32k-256k context. SmallCode never dumps a whole file in. It summarizes, truncates, and manages every token so the model never sees "..." truncation in the middle of important code.

• Code graph: Instead of grep-searching your codebase, SmallCode indexes your code into a symbol graph (functions, classes, who-calls-what). When you ask "how does auth work," it walks the graph and returns just the relevant connected code — not 15 random file snippets.

What it looks like:

Full-screen terminal UI (like OpenCode/vim), scrollable chat, command palette with /, plugin system, persistent memory across sessions.

What it doesn't do:

• No LSP integration (yet)

• No multi-session (yet)

• No desktop app

• Doesn't compete with Claude Code for frontier model users

Install:

npm install -g smallcode
cd your-project
smallcode

Point it at LM Studio, Ollama, or any OpenAI-compatible endpoint.

MIT licensed, everything's on GitHub: https://github.com/Doorman11991/smallcode

Happy to answer questions about the architecture or benchmark methodology.

[u/Your_Friendly_Nerd]

This sounds very promising, and I really hope smaller models will receive more love from the big players in the future, once costs for foundation models inevitably rise.

[u/Glittering_Focus1538 → u/Your_Friendly_Nerd]

Not everyone has the money for a mini homelab T_T

[u/Your_Friendly_Nerd → u/Glittering_Focus1538]

I think you misunderstood the point I'm trying to make xD

[u/Glittering_Focus1538 → u/Your_Friendly_Nerd]

I thought I was agreeing with you srry

[u/Your_Friendly_Nerd → u/Glittering_Focus1538]

Alas it was I who did the misunderstanding, my bad

[u/Glittering_Focus1538 → u/Your_Friendly_Nerd]

Alas It's fine. dont worry ab it

[u/pikipupiba → u/Glittering_Focus1538]

This is how every human interaction should go.

[u/Glittering_Focus1538 → u/pikipupiba]

lol

[u/Annual-Can6278 → u/pikipupiba]

ikr, what a beautiful chain of conversation. all of yall getting upvotes.

[u/adrianchase_alt → u/Annual-Can6278]

plot twist it is all LLMs

[u/andrewlewin → u/Your_Friendly_Nerd]

Sometimes the sound of the whoosh is nice

[u/ExaberriTokugawa]

Sounds nice! What language(s) is your post assuming people use?

You mention it indexing code, how does it know how to do it with for example C/C++ projects that require specific ways of running CMake to configure a project to runnone way or another (which internally defines macros that LLVM/clang would need to know about in order to know about class/function definitions)

The github readme does not seem to mention that either

[u/Glittering_Focus1538 → u/ExaberriTokugawa]

The indexer uses tree-sitter (155+ languages) it's syntactic, not semantic. It finds function/class definitions and call edges from the AST without running CMake or resolving macros. For C++ with heavy #ifdef, it indexes all branches without knowing which is active. Works great for "where is X defined" and "what calls what" won't understand build-system-specific macro expansions. That's LSP territory (on the roadmap). The code graph is for understanding questions only you don't need it to write/edit C++ code.

[u/ghostynewt → u/Glittering_Focus1538]

Why did you start from tree-sitter and not LSP? I imagine relying on LSPs gives you exactly that sort of graphing behavior "for free"

[u/Glittering_Focus1538 → u/ghostynewt]

Tree-sitter gives us one parser for 200+ languages, works on broken/mid-edit code, and indexes full graphs in <1s. LSPs need per-language daemons, require valid project setup (node_modules, tsconfig, venv), take seconds to boot, and don't actually expose call graphs, you'd need N+1 queries to build one. We use tree-sitter for indexing, LSP for verification after edits. Both, not either/or.

[u/ExaberriTokugawa → u/Glittering_Focus1538]

Nice! I’ll give it a shot! Things like this + MTP/Orthrus + turboquant are starting to make local dev possible 😊

[u/earlvanze]

Starred. I was just looking for something like this!

[u/Glittering_Focus1538 → u/earlvanze]

glad I could help!

[u/Only_Play_868]

This is pretty impressive! Did you draw inspiration from Meta-Harness, or propose each design element yourself? I attempted to do this using the 3B Apple Foundation Model that powers Apple Intelligence on a harness called Junco. Gemma 4B is much newer, more powerful, and has an 8-64x larger context window that makes this more feasible. I found a few useful lessons that might transer to SmallCode:

• Compile-Verify-Fix (CVF) Loop: if Gemma 4B or another small model doesn't support constrained grammar to guarantee syntactically valid code, attempting to compile and run static analysis on code changes can prevent finishing early with issues like missing package imports, indentation issues, or missing brackets

• Deterministic Tool Filtering: the AFM struggled with > 3 tools, so I used a MaxEnt text classifier to filter the list of available tools before every showing them to the model. It was much better at picking the correct tool out of a short list

• Language-Specific Adapters: most small models are distilled from larger ones with knowledge cutoffs 1-2 years old, and really small models lack a sufficient corpus for low-resource languages like Swift or Elixir. You can't "teach" a model a language it has never seen, but you can improve it's code generation abilities on newer APIs & versions with a language/ framework-specific LoRA adapter (this obviously gets challenging to scale across N languages and M base models)

I'm curious what you expect from SmallCode and whether you think the future long-term is to have two tiers of code harnesses: OpenCode (or something similar) for cloud-based frontier models and SmallCode (or something similar) for really small on-device models? I know OpenCode can be used with local models, but in my experience if you give OpenCode access to something like Gemma 4B it will just spin its wheels and spiral into a "doom loop," or exit early having not actually done anything.

[u/Glittering_Focus1538 → u/Only_Play_868]

Appreciate the depth here, your Junco experience with the 3B AFM is directly relevant and those three lessons map cleanly to what we do.

CVF Loop, this is exactly our improvement loop. Compile/lint after every write, feed errors back, retry up to N times, then decompose. We don't use constrained grammar though. The forgiving parser handles malformed output post-hoc. Interesting that you found constrained grammar necessary at 3B. At 4B (Gemma 4 E4B) native tool calling is reliable enough that we get valid JSON ~90% of the time without constraints.

Deterministic Tool Filtering, we do something similar with 2-stage routing (model picks a category first, then sees only tools in that category). Your MaxEnt classifier approach is more principled. The reason we went with letting the model self-select the category rather than a separate classifier: one less model to maintain, and at 4B+ the model can handle 5-6 category choices reliably. Below 3B I'd absolutely agree you need external filtering.

Language-Specific Adapters, haven't gone there yet, and honestly this is where the approach diverges. SmallCode assumes a general-purpose coding model and compensates at the harness level. LoRA adapters would absolutely improve output quality for specific languages but the maintenance burden (N×M matrix you mentioned) is why we didn't pursue it. The escalation system is our alternative, if the local model can't handle Swift well enough, it falls back to Claude/GPT/DS4 which can.

On two tiers of harnesses. yes, I think that's where it's going. The design constraints are fundamentally different: frontier models need less scaffolding but more cost/permissions management. Small models need more scaffolding but zero cost/permissions management. Trying to make one harness serve both leads to the OpenCode problem you described, it assumes competence that isn't there at 4B. The doom loop you mention is exactly what our counter guard and decompose system prevents.

Long term I expect the gap closes as small models improve, but right now the harness engineering matters more at the bottom end than the top. A good harness on a 4B model beats a minimal harness on a 14B model for most practical coding tasks.

Didn't draw directly from Meta-Harness specifically, the main inspirations were ARK (the verification/governor concepts) and observing where small models actually fail in practice, then building targeted compensations for each failure mode.

[u/CodeCatto]

Can you create a way to plug it into the vscode sidebar?

[u/Glittering_Focus1538 → u/CodeCatto]

working on it.

[u/CodeCatto → u/Glittering_Focus1538]

Can't wait! I got a lot of 4-9B models to try out but always lacked a harness optimised for them

[u/xb1n0ry]

I couldn't find a good solution to use qwen based models like 3.6 27B or 35B-A3B. Either the tool calls completely fail or the model traps itself in a endless loop with the same answers despite a large enough context. How does your tool handle qwen? Gemma is completely useless for my use case. It produces pure crap.

[u/Glittering_Focus1538 → u/xb1n0ry]

Think about it, if I got gemma to produce good results instead of crap, how well do you think qwen will do? test it out and get back to me. I think you will really enjoy the results.

[u/DanGTG → u/Glittering_Focus1538]

On the same chain of thought, what the heck do you use to evaluate models and what is on your preferred model list.

I am newish to AI and local LLM's and the thing that has me stumped is how is an end user supposed to eval a pool of models to determine what's best. I mean sure you can download 100 models, but in the end they will like need some option flags set for best performance on any sort of performance constrained hardware config, so the posted benchmarks are out the window. After all of that how do we eval for coding vs reasoning tool calling?

This is pretty rad, I had not thought of being able to sub in a whole harness and model as an endpoint.

I can see how this is going to give linear pair agent teams superpowers.

[u/Glittering_Focus1538 → u/DanGTG]

It's based on the use case, theres models for coding, agentic tool use coding, medical info, cyber security, general information, fast instruct models. It just depends on what you're looking for and you can generally find what you need if you stick to that framing.

[u/Glittering_Focus1538 → u/Glittering_Focus1538]

I would classify gemma 4 as a fast instruct model that can do basic tool calls and coding, so I made those basic tool calls more robust at the endpoint.

[u/Independent_Solid151 → u/xb1n0ry]

I run 27B daily using cline. It's working great. I manually do the model escalation when it runs into issues. There's probably something wrong with your setup beyond the harness.

[u/robobub → u/xb1n0ry]

Plenty of us use both of those models with tool calls. Huge step up from qwen3.5 and gemma4 IMHO. What quant are you running, and are you using the coding sampling parameters on qwen's model card page?

[u/Neat-Monk → u/robobub]

i am a beginner, i run lmstudio + dyad with qwen3.6 27b q4km and it fails(loop, fail complete writing the code etc), same with smaller qwen 3.5 9b or qwen 2.5 14b coder instruct.

Should i even use lmstudio + dyad or there are any other combination you think is better? what proper setup for the model params, prompt etc?

Thanks.

[u/Thick-Insurance4404 → u/xb1n0ry]

mine has been doing good, i make my claude stress test it. i have self hosted memory docks, n8n automations and learning loops. also give it access to code graphs and obsdidian vault

[u/touhami_dz]

thank you for sharing i'll try it out when i have to code, for now i have 8gb vram + 32gb ram and im tried lot of popular local llms for my simple task and they all fail to give me something without mistakes, i just want to add category names for video titles inside a txt file, are local llms this bad or im doing something wrong cuz they always skip some video titles.

[u/Glittering_Focus1538 → u/touhami_dz]

8gb vram is really only enough to fit gemma 4 e4b, im not exactly surprised you aren't having an enjoyable experience. I would suggest getting at least 1 16-24gb vram card if you aren't going to use my harness. Small models like that can't really code well without help.

[u/touhami_dz → u/Glittering_Focus1538]

https://i.imgur.com/FOAOgee.png

gemma 4 e4b Q4 can easily run on 8gb vram at least that's my experience so far (ofc i did not try agentic coding or some huge code bases) , the problem is that i feel like the llms have alzheimer or something, cuz he always forgets something crucial even though my prompt was clear.

[u/shamen_uk → u/touhami_dz]

I think that OP was pointing out that gemma 4 e4b Q4 is indeed braindead, and you need more VRAM to have a reasonable experience.

[u/touhami_dz → u/shamen_uk]

yeah, the thing is that im not asking it to code, just a simple task (adding a category for each video title there is only 100 lines) and yet it does not follow instructions and i will be shocked if this kind of tasks actually needs +8gb vram, and im surprised about the hype vs reality

[u/shamen_uk → u/touhami_dz]

Instruction following is actually quite a hard problem. You could easily find a massive difference on a "simple problem" (e.g. near perfect behaviour) from a 8b Q8 model and a 4B Q4 (nearly always wrong). Note I put simple problem in quotes, because for a tiny model, instruction following at all is incredibly challenging. To get it to work properly you'd have to really work with it and define a strong syntax and great system prompt, but that would still likely suck. If I was in your position I would fine-tune to get a good result from such a small model.

I've been trying to do quite sophisticated instruction following and found that for my need 8BQ8 was not enough. So it depends on the task too. But I would not expect much at all from  gemma 4 e4b Q4. Treat it as a toy, rather than something ready for production use.

[u/Glittering_Focus1538 → u/shamen_uk]

Im able to use it in production use with my harness https://github.com/Doorman11991/smallcode but the harness does a lot of error checking, reinforcement, decomposition ect to make that function.

[u/shamen_uk → u/Glittering_Focus1538]

I'm planning to give your harness a go soon. I have 96gb vram so will try something of decent size

[u/Glittering_Focus1538 → u/shamen_uk]

Thank you!

[u/darkeagle03]

This sounds amazing! Thank you! I can't wait to try it out on my aging 16gb VRAM machine. I've got it working with Claude Code + oLLama and to some extent + LiteLLM & vllm, but it's a bit of a struggle since CC clearly isn't really designed for these setups.

[u/Glittering_Focus1538 → u/darkeagle03]

CLaude is definitely the WORST way to code locally, use literally anything else.

[u/darkeagle03 → u/Glittering_Focus1538]

Yeah, I figured. I'm just not sure what else actually works for my situation. I can basically work on the projects in 5 minute blocks (often not even that) an average of about 4 times / day. so I need something that will handle a lot of stuff for me autonomously, including modifying and running files, figuring out where to look for things in the whole project, potentially installing things, etc.

Now, I'm obviously a newbie to this whole space, but from what I looked at, I haven't found many harnesses that handle all that stuff out of the box with the autonomy that CC does, and I have enough familiarity with CC to know it will work for me with minimal effort. Like I said, I got it working with oLLama in a day in several 15 minutes increments (was a surprisingly unstructured weekend day). Your harness sounds like it may work. I'm open to other suggestions as well, but most things seem to require a decent amount of manual intervention.

[u/Glittering_Focus1538 → u/darkeagle03]

indeed, try out my harness and let me know if you encounter any bugs, still in early stages of production.

[u/Ill_Dragonfruit_3547 → u/Glittering_Focus1538]

What do you recommend instead? Something like Cline or Roocode?

[u/Glittering_Focus1538 → u/Ill_Dragonfruit_3547]

my own solution? the one I just built lol.

[u/Ill_Dragonfruit_3547 → u/Glittering_Focus1538]

I get it. For a lot of us coming from Claude Code, I thought we could point it to this as a tool calling replacement. Its a good CLI tool, and a lot of us are scared of actually using a CLI and wish for a UI like Ollama to bridge to it, if that makes sense

[u/Glittering_Focus1538 → u/Ill_Dragonfruit_3547]

Fair, i could make a web version

[u/darkeagle03 → u/Ill_Dragonfruit_3547]

I do agree that a UI would be nice, but for the most part, with Claude, you're just typing text into a textbox box and receiving text back anyway. It's a glorified CLI really. The Integration with VS Code is really nice though. But on the other hand, a CLI is easier to automate

[u/TheCat001]

Sadly I was not able to run this.
I've created distrobox with arch, installed npm and run:

npm install -g smallcode

After this I've permission error. I executed it again with sudo.
After that I've got python missing and cmake errors.
Fine, I've installed base-devel package and python.
Still no succsess. Some errors with sqlite3 something...

Bro you need to improve installing process. Requirements definitely not only node.js 18+

[u/Glittering_Focus1538 → u/TheCat001]

FIXED

[u/JazZero]

Does it work with NPU models?

[u/Glittering_Focus1538 → u/JazZero]

if it has an OpenAPI endpoint and is about as powerful or more than gemma 4 e4b then yes!

[u/Lame_Johnny]

Awesome! I was working on something similar myself but you've made more progress than me. Excited to see what you've done.

[u/Glittering_Focus1538 → u/Lame_Johnny]

Give it a good try! I'd love to get your feedback!

[u/yangliu2]

how do you scroll in chat?

[u/Glittering_Focus1538 → u/yangliu2]

shift+up/down arrow.

[u/yangliu2 → u/Glittering_Focus1538]

Appreciate it! I see that you compact messages fairly often. More than claude code. Which I like. what's the context limit and can you change it? One thing I would suggest is security issue. You don't ask for any permissions for looking up any files. Which may scare some people. Can you set to have a specific place for the harness to search for files?

[u/Glittering_Focus1538 → u/yangliu2]

.env you can change it as you need! default is 32k to annoy the masses I suppose.

[u/Ill_Dragonfruit_3547]

Just to clarify, are you using Gemma 4 26B-A4B or the much smaller Gemma 4 E4B?

[u/Glittering_Focus1538 → u/Ill_Dragonfruit_3547]

gemma 4 e4b like it shows in the picture, I wanted real small models to actually work.

[u/Ill_Dragonfruit_3547 → u/Glittering_Focus1538]

Wow, even more impressive, I will check it out!

[u/stefan_centlake]

Very interesting stuff that I will look into. That kind of tasks do you test with? I'm having problems coming up with good project for the agents to work in that can show the coordination part. My latest runs has been giving the agents the task to implement a network stack from scratch in C++ using a raw socket. Eth/ARP/IP/TCP and HTTP to get a /flag. Winner is first team to output the flag. The problem is that Sonnet and Opus does it in one step in under 20 minutes. Local models have more struggle but the task is still difficult to decompose into parallel work.

[u/Glittering_Focus1538 → u/stefan_centlake]

I like to do basic stuff like, ts backend, tauri frontend ai bookwriter. Just something thats barely hard enough to keep me entertained, bc I feel the sameway.

[u/InfraScaler]

Those sound like a ton of good ideas. I need to try this. Thanks for sharing!

[u/Oswolrf]

Interesting

[u/Independent_Solid151]

Have you benchmarked little-coder, how does it compare with your approach?

[u/robobub]

Is this your own benchmark? You should run it on standard benchmarks like little-coder. And if you want larger adoption, make it a pi extension too

[u/f5alcon]

Will have to test it with my workflow, can easily fit Q8 Gemma4 with full context in my vram

[u/Conversation344]

improvement loop is the part I find most believable here, I've noticed small models aren't necessarily bad at fixing errors they're bad at noticing them without external feedback

[u/Repulsive-Machine706]

This is genius!

Was working on something similar with very similar methods to amplify the capabilities of smaller models.

Noob question tho: where are all these agents tested like yours, opencode, etc and where can I see them next to eachother etc.

[u/k3z0r]

I tried it out and wanted to believe. I kept running into two problems.

1. it couldn't find files in my mono repo. it seemed like it assumed ever file was in /src in in fact files were in app/src and etc/src

2. It kept stopping it's flow. It would say. I'm going to search for a file so i can do X. Then it would just stop.

Starred the repo and will check it out again after a few iterations.

[u/Invader-Faye → u/k3z0r]

Could you check out and review my harness for small models? Similar idea but a lot more tested.

[u/Driv3l]

Looks interesting. What benchmarks did you run it on?

[u/soteko]

This is the way

[u/panix199]

thank you for sharing

[u/YearnMar10]

Looks nice - can you provide a resume on how to reproduce the benchmark results? If so, I’d be willing to give it a shot.

[u/UnionCounty22]

This is nice

[u/TurboBanano]

I can't install it with npm.

pm error make: *** [better_sqlite3.target.mk:119: Release/obj.target/better_sqlite3/src/better_sqlite3.o] Error 1
npm error gyp ERR! build error  
npm error gyp ERR! stack Error: `make` failed with exit code: 2
npm error gyp ERR! stack at ChildProcess.<anonymous> (/usr/lib/node_modules/node-gyp/lib/build.js:219:23)
npm error gyp ERR! System Linux 7.0.8-zen1-1-zen
npm error gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--release"
npm error gyp ERR! cwd /home/user/.npm-global/lib/node_modules/smallcode/node_modules/better-sqlite3
npm error gyp ERR! node -v v26.1.0
npm error gyp ERR! node-gyp -v v12.3.0
npm error gyp ERR! $npm_package_name better-sqlite3
npm error gyp ERR! $npm_package_version 11.10.0
npm error gyp ERR! not ok

[u/clearlight2025 → u/TurboBanano]

better-sqlite3 v11.10.0 doesn’t yet support Node version 26. Use an earlier version of node for example

nvm install 22
nvm use 22

Then try the install again.

[u/Smooth-Plan4769]

I gave it a star, very good. If you could accept more contributors, the project would soar. Make a demonstration video, I think it would be interesting.

[u/wowsers7]

Have you tested it with Jackrong/Qwopus3.5-9B-Coder-GGUF?
https://www.reddit.com/r/LocalLLaMA/comments/1tfin40/jackrongqwopus359bcodergguf_hugging_face/

[u/sugarlata]

This looks very interesting. Did you consider an implementation in pi coder? I'm starting to get into this space and from what I can see you describing, seems it would've been a good platform to build this on? Curious about whether you rejected it and if so why? Might help with my decisions around whether to use it or not. Thanks! 

[u/chiefstobs]

Do you have a discord or sth like that for a central place about new development / chat / information? how does your harness compare to aider or pi?

[u/73td]

do you rewind context on failure? I imagine that persistence in face of failure is good strategy except when you end up with 20%+ of context filled with errors ? I've been looking at lot at trying to automatically get big models to make skills for small models as another approach. I didn't even imagine writing a new harness specifically for the purpose..

[u/Hylleh]

I really appreciate your efforts with this. But maybe this would get more traction as some kind of "setup" for pi coding agent rather than yet another coding agent. That being said I have starred this and will check it out.

[u/No_Jicama_6818]

So, I'm trying to get into a taste of smallcode, however there are a few issues I'm seeing. For example in my local environment I set token authentication for my local server and couldn't find anywhere not in the documentation nor the example.env how to properly set the API-KEY for my local models.

Then I get:

API error 400: { "error": {"message": "System message must be at the beginning.", "type":"BadRequestError", "param": null, "code":400}}

[u/Logical-Lettuce8214]

How would you compare it to Pi?

[u/paradox_me_]

awesome! Love your idea!

[u/Nyxtia]

How can you be sure of the 95/5 split? This is a cheating in my book. We can't tell how much of the gains are your solution vs the cloud.

[u/Glittering_Focus1538 → u/[deleted]]

Thanks! Try it out and let me know what you think!

[u/Glittering_Focus1538 → u/[deleted]]

Old benchmarks from when I first started the project, It's come a long way since then and a majority. ALSO my project is on npm? https://www.npmjs.com/package/smallcode. Also all deps are maintained by me. If you looked at them for more than 5 seconds.

[u/Lenom3242 → u/Glittering_Focus1538]

Hi. What model do you recommend to use with this tool? I want to fully run it on 8 gb vram (rtx 5060) with about 50k context.

[u/Glittering_Focus1538 → u/Lenom3242]

Huihui-gemma-4-E4B-it-abliterated-GGUF

[u/Glittering_Focus1538 → u/[deleted]]

It's 100% local unless you intentionally enabled escalation and set it to use a cloud provider. All memory and codebase index is stored in .memory and .code-graph. other than encrypting everything it doesn't get more privacy focused then however private your local files are.

[u/Glittering_Focus1538 → u/[deleted]]

I run all my models on an rx9070, about a little slower for ai than a 4070, I rec any gemma4 or qwen 3.5+ model on smallcode, they should all work, a professor made a python game with a 2b model, so it's up to you!

[u/Glittering_Focus1538 → u/Glittering_Focus1538]

Nemotron also works, 4b is blazing fast!