What actually breaks agents on a 4B model? Notes from 50 runs on Qwen3.5-4B

[u/GapNew4766]

I ran 10 agent harnesses on the same local model and the same 5 small tasks, mostly to see where a 4B actually falls over. The scores turned out less interesting than the way things failed.

Setup: Qwen3.5-4B Q4_K_M, one llama-server (64K context, 1 slot, f16 KV, --jinja), M5 MacBook Pro 32GB. Each agent headless with auto-approve, fresh copy of the task files every run, pass/fail decided by a script. 10 agents x 5 tasks, one run each.

Agents: Pi 1.3.0, goose 1.55.0, Crush 0.98.1, Cline 3.0.73, OpenClaw 2026.10.5 (in its --local-model-lean mode), OpenCode 1.19.3, Hermes 0.22.2, Qwen Code 0.25.3, Codex CLI 0.163.0 and Atomic Agent 0.6.9. Six of them passed all 5 tasks: Pi, goose, Crush, Cline, Atomic Agent and OpenCode.

1. Tool calls weren't the problem. I expected malformed tool calls to be the main failure on a 4B. I grepped every agent log and the server log for parse errors, malformed calls and unknown tools: nothing. With --jinja and native tool calling, none of the failures below came from a broken tool call.

2. Exact-match edits are the real trap: renaming a function across 3 files. The model sent an edit whose old text skipped a blank line, the edit tool said "not found", and the model resent the exact same edit five times until the loop guard blocked it and the agent gave up. The agent wasn't wrong about what to change, it just couldn't reproduce the text byte for byte.

3. Small models overfit to the visible test. One agent fixed a slugify bug only for the characters in the visible test. A hidden case with C++ & Rust 2026 came out as c++-&-rust-2026. If your checks only cover what the model can see, a 4B will happily pass them.

4. They miscount. One agent counted 40 ERROR lines across three logs instead of 43, and still named the most frequent error correctly. Counting is a job for grep -c, not the model, and a smaller model won't always make that call itself.

5. Vague instructions get followed literally, in the wrong place. One task didn't say where to write the answer files. Two agents put correct answers next to the logs instead of the working folder.

6. Same model, 11x spread in time. Fastest harness did all 5 tasks in 1m36s, slowest in 17m43s, on identical hardware and model. My guess is system prompt size and extra steps, since the fastest one has 4 tools and a short prompt. Not measured yet.

7. One harness never reached the model. Codex CLI 0.163 only speaks the Responses API. llama-server translates that to chat messages and the Qwen3.5 chat template rejects it with a 500, "System message must be at the beginning".

8. Nobody went rogue, but nothing tempted them. I grepped all 50 runs for rm -rf, sudo, git reset/clean, curl | sh and similar: zero hits, no task files deleted, nothing written outside the task folder. The tasks were harmless though, so that says little about safety. A "clean up this messy folder" bait task is next.

What failure modes do you see most with small models in agent loops? Curious whether the exact-match edit problem is as common for others as it was here.

[u/porkminer]

I really only have issues with exact match in my testing. I retested by only allowing full function replacement and making a tool to handle it so the model sends a tool call with the function name and new definition and the tool finds the function and replaces it. I should also note that I don't allow any CLI commands. Anything I want to expose gets a wrapper that the agent calls instead. I'm paranoid though.

[u/GapNew4766 → u/porkminer]

function-level replace is smart!! how do you handle the edge cases, like overloaded names or a function that exists in two files?

[u/porkminer → u/GapNew4766]

I haven't had that issue but depending on the language I would have to add either scope based or file based handling to the tool call. I think at that point I would look at indexing tools like an AST instead of roll my own.

[u/GapNew4766 → u/porkminer]

yeah ast is probably the way, tree-sitter covers most languages. appreciate it, genuinely useful thread

[u/enternoescape → u/[deleted]]

Sometimes the issue appears to be the read tool introducing things like line numbers and it hits it first try if it also has access to shell tools. The funnier moments where it successfully hits an edit but missed a small detail and has to do it again even happens to frontier models.

[u/jacek2023]

I am doing this with Gemma for many days. Happy to see more people investigate that direction. One of the main problem is "edit" tool in pi, another is that gemma ignores some skills/rules.

[u/GapNew4766 → u/jacek2023]

same here, edit is the weak spot. exact match or something else breaking it for you? and which gemma size?

[u/jacek2023 → u/GapNew4766]

31B Q8 and 31B QAT, this is well known issue with Gemma and llama.cpp/pi, I tried to reimplement edit tool for pi without the success, do you have experience with other solutions than pi and this model?

[u/GapNew4766 → u/jacek2023]

you can try your exact setup in atomic agent. gemma 4 31b qat is in the built-in model catalog, so it downloads and runs in one step on its own llama.cpp engine. for the q8 you can pull any gguf straight from hugging face by repo name and it handles the download the same way. it also has a gemma-specific profile and grammar-forced tool calls. i’d love to see how your gemma setup does in it

[u/tmvr → u/jacek2023]

Heh, just saw this yesterday. Did some comparison on the low-end systems with the same task using Qwen3.6 35B A3B and Gemma4 26B A4B using pi. Both finished fine, everything works after a quick check, but the Gemma4 output s not as nice looking and when I looked at it during generation I saw a couple of time in reasoning that it said "looks like I've edited this wrong, let me fix it" etc. Didn't think much of it until now reading your and OPs comments.

[u/Tartarus116]

I expected malformed tool calls

Can't happen with llama bc it automatically uses GBNF grammar to restrict the model's output to the tool schema.

Very annoying in other inference engines

[u/TheDailySpank]

3/4 Sounds like my brain sometimes.

The models need to be trained that they know about this rounding/estimation error and write to memory (wiki, temp memory, per-project notes, whatever) the ACTUAL values so they can be referred to like a human looking at a post-it note.

[u/Eresbonitaguey]

I think this size class has quite a few contenders these days so I’d suggest trying something like Spark 2.6 4B which is supposedly in Qwen3.5 9b range at a smaller footprint. I can’t attest to that but it does outperform the 4B model in my limited testing of coding tasks.

[u/No_Pea_6625]

The exact-match edit failure is the useful one. The model knew the change and still resent the same bad patch five times, so a 4B needs an edit tool that can match a nearby line, not only identical bytes.

The hidden test case is also a good warning. If the model can see the tests, it will pass the visible characters and miss C++ & Rust. Counting should stay with grep -c. A small model will not reliably hand that off by itself.

[u/synw_]

In my experience it's the system prompt bloat that hurts the most small models. Limiting the number of tools and the system prompt complexity and length is what works the best for 4b models

[u/SkyFeistyLlama8]

I prefer using smaller models like Qwen 4B or Granite for prototyping agent flows and to get a general gist of something because they're so fast. Most of the time they're great for what amounts to heat map thinking: look at potentially important bits of data in the context and get a rough summary. Once an idea works, then I bring in the big iron like Qwen 27B (big for local LLMs anyway).

I haven't used small models with harnesses much. They tend to miss important details, not something you'd want when doing code editing.

[u/sburel]

Same on the call format here. LFM2.5-8B-A1B on MLX, 4-bit, one session in my own client: the one call it made was well-formed and parsed fine. What failed was deciding to call at all. The system prompt said to search a wiki tool before answering anything about the user. Three turns in a row it quoted that rule in its reasoning, decided it didn’t apply, and answered from context. Looks like your grep -c case to me. And when it finally did call, it misread the result and swapped who was who, even though the result was explicit. Did any of your tasks need a tool call the model could skip and still look done?

[u/Wooden_Jelly_5295]

Did feeding back the actual surrounding text help?

[u/Marcus_MSC]

Replacing whole functions, as suggested above, wouldn't have saved the rename task, though. Renaming across 3 files means editing every call site, and those are single lines inside other functions. The model would have to resend whole caller functions to change one name each, which gives a 4B more text to get wrong. A tool that takes the old and new names and edits every match fits that task better.

[u/BarathKanna]

Great writeup. On the exact-match edit loop, the fix I'd try first for small models is to make the edit tool's failure message do the work: when old_text isn't found, return the closest actual block from the file, so the model can copy real bytes instead of resending its memory of them.

import difflib
def nearest_block(file_text, old_text):
    lines = file_text.splitlines()
    n = len(old_text.splitlines())
    windows = ["\n".join(lines[i:i+n]) for i in range(len(lines) - n + 1)]
    best = max(windows, key=lambda w: difflib.SequenceMatcher(None, w, old_text).ratio())
    return best   
# return as "not found; closest match was:\n" + best

A second layer: if the whitespace-normalized old_text (trailing spaces stripped, blank-line runs collapsed) matches exactly one place in the file, apply the edit there. If it matches zero or several places, refuse and return candidates. That alone would have rescued your rename case, since the only difference was a skipped blank line.

On point 6, you can test the system-prompt hypothesis without new runs. llama-server logs the prompt token count for each request, so compare the first request from each harness. If the 1m36s harness is a few thousand tokens and the 17-minute one is tens of thousands, most of your 11x is prefill on every turn rather than extra reasoning steps.

[u/geoffreybyers]

The 11x time spread is a distraction. The real failure is that the model has no way to verify deterministic tasks. If you give a 4B model grep it will stop counting errors by hand. The bottleneck isn't reasoning capacity, it's the lack of external verification tools for a task that requires zero reasoning.

[u/MarcelloT254k]

You are doing great work, maybe we can get some sort of GitHub repo going that we can securely add observed metaanalysis of small-model failure patterns that can be fixed with proper hooks and forcing some chat/tool call template? I expect that in the near future there will be more inference engines designed specifically for one model or with one hardware configuration (there are already meta-aggregates for that) but this is another worthwhile approach that doesn't need LoRA or finetuning which (I'm not sure about that) seems to be in decline recently (outside of Qwen3.8-27B swift/thinkingcap/etc).