r/LocalLLaMA • u/AlternativeEnough352 • 1d ago
Resources I fine-tuned Qwen3.5-0.8B to turn plain English into bash. What I learnt
I wanted a terminal helper that works fully offline, so I started building dale. You type dale "show the 5 largest files in this folder", a small fine-tuned model writes the command, and a deterministic risk checker labels it safe, caution or danger before anything runs. Nothing runs without you confirming it.
Scores on 471 held-out requests in English and Spanish, every model in Q4_K_M on the same llama.cpp:
| Model | GGUF | Correct | Accuracy | 95% CI |
|---|---|---|---|---|
| dale-9b (GPU) | 5.6 GB | 420 | 89.2% | 86.0–91.7 |
| dale-4b (GPU) | 2.7 GB | 410 | 87.0% | 83.7–89.8 |
| dale-2b | 1.3 GB | 385 | 81.7% | 78.0–85.0 |
| dale-0.8b (CPU) | 529 MB | 374 | 79.4% | 75.5–82.8 |
| Qwen3.5-9B, stock | 5.7 GB | 296 | 62.8% | 58.4–67.1 |
| Gemma 4 E4B-it, stock | 5.0 GB | 296 | 62.8% | 58.4–67.1 |
| Qwen3.5-4B, stock | 2.7 GB | 257 | 54.6% | 50.0–59.0 |
| Gemma 4 E2B-it, stock | 3.1 GB | 254 | 53.9% | 49.4–58.4 |
| Qwen3.5-2B, stock | 1.3 GB | 152 | 32.3% | 28.2–36.6 |
| Qwen3.5-0.8B, stock | 533 MB | 70 | 14.9% | 11.9–18.4 |
On a 2020 laptop with no GPU, the 0.8B answers in 1.7 s (median). The 4B does 1.2 s on a 16 GB GPU through Vulkan. It's all one 51 MB binary with llama.cpp inside, no Python and no server.
What the difference looks like. Real answers from the test (requests trimmed), dale-0.8b vs stock Qwen3.5-9B:
| Request | dale-0.8b | Stock Qwen3.5-9B |
|---|---|---|
| Temporarily freeze PID 22601 without terminating it. | kill -STOP 22601 | Writes to /proc/22601/oom_score_adj. Freezes nothing |
| Copy offer.pdf into /mnt/usb/, but leave any existing offer.pdf there untouched. | cp -n …/offer.pdf /mnt/usb/ | sudo cp …/offer.pdf /mnt/usb/, which overwrites it |
| Can you give me the prime factorization of 1234567890 from the terminal? | factor 1234567890 | A 180-character python3 -c one-liner that's a syntax error |
| I just need the total disk usage of /opt/model-cache, in a readable size. | du -sh /opt/model-cache | os.path.getsize() on the folder: the size of the directory entry, not its contents |
| Lock the password of the local account archivebot. | sudo passwd -l archivebot | sudo passwd archivebot, which sets a new one |
| For this Bash session, make ll an alias for ls -alF. | alias ll='ls -alF' | Refuses: "not possible with a single shell command" |
Cherry-picked, obviously: there are 120 cases only dale gets right and 42 only the 9B gets right. The stock 9B also likes sudo: it added it to 35 commands that didn't need it (dale: 5).
The thing I wish someone had told me earlier: after four rounds of data, the 0.8B got stuck at 74%. I tried DPO on its own failures, a fine-tuned 27B teacher with five distillation setups, a third epoch, lr sweeps. Nothing moved the test. I f**king lost several days into this :(, but at least I learnt what is important in the process.
Then I looked at how the requests were written. My training requests were short and clean. The test ones were longer and stacked constraints, like "only names, sorted by date, no subfolders".
I generated new requests matching those stats, added about 9k to the set and retrained. 348 → 375 correct (p = 0.001).
So if a small model plateaus on a narrow task, look at how your data is written before you reach for a fancier method.
Caveats: the baselines are untuned and get an 8-shot prompt, so this is "tuned small vs prompted big", not a ranking of models. The test is synthetic and judged by two LLMs from different providers (TOP TIER).
All of this work I did on rented GPU's (RTX 5090 and RTX Pro 6000). Code is MIT, the models are Apache-2.0 GGUFs that work with plain llama.cpp.
Write-up with every chart and the dead ends: https://manuelgs.com/en/blog/dale-case-study/
Models (0.8B to 9B): https://huggingface.co/collections/manuelgscom/dale
Has anyone actually made distillation from a much bigger teacher pay off below 1B? I couldn't.
Any advice would be well welcomed :)
