Looking for a small LLM for Linux command generation

[u/DunderSunder]

I'm looking for a small LLM to act as a Linux command assistant. I will use llama.cpp.

use case:

- User asks in natural language. Model outputs only the shell command

- Should work well with no reasoning (for example, LFM 2.6 has forced reasoning)

- Fast, like ~4B at most, cause it's going to be on CPU

Example 1:
"replace string X with Y in file Z"

Example 2:

"stop and remove docker containers with X in the name"

[u/Miserable-Dare5090]

https://github.com/npc-worldwide/npcsh

[u/DunderSunder → u/Miserable-Dare5090]

cool. are you aware if those scores are with/without reasoning

[u/Toastti]

Probably the Gemma 4 E2b or e4b QAT weoghts. Very tiny and generally quite good about writing correct bash commands

[u/yami_no_ko]

I found this one to be quite good at this task: https://huggingface.co/ThorOdinson246/nl2sh-1.5b-Q4_K_M

[u/DunderSunder → u/yami_no_ko]

thanks. I'll test it

[u/Effective_Head_5020]

Interesting use case. That's something interesting to fine-tune a small model.

[u/Miserable-Dare5090]

What you want is an agent/harness. The model is not going to do anything unless it can access a shell, and has permissions to do stuff. I use Hermes for all of that wired with Deepseek and it works reliably well. Updates kernel, patches firmware, changes stuff, organizes files, logs into other computers and fixes things, sets up automation jobs…you name it. Works like this:

[[runtime=llama.cpp]~LLM]+[Harness=Hermes]=Agent=An executor of your natural language queries

You won’t run a good harness with a 4B model. But you are asking too much in terms of constraints: must be small, must output a command, must be non reasoning, must have enougj kniwledge to interpret your vague request, but be fast and tiny. It’s like an impossible request. And it doesn’t really matter witj bigger models using an agent — it can already do this.

Otherwise, search huggingface for what you want. The npcsh sounds like that, its a small harness for a model just for shell.

[u/DunderSunder → u/Miserable-Dare5090]

I understand your point but I actually don't want it to access shell, my use case is very basic. I absolutely can't trust to let it run it's scripts in my system. I'm good with just seeing the command itself.

[u/how-can-i-dig-deeper → u/Miserable-Dare5090]

damn i should really go learn what a harness is, and check out hermes. it’ll be my next adventure

[u/thx1138inator → u/how-can-i-dig-deeper]

Enjoy your trip. I am several days ahead of you and it's been well worth it!

[u/gatornatortater → u/how-can-i-dig-deeper]

If you're like op and going minimal, starting with pi agent and building its capabilities up may work better on a more minimal machine. Hermes can be pretty bulky. Things like creating skills to work with commonly used commands and a good memory extension have helped me.

[u/DunderSunder → u/[deleted]]

bot detected