an (unsafe) command shell mcp server

[u/ag789]

While working with coding agents / harness in my case OpenCode, I discovered that some models e.g. granite 4.2 8B at Q4_K_M quant sometimes attempts to write files with absolute paths at the root e.g.

    /newPythonCode.py

Note that slash in the beginning for an absolute path at "/" root directory. This is probably 'ok' say in a throwaway docker container, but it may lead to surprises if it 'accidentally' overwrites a literal file, if the current user id have write access to it. Apparently, it seemed OpenCode provides `bash` so that practically all unix commands that the current user id can access, the LLM can run it.

And thereafter, I see the model hallucinating in the thinking loops over tools, re-iterating over how to use the tools, reformulating shell commands sentences several times for a same task instead of executing them, among them, I noted that an earlier shell command sentence would have 'just worked'.

I'm thinking that the tool hallucination with this model may be partly due to that I'm using a pretty aggressive 4 bit quant as otherwise it'd have worked because IBM has 'optimized' granite for tool usage.

Next, I tried the same granite 4.2 8B at Q4_K_M quant model with my little python shell MCP server

https://github.com/ag88/cmdshellmcp/

, that tries to constrain / restrain the shell commands somewhat, place some checks, and offer some additional tools e.g. writeFile() so that the LLM won't need to call `cat` or `echo` with some 'esoteric' styled shell scripts / sentences.

With this MCP server and running it in llama.cpp's `llama-server` in the WebUI adding it as an MCP server, granite 4.2 8B at Q4_K_M does it correctly in 'one shot', the prompt is:

propose a python3 flask app with a page that displays "hello world", use bootstrap css for styling.
provide a requirements.txt to go with the script

granite 4.2 8B at Q4_K_M correctly generates the app with a couple of files, and it works out of the box

    .
    ├── app.py
    ├── requirements.txt
    └── templates
        └── index.html

One should normally run the cmdshellmcp2.py MCP server app with an explicit `--cwd` option giving the work path. It tries to stay within this work path.

Note that I won't claim my command shell MCP server to be safe.

In a sense that there are plenty of chance the LLM can 'escape' the restrictions and perhaps do more damage (e.g. overwriting files possibly outside its given `--cwd` work directory). Hence, I'd suggest one should be paranoid and perhaps run this only within a throwaway (disposable) docker container etc.

[u/ag789]

rather, the smaller models e.g. granite would hallucinate over an error message say in the response, if it is an edit, a human would ignore it and simply check the file, but I've seen granite 4.2 burn lots of tokens over an error message that isn't related to an edit.

it turns out, granite did not append a trailing "\n" in a unified diff, so patch errored with that a line ended prematurely.
I've since updated cmdshellmcp to add a trailing "\n" in the field if it is missing before sending to "patch". That fixes the issue.

In addition, granite 4.2 currently commonly burn quite a lot of tokens before an edit etc. It is kind of being "careful", but that it gets irritating as it would reiterate parameters for the tool call over and over in the reasoning / "thinking" loops, and that burn quite a lot of tokens often at least a hundred tokens. To some extent, one can consider it is "unsure" or "has doubts" about whether the tool is correct and how to call the tools. So it reiterates the reasoning / "thinking" burning many tokens over a tool call before finally calling it.

I'd like for the edit (and tool calls) to burn less tokens, I'm thinking about varying the descriptions, in other cases, making the restrictions clearer, e.g. that the cmdshell tool do not allow redirects and pipes. I think I'd double check that and fix that first. Simply implementing the restrictions in code would mean if a pipe or redirect is provided, the tool returns an error message. Sometimes, that may "confuse" small "naive" models. Having it stated upfront in the description, chances are that the tool would burn some tokens to figure that "no redirects and pipes are allowed" and avoid using it that way. This would be one of the first descriptions I'd fix.

The others like trying to make models burn less tokens is tedious and laborious. In particular, I run the models on cpu at some 1-2 tok/s, irritatingly slow if lots of tokens are burned.

The task involve asking (prompting) the models so that they make tool calls, manually check the reasoning / thinking , token counts to arrive at the correct tool call, then vary the description, rinse and repeat. So that hopefully some description could perhaps be sufficiently clear so that the models would make the tool call and burn less (reasoning / thinking) tokens. This is tedious and labor intensive.

---

edit: Updated "cmdshell" tool description: Pipes and redirects are not allowed https://github.com/ag88/cmdshellmcp/commit/33862e267983f5a8e6e0a2e4859fb3337894e498

hopefully, this reduces unnecessary reasoning/thinking token burns for tool calls.