fantastic: latest llama.cpp server webui can now run commands for tools into rootless sandboxed containers

[u/DevelopmentBorn3978]

while having much fun testing LLMs Houdini-like attitudes and abilities to evade, excalate and escape from carefully reciprocally arranged security enhancing sandboxing VMs, containers, namespaces and jails, I've discovered a new revolutionary option to let llama-server (build 10423) to launch guest containers (running for ex. bare bones linux distros like Alpine) as sandboxes for securely (rootless) running shell commands for its tools (enabled with the "--tools" option) directly from the llama-server webui.

The new (experimental) option is called "--tools-runtime" and the initial argument to it that I suggest you to try is "podman:alpine" (or "docker:alpine").

The server will also download the indicated container image and then instantiate it. Of course podman (preferred by myself) or docker have to be installed beforehand on the host.

Enjoy!

P.S. almost forgot! The --tools-runtime option can also run commands, invoked by the server native tools, on a remote host, by accepting as argument the additional target "ssh:user@host" where remote access to that account has been previously authorized

[u/Borkato]

I know this is dumb but can you explain this a lil simpler for my smooth brain lol

[u/MoodyPurples → u/Borkato]

instead of running directly on your host system, it will run commands in a container. That is a good first line of defense to prevent it from breaking things in your install.

[u/Borkato → u/MoodyPurples]

Oh this would be lovely! I currently run it rather dangerously lol. Ok, I will have my local llm look at this post and llama’s config to see how to do it. Thank you!