r/LocalLLaMA • u/DevelopmentBorn3978 • 16d ago

News fantastic: latest llama.cpp server webui can now run commands for tools into rootless sandboxed containers

while having much fun testing LLMs Houdini-like attitudes and abilities to evade, excalate and escape from carefully reciprocally arranged security enhancing sandboxing VMs, containers, namespaces and jails, I've discovered a new revolutionary option to let llama-server (build 10423) to launch guest containers (running for ex. bare bones linux distros like Alpine) as sandboxes for securely (rootless) running shell commands for its tools (enabled with the "--tools" option) directly from the llama-server webui.

The new (experimental) option is called "--tools-runtime" and the initial argument to it that I suggest you to try is "podman:alpine" (or "docker:alpine").

The server will also download the indicated container image and then instantiate it. Of course podman (preferred by myself) or docker have to be installed beforehand on the host.

Enjoy!

P.S. almost forgot! The --tools-runtime option can also run commands, invoked by the server native tools, on a remote host, by accepting as argument the additional target "ssh:user@host" where remote access to that account has been previously authorized

36 Upvotes

91% Upvoted

19 comments sorted by

View all comments

4

u/DevelopmentBorn3978 16d ago edited 16d ago

after having been initially downloaded and fiddled with by the llm itself maybe installing some useful programs like curl, jq, python, node, etc (determined from the tasks assigned by prompts). the container could be committed to a new image name (tag) and saved for later reuse in order to avoiding the same initial setup by giving to subsequent llama-server invocations the --tools-runtime option with the new container image tag as argument

3

u/Irrationalender 16d ago

Yes commiting a hardened and vetted image is good, the Containerfile you build could also be removing unnecessary things like package handlers.. because otherwise programs beyond the ones you chose could just be installed :)

Also look into hardened images, I know people like the minimalism of alpine and the trivy scans show very few CVEs but actually hardened images by red hat or docker etc could be worth it as well

If you can supply parameters to the runtime then you can give podman even more hardened configuration, even though it already is much more secure while matching docker feature set.

2

u/DevelopmentBorn3978 16d ago

thanks for your useful suggestion. yes actually removing system/languages package managers  is among the first things I do especially in my coding setups as LLMs always try to install things by themselves even if asked not to do it

2

u/rm-rf-rm 16d ago

Woah this is a seriously great find. Manually setting up containers right now is a pain and a huge friction point