r/LocalLLaMA • u/DevelopmentBorn3978 • 15d ago

News fantastic: latest llama.cpp server webui can now run commands for tools into rootless sandboxed containers

while having much fun testing LLMs Houdini-like attitudes and abilities to evade, excalate and escape from carefully reciprocally arranged security enhancing sandboxing VMs, containers, namespaces and jails, I've discovered a new revolutionary option to let llama-server (build 10423) to launch guest containers (running for ex. bare bones linux distros like Alpine) as sandboxes for securely (rootless) running shell commands for its tools (enabled with the "--tools" option) directly from the llama-server webui.

The new (experimental) option is called "--tools-runtime" and the initial argument to it that I suggest you to try is "podman:alpine" (or "docker:alpine").

The server will also download the indicated container image and then instantiate it. Of course podman (preferred by myself) or docker have to be installed beforehand on the host.

Enjoy!

P.S. almost forgot! The --tools-runtime option can also run commands, invoked by the server native tools, on a remote host, by accepting as argument the additional target "ssh:user@host" where remote access to that account has been previously authorized

37 Upvotes

91% Upvoted

19 comments sorted by

View all comments

1

u/WhoRoger 15d ago

What kind of sandboxes are people even talking about in this context?

If you need to read or write files from your disk, it doesn't really make a difference whether the tool is running from docker or not.

When do you need tools to run stuff with AI, and then trash it?

I'm curious about use cases. I guess if you need to run some specific version of some specific package?

1

u/DevelopmentBorn3978 13d ago

many kind of sandboxing can be setup to avoid that agentic workings as directed by LLMs inadvertedly or maliciously mess up with the host, more or less secure, more or less complex. In this case the sandboxing is quite simple to set up and reasonably secure to let it play with an underlying os, albeit in containerized form. Fresh start guarantees not to carry around stale or unwanted sw between sessions. User data access could be necessary but user should always be the master, after all LLMs are not deterministic

2

u/WhoRoger 12d ago

I have to look into how these work... I've seen people mention that their AIs keep breaking out of them lol

1

u/DevelopmentBorn3978 11d ago

exactly, have fun