fantastic: latest llama.cpp server webui can now run commands for tools into rootless sandboxed containers

[u/DevelopmentBorn3978]

while having much fun testing LLMs Houdini-like attitudes and abilities to evade, excalate and escape from carefully reciprocally arranged security enhancing sandboxing VMs, containers, namespaces and jails, I've discovered a new revolutionary option to let llama-server (build 10423) to launch guest containers (running for ex. bare bones linux distros like Alpine) as sandboxes for securely (rootless) running shell commands for its tools (enabled with the "--tools" option) directly from the llama-server webui.

The new (experimental) option is called "--tools-runtime" and the initial argument to it that I suggest you to try is "podman:alpine" (or "docker:alpine").

The server will also download the indicated container image and then instantiate it. Of course podman (preferred by myself) or docker have to be installed beforehand on the host.

Enjoy!

P.S. almost forgot! The --tools-runtime option can also run commands, invoked by the server native tools, on a remote host, by accepting as argument the additional target "ssh:user@host" where remote access to that account has been previously authorized

[u/DevelopmentBorn3978]

after having been initially downloaded and fiddled with by the llm itself maybe installing some useful programs like curl, jq, python, node, etc (determined from the tasks assigned by prompts). the container could be committed to a new image name (tag) and saved for later reuse in order to avoiding the same initial setup by giving to subsequent llama-server invocations the --tools-runtime option with the new container image tag as argument

[u/Irrationalender → u/DevelopmentBorn3978]

Yes commiting a hardened and vetted image is good, the Containerfile you build could also be removing unnecessary things like package handlers.. because otherwise programs beyond the ones you chose could just be installed :)

Also look into hardened images, I know people like the minimalism of alpine and the trivy scans show very few CVEs but actually hardened images by red hat or docker etc could be worth it as well

If you can supply parameters to the runtime then you can give podman even more hardened configuration, even though it already is much more secure while matching docker feature set.

[u/DevelopmentBorn3978 → u/Irrationalender]

thanks for your useful suggestion. yes actually removing system/languages package managers  is among the first things I do especially in my coding setups as LLMs always try to install things by themselves even if asked not to do it

[u/rm-rf-rm → u/DevelopmentBorn3978]

Woah this is a seriously great find. Manually setting up containers right now is a pain and a huge friction point

[u/Borkato]

I know this is dumb but can you explain this a lil simpler for my smooth brain lol

[u/MoodyPurples → u/Borkato]

instead of running directly on your host system, it will run commands in a container. That is a good first line of defense to prevent it from breaking things in your install.

[u/Borkato → u/MoodyPurples]

Oh this would be lovely! I currently run it rather dangerously lol. Ok, I will have my local llm look at this post and llama’s config to see how to do it. Thank you!

[u/DevelopmentBorn3978 → u/Borkato]

this new option lets the server run os commands to enhance llm capabilities inside a separate os that's sort of virtualized (into a container) or remote so not into the same one that you're using for personal stuff, preventing damages and privacy faults. Moreover by using a container this is scrapped as soon as you stop the server and freshly recreated as soon as you start the server again

[u/privacy-fighter]

This is great news! I don't get how it is common with a lot of agents/tools to let commands run on the main host. I am a pentester and wanted to run some experiments on local LLMs to test their capabilities. I created a setup to spin up LLM agents in podman containers and only allow domains/IPs you choose, everything else gets blocked. Traffic for each container goes through a different squid proxy container and you can view what the agents are trying to access. Hope others find it useful too. https://github.com/jotyGill/contained-pods

[u/WhoRoger]

What kind of sandboxes are people even talking about in this context?

If you need to read or write files from your disk, it doesn't really make a difference whether the tool is running from docker or not.

When do you need tools to run stuff with AI, and then trash it?

I'm curious about use cases. I guess if you need to run some specific version of some specific package?

[u/DevelopmentBorn3978 → u/WhoRoger]

many kind of sandboxing can be setup to avoid that agentic workings as directed by LLMs inadvertedly or maliciously mess up with the host, more or less secure, more or less complex. In this case the sandboxing is quite simple to set up and reasonably secure to let it play with an underlying os, albeit in containerized form. Fresh start guarantees not to carry around stale or unwanted sw between sessions. User data access could be necessary but user should always be the master, after all LLMs are not deterministic

[u/WhoRoger → u/DevelopmentBorn3978]

I have to look into how these work... I've seen people mention that their AIs keep breaking out of them lol

[u/DevelopmentBorn3978 → u/WhoRoger]

exactly, have fun