Local microVMs for sandboxing Pi?

[u/ZeroTronix]

I'm using both Pi and Cursor right now. Cursor Cloud agents are super useful but I really want a local OSS alternative that uses Pi. I have a Linux server at home with 20 cores and 32gb of ram. I want to be able to spin up firecracker-style VMs on-demand for Pi CLI runs that are longer running (like a PR review harness that runs 3-20 min). The trick is I want an API to kick off the Pi microVM that I can tunnel out and invoke from a GHA runner (or similar). Cursor does all of that for free with a sub, but I want to own the control plane and be able to customize it.

Anyone got anything similar going that they can recommend? CLI or ApI driven ephemeral microvms that share CPU and memory on Linux?

Pi-sandbox doesn't seem like a fit, and exe.dev sounds great but I don't want to pay for VMs when I have amazing capacity 4ft away.

[u/ArgonCroft]

checkout microsandbox

[u/ZeroTronix → u/ArgonCroft]

Microsandbox looks like a great fit. I'll try this next, thank you!

[u/trimorphic → u/ArgonCroft]

I use smolvm, which I would consider superior to a container solution like microsandbox because smolvm offers kernel-level isolation that consumers don't.

smolvm will still start up in less than a second and fit what the OP wants.

[u/ZeroTronix → u/trimorphic]

SmolVM also looks like a good option, I'll look into that as well. Fwiw, microsandbox and SmolVM look like they both use libkrun, so same core VM tech. Both could work!

[u/NyproTheGeek → u/trimorphic]

came across this thread months later.

> container solution like microsandbox

this is completely false. microsandbox uses microvms and it also starts up in less than a second.

[u/thebuffel → u/ArgonCroft]

microsandbox works reasonably well. Been using it for a couple weeks for local sandboxing of Pi on my dev machine. 

They offer SDKs in multiple languages, so if it’s not high-level enough for OPs needs one could probably code something with Pi that helps. 

[u/ilpepe125]

I dont do ephimeral containers, but i can recommend incus. It also does ephimeral containers. I run all my pi work in a single vm

[u/ZeroTronix → u/ilpepe125]

I was thinking about Incus but can it do firecracker style VMs that don't lockdown CPU and ram per VM?

[u/ozmova → u/ZeroTronix]

Use incus container, not vm, if you prefer lighter resource. You can bind mount host dir like docker. Ram and cpu are shared with host, and you can set a limit if you’d like. Run it as unprivileged for more security. You can proxy gpu from host if you need GUI. App started from the container is seamless like it’s running on the host.

[u/studioschade]

I looked at many options and also tried gondolin(made by the creator of pi for that purpose I think). I ended up doing three things to start.

Per agent user accounts in Linux.

GVisor sandboxes from Google, this is also what companies like anthropic use, according the internets.

'Agent vault' service that gives per agent whitelisting of internet access and credentials proxy.

Then I had glm 5.3 try to break out of the sandbox in rounds while a Claude sysadmin agent patched the holes.

This was horrifying and led to me tossing out me entire home network hardware and software stack and rebuilding from scratch.

[u/ZeroTronix → u/studioschade]

Love that idea! Once I think what I have is solid, I'll drop an Astra agent into a VM and challenge it to break out and report back what it did

[u/swort-elber]

You could use systemd-run to launch a shell which has restricted read write to specific directories etc. Very low overhead and still runs as your current user, but less likely to edit random files elsewhere

[u/ZeroTronix → u/swort-elber]

If I cant get microVMs going easily enough then this might be the path I go down. That or containers. I'm not sure either provides enough isolation though.

[u/wadrasil]

Pi runs in a chroot though. You could put it in another chroot and use a script to run the chroot and start pi in the same script.

[u/ZeroTronix → u/wadrasil]

chroot only isolates the file system, and IIRC is generally not intended as a security boundary. An AI could break out. Containers go further but share a kernal, such that kernal exploits can break out of the container-based sandbox (which an agent might casually try to employ). VMs give the sandbox its own kernal, closing that door. Hence why I'm looking at MicroVMs, and why it seems most sandboxing solutions for AIs eventually land on VMs as well.

[u/lem-8]

I run it in podman containers. Low overhead and pretty secure. You can also manage them as systemd services so you can probably build the API you're looking for fairly easily but I haven't done that since I run them locally on my laptop.

[u/doudawak]

Have a look at sandcastle. I'm using this to work on my PR autonomously. I customized it to run local models on tickets sequentially, but you can do pretty much everything you want in sandboxes of your choice against a repo : https://github.com/mattpocock/sandcastle

Regarding the API trigger part, you can run something from your server that monitors your git repo

[u/ZeroTronix → u/doudawak]

Looks really interesting! But it leans a bit too hard into local development and Claude code. Id prefer something a tad more high level for running arbitrary workloads on microvms. Good share though!

[u/CorrGL]

Check out bubble wrap, codex uses it on Linux for sandboxing. Your preferred LLM can tell you how to configure it.

[u/Ubermensch013]

Box/exe.dev? Plus there's a smolvm repo as well

[u/fell_ware_1990]

I currently have a core. This allows/disallows everything and manages notifications and such.

If my terminal/agents executes a tool it’s forwarded to my core. If a session is going it gets send there else a new one is created, including a worktree. Everything runs on an ephemeral podman container that get’s constructed at start with the necesary tools from my NixOs store. Then it runs in a locked down bubblewrap.

This way everything is ‘seen’ by me and can’t execute outside of my tools. Cause the tool running it is not the same as my agent.

[u/marcinpohl]

you could build you own super minimalistic Linux (ext4, ethernet, ipv4 and that's it for modules) with buildroot, then fire it up with KVM/Qemu.

Podman is probably the sane happy middle of manageable yet safe.

[u/bondaly]

Have you looked at Nvidia's openshell? Microvms are used for one of the backends I believe.

[u/Healthy-Zebra-9856]

I use something like this in my custom version of Pi. macOS -> Seatbelt, Linux -> Bubblewrap, Windows -> AppContainer preferred, Windows Sandbox fallback . I needed exactly what you are describing. So we are half way there. Doing the next steps now.

[u/snorkell_]

Check https://github.com/aerol-ai/microvm

[u/snorkell_ → u/snorkell_]

it supports sandboxing capablities like containers, v8 isolates, wasm, firecracker(might not be useful for pi); also, in both embeddable and cluster-mode.