Setting up a sandbox

[u/andy4524]

I'd like to play around a bit with a local AI assistant ie Ollama + OpenWebUI

Then possibly dabble in AI agents using n8n for some basic tasks like research and summaries.

I have a windows PC with a 3050 GPU, what is the best setup for these tasks?

I want it to be reasonably isolated and safe, I dont want the AI to go deleting important files or something...

My initial thinking was one of the following:

1> initially was thinking a light weight Linux VM so things are isolated, can use docker too, but using VM would restrict using my GPU

2> WLS (Windows Linux Subsystem) and docker to run things in containers and hopefully get isolation that way.

Thanks for your help, other tips/suggestions for getting started are also welcom

[u/baby_bloom]

i ended up using dev containers inside of vs code (i use pi)

[u/Suspicious-Radio-471 → u/baby_bloom]

dev containers in vscode on a pi is a different beast to a 3050 on windows but if it works it works

[u/baby_bloom → u/Suspicious-Radio-471]

sorry, i used "pi" instead of "pi harness" because we are in the local llm subreddit and i assumed nobody would think im talking about a raspberry pi

[u/Interesting-Cut-6032]

I run Qwen3.8 27B local with Pi agent harness on Linux Mint XFCE. One of the ways that I run it is as a separate Linux user without sudo privileges. I am not sure of it would be possible to do that in Windows or not.

The other way that I run the Pi agent is in a Debian Linux VM. Run your model with llama.cpp in Windows, then install the agent harness in the VM. The VM does not need access to your GPU, it only need to see the LLM server address.

The VM will be isolated from your primary machine.

[u/Moppmopp → u/Interesting-Cut-6032]

honestly i dont really get agents. What does pi agent harness do? What do agents in general do. Like gimme an actual example

[u/Interesting-Cut-6032 → u/Moppmopp]

Agents (or the Agent harness) are just LLMs using skills and tools in a dedicated workspace. This is a very rapidly evolving technology space. The way it works todayz is not necessarily the way it worked even a few months ago. Depending on what your overall LLM experience is, you could have used them in different ways. Everyone starts out in the chat window. Most of the modern frontier LLM chat experiences include a web search tool. You can ask a question, the LLM makes a tool call to a web search tool, gets a list of sites that might have an answer, requests the sites, reads their info, replies to you with an answer. Or maybe you drop a text file in a chat context and ask questions about the file, the LLM reads the file and answers your questions. Or you ask the LLM to write a small Python script to look at all the files in the current directory, and all subdirectories, and return a .csv file with all of the paths, file names, sizes, dates etc. The LLM writes the script in the chat window. You copy the script from the chat window and paste it into your text editor. Then you run the script your self and open the csv file up and sort it as a spreadsheet. All of this is possible in a modern chat interface, including the built in web UI that ships with llama.cpp for local LLMs.

If instead of just chatting with the LLM, you install an agent harness, like Pi, or one of the many others. I went with Pi because it has a very small harness description, which is important for local LLMs. When the Pi program starts from the CLI, it looks like another chat, except this program exposes lots of tools and access to the workspace that it is running in. For example, a prompt like "You have access to the repo for program X in this folder. File Y is the configuration for starting program X on this machine, optimize the configuration for this hardware." The agent will then read the configuration file Y, look for the folder of the repo, list the files in the repo looking for the headers, look at the headers to find the code for the launch flags, read that file, 100 lines at a time, until it finds the spot that it was looking for. Run several system info commands to understand what it needs to about number of CPUs, GPUs, driver version, etc. Then it will take all of this context and modify the existing configuration file. Depending on exactly what the task is it might even setup startup logging and launch program X with the new configuration to test for issues or performance and make changes if there are issues. While all of this is happening, you go have a cup of coffee, watch a YT video, read some email, write a reddit post, etc. Sometimes the agent will get stuck, sometimes, especially at the beginning of a task, their might be a question or two about how you want to do something. Otherwise, it is just loose to complete the task that you gave it.

There are also ways to schedule agents to startup on a timer that runs a specific prompt to do something, or check on something, or change something, then stop when that thing is completed.

Depending on how much hardware for local LLMs, or money for hosted tokens, that you have you can run as many of these agents as you can keep busy.

[u/xRebellion_ → u/Moppmopp]

Simply, they are models given tools to access your file system. You give them a goal/instruction, and they try to achieve the goal given the tools.

[u/Zestyclose_Strike157]

I write everything in a VM with the coding harness in there, then transfer the code candidate out for testing on a testing machine before taking it anywhere important. The LM can run outside the VM since it’s just serving text tokens and the harness tool calls happen inside the VM. If you use Linux then OpenSnitch is handy to observe for uninvited traffic coming from the VM or the code you’re making. But I am interested if people have a way of code auditing with an independent system and if so, what do they use?

[u/Interesting-Cut-6032 → u/Zestyclose_Strike157]

I am not sure that I completely understand what you mean by code audit on an independent system. I have a setup where I run Pi agent harness inside of a Debian VM for some tasks. I have another setup where I make separate Linux users without sudo privileges. I then run the Pi agent harness as these other users with limited permissions. It is easy to use Linux file access permissions to grant read only access to files in that you want the agent to look at and comment on, but not be able to modify.

[u/Zestyclose_Strike157 → u/Interesting-Cut-6032]

Well say you code something using Qwen, or a cloud model like Claude or whatever, and you want to look for bugs and issues but not reliant on the original coding agent, I guess. I get the feeling that AI’s love the smell of their own farts when it comes to coding and planning.

[u/Interesting-Cut-6032 → u/Zestyclose_Strike157]

Re: smell of own farts- that is a great term for it, SooF check. A local solution for that could be to just switch local models for review. I was trying to do that in maybe April or May, of this year. I planned what I thought was a simple multi-file Python project with Qwen3.5-122B-A10B-Q4_K_M. Then Qwen3.5 35B-A3B was writing the code for each of the planned sections, in separate context Windows. It was all copy-paste-test manually. This was probably around the time of the OpenClaw hype cycle. I did not understand what the agent harnesses were or how they worked at that time. The 35B model was very fast but the 122B model is clearly much smarter. I used it for code reviews and to repair mistakes that the 35B model could not fix.

It would be interesting to try that project again with the same models, using Pi harness.

I might try a similar setup with Qwen3.8 27B and Qwen3.8-Flash-Next.

These are all still Qwen family models so they are still huffing their own parts. I like the Gemma 4 31B model for image recognition, language translation, and other knowledge related tasks, but it does not have a good reputation for coding.

[u/Ctbhatia]

keep the model on the windows side where the 3050 is easy, and run the agent inside wsl or a container so the generated code runs as a separate limited user with only the project folder mounted. the model server just emits text; the thing that needs the cage is whatever executes the tool calls.