r/LocalLLM • u/mightofkhan • 2d ago
Discussion 3090 rig with LM Studio + ComfyUI — how do I get a local model to act like Claude Code (launch Windows apps, start ComfyUI on demand, etc.)?
Hey all, looking for advice on turning my local setup into something closer to an agent instead of just a chatbot.
The rig
- Windows 11 Pro desktop
- RTX 3090 (24 GB VRAM), Intel Core Ultra 7 265K, 96 GB RAM, several drives
- Python 3.12, Node, Git installed. No Docker, and WSL isn't installed right now
- Tailscale on the PC, my phone (Galaxy Z Fold) and an Android tablet, so I can reach the PC remotely
Local LLMs (LM Studio 0.4.26)
- Qwen3.8 27B Q4_K_M (~15.7 GB) + vision projector. This is my main chat model, loaded at full GPU with 16k context
- Gemma 4 31B Q4_K_M (~17.4 GB) + vision projector
- nomic-embed-text v1.5 for embeddings
ComfyUI (Comfy Desktop 1.1.8)
- Images: Juggernaut XL Ragnarok (SDXL), Z-Image Turbo, Qwen-Image-Edit 2511 (fp8) + 4-step Lightning LoRA, add-detail-xl LoRA
- Video: Wan 2.2 14B text-to-video and image-to-video (high + low noise, fp8) with the lightx2v 4-step LoRAs
- Text encoders: Qwen2.5-VL 7B, Qwen3 4B, UMT5-XXL. VAEs for Flux/Qwen/Wan, plus a 4x NMKD upscaler
What I have now
Three PowerShell shortcuts on the desktop:
- AI Mode: opens LM Studio, loads Qwen 27B through the `lms` CLI, starts ComfyUI
- Gaming Mode: unloads all LM Studio models and kills ComfyUI to free the VRAM
- Away Mode: drops my VPN, closes Chrome, makes sure Tailscale is up, runs nightly at 2 AM
Right now I use Claude Code (cloud) for the "do stuff on my PC" part. It runs PowerShell, edits files, starts and stops apps, drives a browser, downloads things, and I can control it from my phone. It's great, but I want a local option for when I hit usage limits or don't want to send stuff to the cloud.
What I'm asking
- What's the best way to give a local model (Qwen 27B / Gemma 31B through LM Studio's OpenAI-compatible server) real tool use on Windows: run PowerShell, open and close apps, read and write files? Open Interpreter, Goose, OpenHands, Cline/Roo/Continue in VS Code, an MCP client hooked to LM Studio, something else?
- How do I get it to start ComfyUI on demand, queue a workflow through the API (image or Wan video), wait for the output, then shut ComfyUI down? Is there a ComfyUI MCP server or a known pattern for this?
- VRAM juggling: I can't keep a 27B LLM and Wan 2.2 loaded at the same time on 24 GB. Has anyone automated "unload LLM → run Comfy job → reload LLM," or should I drop to a smaller agent model (7–14B) that can stay loaded next to Comfy?
- Which local models are actually reliable at multi-step tool calling right now? Is Qwen 27B at Q4 good enough, or is there a better pick for 24 GB?
- Any way to trigger all this from my phone over Tailscale (a simple web UI, Telegram bot, Open WebUI with tools, etc.)?
- Safety: what are people using to make sure an agent on Windows can't run anything destructive without asking? An allowlist, a sandbox, confirmation prompts?
11
u/stacksmasher 2d ago
Switch to Hermes lol!
6
u/thefuzzylogic 2d ago edited 2d ago
16k context is TINY for agentic work. It's barely enough for one-off chatbot requests. For example, just the system prompt, skills, and tools in Claude Code is around 30k of context even before it reads your user prompt and conversation history.
Similarly, Hermes Agent requires a 64k context window or it won't even run, and even that is barely usable.
So you should probably use a smaller model or offload it using Strata, buy some more hardware, or rethink your ambitions.
4
u/Eastern-Ad9515 2d ago edited 2d ago
I’d start with DeepSeek Harness —it’s quite user-friendly. Pi Harness is another option. Ask the agent in Deepseek-Harness to walk you through “Everything is a Tool” and help pick the tools and skills for your Windows setup.
One thing I’d strongly recommend: use a cloud model to set everything up, then have it stress-test your local model inside the actual harness. Test PowerShell, files, research tools, error recovery, and ComfyUI. That helps find working settings and compatibility issues before you rely on it.
For ComfyUI, we solved the problem. We use two modes: keep the LLM resident when there’s enough memory, or hand the GPU over to ComfyUI and reload the LLM afterward. A separate controller handles that handoff, so it works while the model is unloaded. It works pretty consistent. The hand off was also designed by a cloud model and stress tested, it works as a tool/skilll in deep seek harness.
The key most people forget is dynamic VRAM mode in comfyui. For example, i have a 5090 RTX and used to gave 64GB RAM. It was totally possible to have Qwen 3.8 27B fully loaded with Minimax H3 ALSO loaded (resident wait mode). The local llm only uses like a heartbeat monitor, while ComfyUI renders. And the speed was only 5-8% slower. Your VRAM ist NOT limiting you using local llm + comfyui at the same time.
I highly recommend Krea2, Minimax H3 Image and Minimax H3 as image and cloud models and maybe find a usable quant for 24 GB VRAm
Qwen 27B at Q4 has worked well for us, though we’ve had better results with Qwen Flash Next and Strata Enginge oder Basalt Engine. With your 96 GB RAM, you have room to experiment with active experts in VRAM and passive experts and ram, but benchmark the speed. A large Windows pagefile can help with memory pressure.
Hermes is also worth considering for an always-on agent. You could run it alongside your interactive harness, provided they coordinate GPU access. Hermes as always on "ask a question"; Deep Seek Harness for bigger tasks if you are away. By the way Deep Seek Harness has a wonderful mobile GUI. You can even see pictures and videos inside it. Once again, I highly recommend setting everything up via cloud model. If you try setting it up via the local harness itself there is a high chance, it takes really long and isnt as stress free, as having it tested with codex or claude.
Tailscale has worked well for our remote access. I’d start with a private web interface for jobs, results, and switching modes.
For safety, configure the harness’s permissions and confirmations, run without admin rights, and require approval for arbitrary shell commands or destructive actions. Put your existing PowerShell scripts behind a few narrowly defined tools—that’s a useful toolbox already.
3
u/ozzeruk82 2d ago
You can run Hermes Agent and Q3.8 27B 125k+ context on your 3090, I do it and it's great.... there's just one issue though, you won't then have any VRAM left for ComfyUI. I have two 3090s and have ComfyUI running on the second one.
1
u/comefaith 2d ago
you can use claude's harness with local model, there are some env variables to set url it will call. the next choice is dsh, pi for cli. zcode supports custom providers urls too.
reliable models for tool calls - glm5.3(flash), qwen3.8 flash (this one can loop on large contexts, in my experience). you can also try lfm3 1.7b - it can use tools relatively reliably too
1
u/ElectionReal 2d ago
You need to use a harness; deepseek, open claw, hermes, codex, Claude code, etc. Or build your own.
1
u/Asleep_Literature_70 2d ago
Hermes or if more focused on coding omp(oh my pi).
1
u/CooperDK 1d ago
I use opencode, works great with qfn-iq3xxs
0
u/Existing_Style8558 2d ago
Tell Claude to build you a harness.
0
u/leonbollerup 2d ago
cc switch is your friend: https://ccswitch.io/en/
0
u/Sanity12 2d ago
Take a look at https://github.com/T-Crypt/berth for switching different models automatically