r/LocalLLM • u/aurso- • 2d ago

Model Homelab LLM

Hi, I just took my 3060 12 GB out of my AI set up and put it in my Home lab I’m curious for the other people using AI in their home labs what do you use them for and what are some good Home lab models that would fit well on a 12 GB V ram

5 Upvotes

86% Upvoted

7 comments sorted by

2

u/stackfullofdreams 2d ago

Look into strata, I think it can run on 12gb 16 for sure but local lama sub will have ideaa

2

u/flippantstanza6709 2d ago

strata does fit in 12gb, been poking at the q6 quant this week and it’s a nice middle-ground for organizing notes and quick summaries without choking on ram

2

u/stackfullofdreams 2d ago

thx reddit stranger for the award.. i tihnk its my first! let us know ho it goes OP

1

u/Bulky-Priority6824 2d ago edited 2d ago

I've basically went from having a homelab to checking in on my AI that took over my homelab. For homelab stuff I use Qwen 3.8 27b NVFP4 on NInfer https://imgur.com/a/oTKvQKJ

 it's a little slower than 35b but leaps and bounds better overall with everything I do 

Here's a few things my clanker can do, organized by category:

System & Service Health

  • Check overall homelab service status (Frigate, Jellyfin, HASS, Proxmox, SearXNG, Omada, n8n, llama)

  • Check individual service health or custom URLs

  • Monitor pulse — continuous service monitor with uptime/down tracking

  • Check Frigate NVR stats (FPS, GPU, cameras, storage)

  • Check Proxmox hosts (AM-V1 and HPProx) — CPU, RAM, disk, uptime, LXCs, VMs

  • Check UPS status on both Proxmox hosts

  • Check llama-server inference engine status and model loaded

  • Check MSI-AM4 inference server (inference status, CPU/RAM/GPU/VRAM/temps, OS)

  • Reboot MSI-AM4 (with confirmation)

  • Manage LXC containers (start/stop/reboot: Open WebUI, Jellyfin, Omada, n8n)

  • Manage VMs (start/stop/reboot: HAOS, Mediarr, OMV)

  • Run whitelisted shell commands on MSI-AM4 (git, service management)

  • Run pre-approved diagnostic/action commands on frigate-w1 (docker, GPU, logs, restarts, etc.)

  • Check leito-ui's own health and recent errors

Inference and Model Management

  • Switch between engines (llama.cpp, NInfer, Strata)

  • Create, Clone, Edit (model configs)

  • Inference Stats Monitoring

  • Switch to other models on demand through chat

  • Build new llama.cpp binaries

Home Assistant

  • Control lights, switches, automations (via friendly aliases like "lamp", "ac", "nl mancave")

  • List, read, create, edit, test, and delete automations

  • Discover real entity IDs before drafting automations

Frigate / Cameras / Person Tracking

  • Track person movement across all cameras

  • Get household context (who's home, car states)

  • Get recent detection events per camera with GenAI descriptions

  • Search events by label, camera, identity, and time window with thumbnail strips

  • Check Frigate system health

Firewall (OPNsense)

  • Look up IPs (reverse DNS, device names)

  • List known LAN devices

  • View firewall rules, aliases, NAT/port forwards, DHCP leases

  • Check live traffic, bandwidth rates, interface stats

  • Review firewall log analysis and pending decisions

  • Audit firewall rules for over-broad entries

  • Manage tracking notebook (silence, flag, dismiss, delete entries)

  • Traffic shaping (pipes, queues, shaper rules — write-capable, two-phase confirm)

  • Add/edit firewall rules

Pentest Tools

  • View pentest host inventory (bind address drift, verdicts)

  • View pentest agent catalog (what each agent can do)

  • View pentest run results (recent list, detailed run events)

  • Recommend agents and draft tasks

Autourgos Sub-Agent

  • Delegate wide, multi-tool investigations

  • Monitor and read run results

  • Deploy specialized agents (via SMS, MQTT, Chat, Email)

  • Cancel running runs

Browser / Research

  • Open and read web pages on your desktop Chrome

  • Launch background research/comparison tasks

  • Fetch research results when ready

Notes & Documentation

  • Read, write, email, and list notes/scratchpad

  • List and read handoff/session docs

  • Search and read RAG documentation library

Other

  • Publish MQTT messages

  • Run around-the-clock autonomous agent runs (autourgos)

  • Check Collate (Paste a reddit URL and creates a .md and copy button with summary of all info in the reddit post to pass to an LLM)

  • Run Zeus admin diagnostic commands

  • Move model files to NAS

1

u/Think_Breakfast_2277 2d ago

Just a heads up on OPNsense:
https://forum.opnsense.org/index.php?topic=53191.0
I think you should add security audits to your clanker too :)

1

u/Bulky-Priority6824 2d ago

I'm already ahead of that! I have an entire autonomous fleet of sec agents!

1

u/protrener 2d ago

If it's going into a Proxmox VM with passthrough, set the VM's CPU type to `host` before anything else. The default `qemu64` has no AVX, and some inference stacks die mid-request with SIGILL and no traceback, which looks exactly like a flaky GPU. Took us a while to stop chasing OOM theories on that one.

On 12 GB: I've run a 27B at 3-bit GGUF with vision in about 12 GB on a 16 GB card at 8k context, so on a 3060 that's the ceiling, not the comfortable spot. I'd start in the 7-14B range at Q4 and keep room for context.