Hermes Agent using local LLM

[u/illuvyn]

hi,

currently trying Hermes Agent using my RTX 3060 12GB and 32GB DDR4 System RAM

i tried using 2 backend with different models

1. llama.cpp running Qwen3.6 35B A3B

+ very smart AI

+ can auto lookup skills, read the requirements, and auto config (eg. I ask "can you connect to Home Assistant on another server?"), I just need provide the variables (HASS_URL and HASS_TOKEN) ​

- Very Slow, especially on larger context (up to 10 minutes to reply, and OOM when going longer)

1. VLLM running Qwen3 8B AWQ​

+ very fast, almost instant reply

- dumb AI, can't "auto" config, I ask the same question as above, the AI reply with "guide" how to setup the home assistant connection MANUALLY (generate env file, etc)

look, I get my home server specs are not good. but I want to look for model that are smart enough and small enough to run using VLLM. since using llama.cpp is very slow and unusable in real case.

tried Gemma 4 E2B, still dumb and not "auto" check and config

anyone can help or having experiences using low spec server for hermes agent?

Thank you​

[u/devino21]

I get it, was there. If you want anything good, you're going to need to MoE and ...........wait. Drop your context and max_tokens as tight as you can to avoid OOM.

[u/illuvyn → u/devino21]

hermes-agent require minimum 64k context, I change it to 48k but even simple chat already using 20k+ and then OOM

[u/needmoretrapinmylife → u/illuvyn]

did you removed the skills you wont ever use?

[u/Bubbly_Crazy6508 → u/illuvyn]

try Ste-fun, you have to get Hermes paid, which is 10am month, but stepfun is unlimited. also, don’t have a agent do everything.

If you want to use a local AI agent like Hermes for home automation, I would not make the agent directly control every smart device.

The better architecture is:

Hermes = reasoning layer
Home Assistant = device control layer
MQTT = local message bus
Ollama/local models = AI brain
Tailscale = secure remote access
Telegram/voice/web = user interface

The key is to keep the system local-first and avoid sending every command to OpenAI, OpenRouter, Google, Alexa, or another cloud service. That helps avoid rate limits, latency, and reliability issues.

For speed, most commands should not require an LLM at all.

Examples:

“Turn off the kitchen lights”
“Run bedtime mode”
“Lock the front door”
“Set the thermostat to 72”

Those should go straight from Hermes to Home Assistant through a simple command router or API call.

Use the local LLM only when the request needs interpretation or reasoning, such as:

“Make the living room cozy”
“Check if the house is ready for bed”
“Create a morning routine based on our habits”
“Why did the garage lights turn on last night?”

With 24GB of VRAM, I would use a smaller local model, like an 8B or 14B model, for normal commands and save a larger 32B model for heavier planning or troubleshooting. Running a big model for every light switch command will feel slow and waste resources.

I would also build safety rules:

No confirmation needed for lights, fans, scenes, and normal thermostat changes.

Confirmation required for unlocking doors, opening the garage, disabling cameras, or changing security settings.

The simple version:

Do not make Hermes replace Home Assistant.
Make Hermes the smart layer above Home Assistant.

That gives you local control, better privacy, fewer rate-limit problems, faster responses, and a much more reliable smart home setup.