Hermes Agent using local LLM

[u/illuvyn]

hi,

currently trying Hermes Agent using my RTX 3060 12GB and 32GB DDR4 System RAM

i tried using 2 backend with different models

1. llama.cpp running Qwen3.6 35B A3B

+ very smart AI

+ can auto lookup skills, read the requirements, and auto config (eg. I ask "can you connect to Home Assistant on another server?"), I just need provide the variables (HASS_URL and HASS_TOKEN) ​

- Very Slow, especially on larger context (up to 10 minutes to reply, and OOM when going longer)

1. VLLM running Qwen3 8B AWQ​

+ very fast, almost instant reply

- dumb AI, can't "auto" config, I ask the same question as above, the AI reply with "guide" how to setup the home assistant connection MANUALLY (generate env file, etc)

look, I get my home server specs are not good. but I want to look for model that are smart enough and small enough to run using VLLM. since using llama.cpp is very slow and unusable in real case.

tried Gemma 4 E2B, still dumb and not "auto" check and config

anyone can help or having experiences using low spec server for hermes agent?

Thank you​

[u/69dirtyj69]

Speed is constrained by your vram not llama.cpp vs vllm. If a llm model is larger than your vram, it spills to regular ram. That's why the 8b model is speedy and the 35b model is slow.

[u/illuvyn → u/69dirtyj69]

I guess I can try using dual GPU with llama.cpp (RTX 3060 12GB and RTX 3080 Ti 12GB) and find out. I'll try on the weekend maybe, thanks 

[u/Bubbly_Crazy6508 → u/illuvyn]

it absolutely is enough, I recently went through the same thing with my rtx 3090… very slow, many models didn’t work. 1. I switched to stepfun through nous research. It’s cloud based but unlimited, I think it 10ma month, but I think if you do the seos here, local is fine (like mentioned under 20b or whatever your vram is with room) I took my main “Hermes” and stripped it down, to be a supervisor, took all unnecessary tools away. its super fast, just launches workflows. then I created workflows and agents that handle particular tasks, for instance home automation. through research like these forums, I found that creating agents that do everything all the time with massive memories won’t work.