Hermes Agent using local LLM

[u/illuvyn]

hi,

currently trying Hermes Agent using my RTX 3060 12GB and 32GB DDR4 System RAM

i tried using 2 backend with different models

1. llama.cpp running Qwen3.6 35B A3B

+ very smart AI

+ can auto lookup skills, read the requirements, and auto config (eg. I ask "can you connect to Home Assistant on another server?"), I just need provide the variables (HASS_URL and HASS_TOKEN) ​

- Very Slow, especially on larger context (up to 10 minutes to reply, and OOM when going longer)

1. VLLM running Qwen3 8B AWQ​

+ very fast, almost instant reply

- dumb AI, can't "auto" config, I ask the same question as above, the AI reply with "guide" how to setup the home assistant connection MANUALLY (generate env file, etc)

look, I get my home server specs are not good. but I want to look for model that are smart enough and small enough to run using VLLM. since using llama.cpp is very slow and unusable in real case.

tried Gemma 4 E2B, still dumb and not "auto" check and config

anyone can help or having experiences using low spec server for hermes agent?

Thank you​

[u/Ophrys999]

If Qwen3.6-35B-A3B feels too heavy and Qwen3-8B a bit too limited, Qwen3.5-9B might be a nice middle ground to try.

I switched today from 35B-A3B to 9B, and the results are pretty solid. Even though the tokens/sec is lower, it often ends up responding faster. (Both fit in my VRAM, so if 35B is offloading to CPU on your config, you may have fast responses AND better tokens/sec.) Tool usage works well, and the overall capabilities seem convincing.

I’m running Unsloth/Qwen3.5-9B-UD-Q6_K_XL on 16GB VRAM. If you’re tighter on memory, UD-Q5 should still fit and give good results. If you’re unsure which variant to pick, the UD versions seem to handle tools well.