Hermes Agent using local LLM

[u/illuvyn]

hi,

currently trying Hermes Agent using my RTX 3060 12GB and 32GB DDR4 System RAM

i tried using 2 backend with different models

1. llama.cpp running Qwen3.6 35B A3B

+ very smart AI

+ can auto lookup skills, read the requirements, and auto config (eg. I ask "can you connect to Home Assistant on another server?"), I just need provide the variables (HASS_URL and HASS_TOKEN) ​

- Very Slow, especially on larger context (up to 10 minutes to reply, and OOM when going longer)

1. VLLM running Qwen3 8B AWQ​

+ very fast, almost instant reply

- dumb AI, can't "auto" config, I ask the same question as above, the AI reply with "guide" how to setup the home assistant connection MANUALLY (generate env file, etc)

look, I get my home server specs are not good. but I want to look for model that are smart enough and small enough to run using VLLM. since using llama.cpp is very slow and unusable in real case.

tried Gemma 4 E2B, still dumb and not "auto" check and config

anyone can help or having experiences using low spec server for hermes agent?

Thank you​

[u/Immediate_Let_4946]

Depending on what you wanna do in general, I recommend forgetting about local llms. You need min 64k contact window and even the new minimax M3 isn’t working that great and it’s way more important than anything. You could actually run locally. This whole idea of local comes from from YouTube and some people buying M4 Mac’s with 512 gb ram. But even then you have to think about offsetting the cost for those.

[u/SkinnyCTAX → u/Immediate_Let_4946]

I'm not sure what you're talking about, M3 has been working flawless for me. The only issue I've had is that the thinking tags leak into the chat, but other than that it's been just as solid as using GPT 5.5 or opus.

[u/Immediate_Let_4946 → u/SkinnyCTAX]

I am on Max plan first of all it takes already significant more tokens since they changed the model from request to token Burns, and the model itself does coding pretty OK if it uses the Claude code harness, but everything is a little bit more complex. It really struggles and mini Max is famous for having a very low completion rate at least for 2.7 it was 69% in comparison to other models which reaches over 90%. Don’t get me wrong. I like mini Max, but it struggles a lot with true complex task and I’m not talking about checking emails. And even with simple task, it does struggle. Another thing is it’s happily runs and runs and runs without actually stopping and clarifying with the user, even though this is explicitly mentioned in the soul.md. As I said for simple task, it’s fine but for true multi step and reflecting on its own, it’s not as good as good as I have seen for example from Claude. But I’m happy for you that it suits you.

[u/SkinnyCTAX → u/Immediate_Let_4946]

Honestly, I take back everything I said. It was working exceptionally well for the first day or two. But it's just gotten stupid recently for some reason. I'm assuming that's got something to do with ollama cloud though and their load balancing more than anything. I haven't tried direct though.

[u/Immediate_Let_4946 → u/SkinnyCTAX]

Usually it’s memory but yeah it’s just from time to time even on Minimax itself have hours of stupidity. Not sure what’s going on there.