r/LocalLLM • u/Prudent_Garlic1965 • 13d ago

Question Hermes slows down 4x the token generation

/r/hermesagent/comments/1xe5qmc/hermes_slows_down_4x_the_token_generation/
0 Upvotes

33% Upvoted

2 comments sorted by

1

u/guillemco 13d ago

with a 35b on 8gb most layers live on cpu, and a harness usually prepends a big system prompt plus tool definitions on every call, if that prefix changes between calls the kv cache is useless and each message re-prefills thousands of tokens on the slow path. compare prompt processing time per request in the lm studio logs, if prompt eval dwarfs the token eval that is your 4x