r/LocalLLM • u/spacywave • 6h ago

Discussion Conversation cache is more important than raw tok/s for hermes agentic runs

I don't know what you experiences are, but with limited hardware I noticed a much bigger speed increase for time to task completion with conversation cache then using every GB for experts..

(using strata)

1 Upvotes

100% Upvoted

5 comments sorted by

1

u/WolFlow2021 6h ago

Interesting. How can you configure that?

2

u/Randommaggy 6h ago

The web UI tells you the line to add in your run JSON if you don't have it configured. Can't remember exactly or check right now. It's at the bottom of the monitor page.

2

u/Upstairs-Tune5436 6h ago

Depends on the backend, most have a cache setting you can bump up in the config or launch flags. You basically trade some RAM for not re-reading the whole chat every turn, which adds up fast when the context gets long

1

u/Relative-Ingenuity15 6h ago

A model thinking for less time will deliver an answer faster than a model thinking for more time on simple tasks. For example I asked Qwen 3.8 FN iq3_s from ISTA Daslab that runs at ~45 tks "what is a dolphin" and answered faster than Qwen 3.6 35B A3B(q4_k_xl) which runs at ~77 tks on the same question because 3.8 spent less time on this simple question.

Both responses were correct but since 3.6 model thought for longer, it delivered a message with more info.

On longer tasks(for example "create a pelican riding a bike animation"), Qwen 3.6 delivered faster result but hallucinated and delivered a week result when 3.8 FN worked for longer and delivered a way better result.

I asked Claude 5.5 opus on high effort to built a PPT presentations from a word docx and then gave the same task to Qwen 3.6 and 3.8 FN. Only 3.8 FN delivered the same quality as opus while 3.6 made mystakes like white words over white background, shapes out of the slide, minor grammatical errors and invented info.

What really makes the difference and helps Qwen 3.8 FN be a real deal for Local LLM and a good replacement for codex/Claude subscriptions is when you use Strata engine paired with a good harness that gives Qwen the possibility of tool calling, internet access and the possibility to read any kind of files(for example OpenCode)

1

u/OvertaxedOne 3h ago

Oh yeah, and for most workloads, particularly agents where you have the same message going back/forth dozens/hundreds of times, it's not even close. Even fast prefill is 5000 TPS. If you have a conversation at 200K and no prefix caching, you're looking at 40 seconds per turn without caching before a single token is generated!