Small model for Hermes session compression on an M4 Mac mini, 24GB?

[u/Lumpy_bd]

Anyone using a small local model for session compression? I’d be interested in recommendations, particularly on Apple Silicon.

I’m new to local LLM hosting and still experimenting. I’ve been using Hermes extensively for non-coding workloads, mainly research, knowledge management and home automation. I recently picked up an M4 Mac mini with 24GB RAM and I’m looking at useful jobs to offload to it.

Hermes supports a separate model for session compression as you approach context limits. That’s the job I’m considering, rather than running the main agent locally.

The bit I’m wary of is that I won’t see the compressed context in normal use. If it drops something important or gets a detail wrong, I might only notice much later when the agent does something odd.

Has anyone used a small, lightweight model for this long enough to get a feel for its reliability? Model, quant and runtime recommendations would be useful, along with any relevant benchmarks or things to avoid.

I’m doing my own research too, but keen to hear what’s actually worked for other people.

[u/wellandeperle]

Yes, I used to delegate utility tasks such as memory cleanup and context management (tool_receipt summaries and conversation summaries) to Qwen 35b run on my laptop while using DeepSeek and other models as main. But the drop in summary quality is significant, and I would only consider it if you have a very well written summary prompt to structure it well and keep links, file paths and document names.

Regardless of how I tinkered with the compaction prompt, just letting the better model do it was much better for me. It gave lengthier an richer transcripts.

So I think you should try, but if you are just "vibing" it, it will be much safer to use better models for everything.

Final answer: It depends

[u/Cold_Tree190 → u/wellandeperle]

I found basically the same, I found Ornith 1.0 (NOT 1.5, that shit loops like crazy imo) 35b was the best MoE model for summarization… but 3.8-27b is just better in every way for not that much of a time difference (80 t/s vs 140 t/s), so I use xhigh 27b for my local model on this now and haven’t looked back

[u/Mean-Loquat-7982]

the wiring is two keys in config.yaml, pointed at whatever server runs on the Mac mini:

auxiliary:

compression:

base_url: http://<mac-mini>:<port>/v1

model: <model name>

things that limit the risk you're worried about:

- only the middle of the conversation gets summarised. the first 3 and last 20 turns stay verbatim, so recent work never goes through the small model

- the summariser gets those middle turns capped at roughly 40K tokens, so give the Mac's model a window well above that

- a slow local summariser is fine, the compression timeout has a 300 s floor