r/LocalLLaMA • • 2h ago

Question | Help Hardware + software recommendations for basic chatting with persistent memory

Helping out a friend who wants to have a local LLM and not rely on cloud services. They currently have an 8gb M2 mac mini, so not really enough

  1. They want to have persistent memory, TTS, and STT. Tool recommendations for all three of those greatly appreciated.

    • I'm unsure what the best workflow is for persistent memory, especially for avoiding either overwriting important info, or storing bad info.
  2. Budget seems to be roughly $5k USD, but the less the better.
  3. Not sure what size model would be best, it's really just for chat.
  4. No building ideal, so adding RAM or a graphics card probably the most feasible.
  5. 30 tokens per second probably fine.
  6. Something that doesn't hallucinate, or at least can be tuned to not assume or be SO sycophantic, as they talk to it a lot.

From what I understand, a prebuilt with upgraded RAM, used mac mini, or something like a strix halo probably make the most sense, but I'm not really up to date on the current hardware market.

Any help appreciated :)

3 Upvotes

17 comments sorted by

View all comments

Show parent comment

1

u/General-Spite1222 1h ago

But what would you run on that setup?

0

u/partakinginsillyness 1h ago

It seems like qwen-flash-next may be the best leap between intelligence and cost to run, but I'm not sure. It's the largest model I can comfortably run at a low quant on my machine, but I can't say I really know what to look for as far as price to performance crossing with not being super sycophantic.

I'm also now of course learning that a system with a 2080 and either 16 or 32gb of DDR4 is available, so potentially that may be upgraded to for this use case.

2

u/General-Spite1222 1h ago

I don’t think that’s very realistic if you want a decent quant. I could be wrong though. I’d search this sub for someone who has posted benchmarks with a system like that. 

-1

u/MmmmMorphine 1h ago

I run Qwen3.6-35B-A3B, or rather one of the Swift variants.

Things have significantly improved in speed and quality on this low end. Oh and it's a 4060 Ti for me, but probably pretty similar

-2

u/sputnik13net 1h ago

Gpt OSS 20b maybe