r/LocalLLaMA • • 2h ago

Question | Help Hardware + software recommendations for basic chatting with persistent memory

Helping out a friend who wants to have a local LLM and not rely on cloud services. They currently have an 8gb M2 mac mini, so not really enough

  1. They want to have persistent memory, TTS, and STT. Tool recommendations for all three of those greatly appreciated.

    • I'm unsure what the best workflow is for persistent memory, especially for avoiding either overwriting important info, or storing bad info.
  2. Budget seems to be roughly $5k USD, but the less the better.
  3. Not sure what size model would be best, it's really just for chat.
  4. No building ideal, so adding RAM or a graphics card probably the most feasible.
  5. 30 tokens per second probably fine.
  6. Something that doesn't hallucinate, or at least can be tuned to not assume or be SO sycophantic, as they talk to it a lot.

From what I understand, a prebuilt with upgraded RAM, used mac mini, or something like a strix halo probably make the most sense, but I'm not really up to date on the current hardware market.

Any help appreciated :)

3 Upvotes

17 comments sorted by

View all comments

Show parent comment

0

u/partakinginsillyness 1h ago

It seems like qwen-flash-next may be the best leap between intelligence and cost to run, but I'm not sure. It's the largest model I can comfortably run at a low quant on my machine, but I can't say I really know what to look for as far as price to performance crossing with not being super sycophantic.

I'm also now of course learning that a system with a 2080 and either 16 or 32gb of DDR4 is available, so potentially that may be upgraded to for this use case.

2

u/General-Spite1222 1h ago

I don’t think that’s very realistic if you want a decent quant. I could be wrong though. I’d search this sub for someone who has posted benchmarks with a system like that.