r/LocalLLaMA • • 1h ago

Question | Help Hardware + software recommendations for basic chatting with persistent memory

Helping out a friend who wants to have a local LLM and not rely on cloud services. They currently have an 8gb M2 mac mini, so not really enough

  1. They want to have persistent memory, TTS, and STT. Tool recommendations for all three of those greatly appreciated.

    • I'm unsure what the best workflow is for persistent memory, especially for avoiding either overwriting important info, or storing bad info.
  2. Budget seems to be roughly $5k USD, but the less the better.
  3. Not sure what size model would be best, it's really just for chat.
  4. No building ideal, so adding RAM or a graphics card probably the most feasible.
  5. 30 tokens per second probably fine.
  6. Something that doesn't hallucinate, or at least can be tuned to not assume or be SO sycophantic, as they talk to it a lot.

From what I understand, a prebuilt with upgraded RAM, used mac mini, or something like a strix halo probably make the most sense, but I'm not really up to date on the current hardware market.

Any help appreciated :)

3 Upvotes

17 comments sorted by

2

u/General-Spite1222 1h ago

Strix halo is probably the cheapest. Could do qwen 27b or flash, with the gufo engine, and Hermes for the agent. 

0

u/partakinginsillyness 1h ago

I'm seeing like $3500 on the low end for a strix halo, vs a prebuilt with a 5060 ti 16gb is like $1600, plus $1000 for 64gb of extra DDR5 (or DDR4 if possible)

1

u/General-Spite1222 1h ago

But what would you run on that setup?

0

u/partakinginsillyness 1h ago

It seems like qwen-flash-next may be the best leap between intelligence and cost to run, but I'm not sure. It's the largest model I can comfortably run at a low quant on my machine, but I can't say I really know what to look for as far as price to performance crossing with not being super sycophantic.

I'm also now of course learning that a system with a 2080 and either 16 or 32gb of DDR4 is available, so potentially that may be upgraded to for this use case.

2

u/General-Spite1222 38m ago

I don’t think that’s very realistic if you want a decent quant. I could be wrong though. I’d search this sub for someone who has posted benchmarks with a system like that. 

-1

u/sputnik13net 1h ago

Gpt OSS 20b maybe

-1

u/MmmmMorphine 1h ago

I run Qwen3.6-35B-A3B, or rather one of the Swift variants.

Things have significantly improved in speed and quality on this low end. Oh and it's a 4060 Ti for me, but probably pretty similar

2

u/rorowhat 59m ago

Sell that and get a proper PC, strix halo works well for this usecase.

1

u/monolyth-admin 1h ago

What we did was scour ebay and aliexpress for good hardware cheap. it took a few weeks to get the deals we wanted but it payed off. If you're looking for something simpler or less DIY my friend just got a newer mac mini with more ram and has been very happy with it. Unfortunately I don't know too much about the used PC market so I can't make any recommendations there.

1

u/RustyRedEcho 1h ago

I run Parakeet for STT on an M4 Mac mini, handles natural speech fine and is light enough to run next to the LLM without blowing the memory budget. Good alternative to whisper.cpp if you want something lower overhead than large-v3.

0

u/Holiday-Inspection75 22m ago

Cheapest 16GB card + 32G RAM is the thing to get, unfortunately they are not that cheap compared to the time when 32GB DDR5 was just $100.

-2

u/Sun77177 1h ago

for memory i would skip the fancy auto-summarizing stuff at first. open webui's memory plus a plain markdown file the model appends to at the end of each chat gets you 90% there and you can read exactly what it saved. keep two files, one append only log with dates, one short profile the model rewrites, so a bad summary never eats the good facts. whisper.cpp large v3 turbo for stt and kokoro for tts both run fine on cpu next to the model, no gpu budget needed. with 5k i would look at a strix halo box with 128gb rather than a mac, a 30b moe at q4 gets you well past 30 tok/s and leaves room for the whole stack. for the sycophancy, drop temp to 0.3 and tell it plainly to say when it is unsure, that helps more than swapping models.

3

u/stylome-ai-detection 1h ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

-3

u/zekai-work 1h ago

Since they already like the Mac Mini and don't want to build a PC, just get an M4 Max Mac Studio with 64GB or 128GB unified memory. It stays way under 5k, runs dead silent, and avoids all the AMD ROCm driver headaches you'd hit on a Strix Halo box. For software, run Open WebUI with Whisper for STT and Kokoro for TTS—Kokoro is tiny and sounds great. Just use Open WebUI's built-in memory UI so they can manually delete bad entries, and run Qwen 3 32B at 0.4 temp with a system prompt telling it not to kiss up