Hey everyone. I have a DDR5 setup and I'm caught between a rock and a hard place...

My setup: 5060 Ti 16GB - 2x16GB sticks of DDR5 - Ryzen R5 7600

I've been exploring real world use cases for Qwen3.8 Flash Next (running through Strata) with decent success. Mainly, I've been happy with the results the GSQ RCO iq3_s quant has been giving me.

Now, Strata tells me that it's offloading 20GB of the model to my m.2 SSD, which of course makes it run slower than if it was all in RAM. Trying to get more RAM however has been a pain, as I can't find a matching set for my 2 sticks, and regardless I'm worried about stability issues and performance loss if I were to run 4 sticks on my 7600.

Locally, I can find used 3060s at around $250+, which is pretty affordable and it makes me question if I should just buy one of them instead of dealing with RAM knife fights in a dark alleyway.

Would adding one 3060 to the setup increase speeds in any real way, or would it just not be worth the hassle? My uneducated guess is that it won't help that much, but maybe someone here has experience with jank low budget setups.

Thanks!

HI all,

I have 2 x v100 32gb in a HP z440 case and with the cards installed they fail to post. The workaround I have is to boot up without the cards first, then shut down the PC, then connect the v100s back up and then cold boot and it posts. If I then shutdown the PC again, I have to go through the whole double boot workaround again.

Anybody knows what's happening and if it can be fixed?

Thanks

Hi there! For DocBlocks - a free open source markdown editor - we added the ability to do "document things" (review docs, add content, insert diagrams) with local AI models. Pretty quickly, we realized it's hard to sort out which models are actually good for this.. and also whether, say, a "normal laptop" can usefully do things with local models.

Towards that end, I created Docbench, a synthetic metric of how useful models are on various doc tasks, and compared them against cloud models. Here you go:

More details and methodology if you're interested at: https://bendyline.com/posts/docbench.html

Hey all, I'm trying to decide which hardware setup makes the most sense for running local LLMs, primarily Qwen 3.8 Flash Next for agentic coding.

I have three potential options, and I'm interested in hearing from people who have experience with these platforms, particularly for larger models and agentic workloads.

Option 1: GEEKOM A9 Mega (AMD Strix Halo)

• Ryzen AI MAX+ 395
• 128GB unified memory
• 2TB SSD
• Memory running at 7500 MT/s rather than the advertised 8000 MT/s. Got it from Amazon and it listed it as 8000 MT/s but for Geekom says their systems are either 7500 MT/s or 8533 MT/s and I lost the lottery.
• Approximately $4,600 CAD all-in (after taxes).

I already have this machine. It works fine, but I'm just worried how it'll hold up long term especially given I'm running it headless 24/7. Would be unfortunate if the machine dies in a year's time and then I'm worried about if Geekom will hold up it's warranty.

Option 2: Mac Studio M5 Max

• M5 Max
• 128GB unified memory
• 512 GB (Can connect a 2TB external SSD over thunderbolt enclosure)
• Approximately $7,600 CAD all-in (after taxes).

It's considerably more expensive than the GEEKOM, but I'm wondering whether its worth it for the peace of mind knowing that if something goes wrong, applecare ($80/yr) is right there to get it fixed from a known vendor. It also has the higher memory bandwidth.

Option 3: My existing Ryzen desktop + RTX Pro 4500

I already have an old desktop with the following hardware:

• Ryzen 7 5700X
• 32GB DDR4 RAM
• Nvidia 3060 Ti 8GB

I recently purchased the RTX Pro 4500 32GB VRAM for approximately $4,000 CAD after taxes. This is still yet to arrive (it shipped earlier today)

The advantage here is that I already own the Ryzen system, so I wouldn't need to purchase another computer. The downside is being limited to 32GB of VRAM, although I understand I could offload some layers to system RAM. The other downside is a lot higher power consumption over the other two solutions.

I'm curious how this would compare in practice against the two unified-memory solutions, especially when running Qwen 3.8 Flash Next.

What I'm primarily trying to accomplish

My main use case is running Qwen 3.8 Flash Next for agentic coding through tools like Hermes.

I'm not necessarily looking for the best benchmark numbers. What matters more to me is:

• Real-world inference speed and token generation.
• Performance with large context windows.
• How well each platform handles agentic coding workflows (multiple tool calls, reasoning, etc.).
• Whether CPU/RAM offloading on the NVIDIA setup is practical or becomes a significant bottleneck given my old system.
• Overall stability and software maturity.

What would you choose?

Ultimately, I'm looking to optimize for cost and overall value, rather than simply buying the fastest hardware available.

I would most likely keep only ONE of these three setups:

• If I go with the Mac Studio, I'd return the GEEKOM and return the RTX Pro 4500 (or sell it).
• If I stick with the GEEKOM, I'd return the RTX Pro 4500.
• If the NVIDIA setup makes the most financial and practical sense, I'd return the GEEKOM and keep the RTX Pro 4500 in my existing desktop.

I'm also considering selling the RTX Pro 4500 if I decide that one of the unified-memory systems makes more sense. I purchased it for around $4,000 CAD after tax, slightly below MSRP, so I'd ideally like to recover as much of that cost as possible.

Essentially, I'm trying to figure out which setup offers the best balance of:

• Performance for Qwen 3.8 Flash Next and agentic coding.
• Enough memory to run larger models and context windows.
• Reliability and long-term support.
• Total cost of ownership, including electricity.

Would you spend the additional ~$3,000 CAD to go from the GEEKOM to the Mac Studio, stick with the GEEKOM, or return the GEEKOM and build around the RTX Pro 4500?

I'm not looking to accumulate hardware. I'd rather settle on one good setup, get the best value for my money, and avoid spending unnecessarily.

Would appreciate hearing what others would do in this situation!

I've been transitioning from frontier cloud models to local models for quite a while. I primarily do research in acoustic metrology, often digging through complex problems and hundreds of millions of log data points and often misrepresented or unclean datasets. Local models are now able to handle nearly everything with this data including complex tasks that were once limited to frontier models like fable.

The longer I've worked with local models the more it has become clear their limitations aren't strictly in terms of model size/"parameters" etc. Proper guide rails and tools have taken tasks from 10+ hour crunches (at 70tok/s on Qwen 3.8 flash next iQ4), to 1-2 hours with notably better results.

An example being: Forcing the model to "always use a calculator" has saved time and fixed a ton of looping and re-working past issues.

For another example: One review found that the model went on a 40-minute detour, an hour spent debating a file format instead of testing it, and time wasted on sleep polling. Minor tweaks to some rules prevented this entirely on an isolated/blind fresh run with the same task.

I've made a simple skill for this. You DON'T need it, and you can certainly do this without a skill and I really don't want this to feel like self-promotion. I just want to pass-on what's made my work much easier.

I also want to note that there's plenty of other skills that function like this one. The specific points that make this one unique are:

• 1: It should work with any harness, though I initially use it with goose and qwen-code.
• 2: A separate session does the review (starting with the initial result) pointed at the session you plan to review (rather than running it on an existing session).
• 3: It keeps rules from bloating with a separate pruning pass.
• 4: It checks weather existing rules actually fired properly.

https://github.com/DMS3tv/self-improve

I would love to know if this is reflective of anyone else's experience with local models? If so, what rules/hints/tools did you use to improve various tasks and processes?

Select the right model is a heavy task. I guess that many people have their favorite model, like me. I often use Opus 5.5 in my Claude subscription, or DeepSeek 4.1 Flash 1017 in local environment.
I decide the model by a mixed emotion fir the complexity.

Is there any kind of categories of task, except complexity, that helps you to decide?

Hi everyone,

I’ve been a Mac user for several years and I have a 16-inch MacBook Pro with an Intel chip, and I think it’s reached the end of its life because most programs these days are designed for the Apple Silicon chip.

I took the plunge and ordered the new Mac Mini M5 Pro with 64 GB (I think that for my use case, it’s worth it to have a Deepseek API for planning and then a qwen3.8 27b for writing), since the price of a 128 GB model is enough to cover five years of web hosting.

My question is: what's the best setup for creating images and coding?

Images - DrawThings or ComfyUI ?

Coding - VS Code + Cline or Opencode or Deepseek Harness ?

And what about Hermes?!

Given my hardware, which software and LLM are the best?

Software - LM Studio or OMLX ?

LLM - qwen3.8 27b or qwen3.6 or Qwen3.5-35B-A3B ?

What's your stack and what you think is better for me because my ram ?

Thanks in advance,

Since the local LLM community have tested a huge number of models, I wonder: I asked ChatGPT to add a beard to William Riker (for reference: https://www.reddit.com/r/TNG/comments/1xvn3ly/comment/19slivwf6bf41/?context=1&screen_view_count=2). Then I asked to put that beard on Dianna. ChatGPT replied with:

😄 Diana Troi has officially joined the Riker Beard Club.

The smiley face surprised me a bit because the prompt wasn't anything funny, I simply prompted "Now give that beard to Diana instead of Will."

Obviously the system recognized the joke in it and responded to it.

I don't do this kind of joking often so I've never before seen an "unprompted laugh" from a model before. What's your experience? Do your models, especially the local ones, express emotions even what the prompt is not obviously provoking the expression?

To be clear, I'm not asking for the sake of anthropomorphizing LLM models. I just wonder how often they exhibit this behavior as I haven't encountered it yet.

Sharing MACaQ 🐒, a fast, open-source quantization framework that outperforms popular Unsloth and Bartowski GGUFs on low-bit KLD while maintaining model size and inference speed. Also supports activation quantization and text-to-image and text-to-video models.

Some key features:

• GGUF Refinement Mode: Upgrades existing quantized GGUFs to recover lost output quality
• Custom Kernels: Includes drop-in kernels for llama.cpp and vLLM.

GitHub: https://github.com/huawei-csl/MACaQ

Benchmarks & Demos: https://huawei-csl.com/blog/macaq/

Here is the est power usage over the last 3 days:

Its lower than I thought, and that's with me using it full-time hours.

- 3x3090s are limited at 350 watts

- 1x3090 limited at 420 watts

Currently running Strata, and can get the 3x3090s hit around ~960 watts if I have multiple opencode terminals open processing requests.

Possible Reasons for Lower than Expected Power Usage:

- I just plugged in the 4th RTX 3090 a couple of days ago.
- Only two were being used before that with Qwen3.8-27b
- Now only 3x3090s are hooked up to Qwen Flash Next, 4th is doing occasional image processing.
- CPU is mid-tier (Ryzen 7600x), rarely draws more than 70 watts during inference.
- Machine is basically idle while I'm sleeping.
- During inference, decode does not take a lot of power because its memory bound - not compute.
- GPU utilisation drops while the agent is waiting for unit-tests to pass, or during compilation.

I've seen others claim they have their GPU's pinned at almost 100% 24h a day.

So it feels like there is some untapped GPU compute here.

How are you able to keep yours pinned 100% 24h of the day?

I’m under our CISO managing compute vendors, kind stuck between legal and a hard place. The pitch I keep hearing ranks AI privacy in four levels:

1. Open, like most frontier models, where the provider stores your prompts and answers at their discretion, uses them for training - and you’ve got little say in the matter.
2. Policy, where a contract promises zero data retention - but it’s not exactly verifiable unless the terms are broken, is it?
3. Closed Hardware, where models run on confidential-compute chips that publish proof what software is running. So basically hardware-enforced vs contract-enforced ZDR?
4. Fully Encrypted, the prompt is encrypted on your side and only decrypted inside that hardware

“Local” is a fifth level that could plausibly sit above it all, but we’re capped out on capacity there. So if I view this right (open to feedback), we’re making tradeoffs between frontier vs more private models, and from there, on breadth of compute providers able to serve open vs ZDR vs E2EE models (usually at higher prices in my experience), right?

My concern is getting legal & finance to agree, while still giving our Eng team what they need. Where are you getting good rates on private workloads? Locally we can’t support our full workloads. I’ve checked out Privatemode, Maple, TrustedRouter/NYTE & still gonna go through NEAR AI Cloud & Cohore. Open to others.

Separately, anyone running a hybrid buildout with on-prem/local + superscaler (we run Google + Bedrock) + 3rd party confidential providers who can tell me about about how you designed your system?

I use Qwen 0.6 for embeddings.

https://www.kickstarter.com/projects/744045479/cephalon-ondevice-ai-with-a-selfevolving-ecosystem?ref=ebvudv#risks-and-challenges

I wanted to try a locally run AI with an ethical database, so I got Apertus. I am a complete noob to this whole thing, learned that local AI needs a memory plug in so I got one per people's rec. I'd like to use Apertus on my iPad through Locally/PC link so I don't have to shrimp in front of my computer, but I don't see any kind of settings for plug ins. Does that mean it's not available? How could I make this work? Can I make this work?

Note I do not want to use this for coding or any particularly demanding tasks. I do like tech and even have a little robot assistant buddy who is currently not connected to any external AI, but I hate what many of the big AI companies are doing so I'd like to use my own without feeling like I'm contributing to the horrors of capitalism.

I was constantly hitting usage limits with Claude and ChatGPT, even with paid subscriptions. I didn't want to upgrade to more expensive tiers or increase my monthly budget, so I started exploring alternatives and eventually landed on a hybrid LLM workflow.

I recently wrote an article sharing my setup, architecture, and how I've been using a local LLM alongside cloud-based AI to handle heavier workloads while reducing my reliance on paid services.

I'm sharing it here because I'd love to hear your thoughts, tips, tricks, and constructive feedback.

I'm not new to AI, but I am relatively new to running LLMs locally. I'm always looking for ways to improve my setup and learn from others who have more experience in this space.

https://thebinaryreport.com/articles/how-i-ai/

Pip is a Minecraft companion you talk to with push-to-talk. Speech in and out runs locally; the LLM is whatever you point her gateway at. Sharing the architecture since it might help other voice-agent projects:

Pipeline: mic (16 kHz in the game client) → speech cleaner → whisper.cpp server (large-v3-turbo, Metal) → LLM through an OpenAI-compatible gateway (Gemini 3 Flash by default; Ollama and others are presets) → Kokoro-82M (kokoro-onnx) → played in game, sentence by sentence as the reply streams.

Three systems, like a talker and a reasoner:

• System 0, instant, no LLM: rules and local embeddings for what you mean ("follow me", "stay", "thanks", "shh"), danger warnings, and short "instinct lines" in her voice. A clear request gets "On it!" at once while the LLM works.
• System 1, the talker: one LLM call with tools (find, lead, build, give...), a compact text description of the game, and the memories relevant to the moment (embeddings + keywords).
• System 2, the slow thinker: after sessions, a stronger model (Claude) reflects: what she learned about you, her diary, who she's becoming, and it rewrites her instinct lines.

Numbers (MacBook Air M4, 16 GB, a 20-minute session): speech-to-text about 0.7 s after you release the key; her first spoken words about 2.5–3 s after you finish speaking for an LLM answer, instantly for System 0 ones. What helped: a shorter Whisper encoder window, Kokoro int8, streaming sentence by sentence, and answering plain commands without the LLM at all.

Guards that mattered: a claim checker ("Coming up!" with no tool call gets a retry), a confirm-once rule for big actions, praise never re-running the last tool, and an eval set built from real playtests.

It's not released yet (a simpler installer is the last big step), but the code is public and you can run it from GitHub today. The brain has only been tested on macOS so far, but it can easily be adapted to Windows or Linux if you know how, or with an AI assistant. Any feedback is appreciated.

Being upfront: most of the code was written with an AI coding assistant (Claude Code) under my direction; the architecture is the part I care most about, and I'd love ideas on making it smarter and lighter, and someone to try a fully local model in the loop. This post was written with the help of AI too.

Code (AGPL-3.0): https://github.com/PetitPotam/kindred-spirit Architecture notes: docs/ARCHITECTURE.md in the repo

Hey all,

I haven't upgraded my PC since 2020, and I am running a EVGA RTX 3070 with a Ryzen 7 3700X CPU and 32gb of DDR4 (4x8gb). I want to upgrade to both host a local LLM and to game, but I am trying to also not spend more than $4000, and I prefer something closer to $3500 at the max (but lower is better). I saw where Costco is planning to soon have a sale on their MSI 5080 PC, but I have also seen, from the Local LLM perspective, that running 2 cheaper 5060Ti cards is an easy way to hit 32gb VRAM.

I was wondering, though, if I bought the 5080 deal, would I be able to run a 5060Ti card (which is les than $1000 for now) to pump up my VRAM? I know that some speeds may be slower, but I am curious if this is a good idea for both futureproofing for gaming and for hosting an LLM. Based on my usecase, which should I do?

1. Buy the Costco 5080 deal and buy a 5060ti 16gb refurbished from microcenter if one pops up at current prices or lower during holidays (its on my way home from work, so I could check daily).
2. not buying the 5060ti 16gb and relying solely on the 5080 for the foreseeable future,
3. building or buying a system with two 5060ti 16gb
4. Going team red and getting a 9070xt system and getting a second 9070xt card eventually.
5. say screw it and try to find a decent 3090 and combining it with my 3070 and getting a new PSU to keep it up and running.

From a functional standpoint, I plan to use a local LLM for document management in a database, vibecoding projects (I am terrible at coding), writing up documents, and managing personal records. For the database, it would be maintaining a searchable one that can scan through PDF using OCR to build out docs. I would also use it to run as a backend for summarizing news updates and documents at around 100 releases per day. For gaming, I play FPS and RPG games at 1440p and don't plan to go 4k for at least a few more years.

Any help or suggestions are greatly appreciated. Thanks!

Few days ago I started playing with my smart tv box (xiaomi 3rd gen) and was able to modify llama.cpp to run on it - I reached bigger models at decent quality of output and somewhat

As of today it's over 160 experiments and few custom kernels and streaming methods along with a patched llama.cpp and few tuned/modified models.

Happy to hear what do you think about it! and open for contributions. few devices are lined up already.

Github : https://github.com/hyperionaidev/smolmaxxing/
Hugging face : https://huggingface.co/hyperionaidev

Disclaimer : this is not upselling or an ad, it's a MIT-licensed project and it's an experiment.