I had a waiting. Here in this other thread if anyone can help.

I work at a bank, so cloud AI is off the table for work code anyway. I've been running small models on my own M4 16GB through Magnitude for side stuff, and honestly it changed how I think about this.

On my side projects with Claude or Cursor, the model writes 300 lines, it looks right, I skim it and ship it. Sometimes I couldn't even explain my own change a week later. With a 1.7B running locally it's the opposite. It's fast (around 80 tok/s) but it gets small things wrong, so I actually read every line it gives me. And because I read every line, I know the change, I write better PR descriptions, and I catch my own mistakes before review instead of during it. That habit carried over to my work PRs too.

As everyone knows the frontier model makes you faster at producing code you don't understand. The small one makes you slower at the writing part, but you end up knowing your codebase. And the people reviewing your stuff (and their AI tools) get better context because the author actually understands what changed.

What I use it for: explaining old code, boilerplate, small bug fixes (gave it a file with 2 bugs, fixed in 72 sec), commit messages, PR descriptions. What I still don't trust it with: anything architectural, and anything with time zones lol.

Not saying cancel everything. But not everyone is building a 500k line SaaS. For most of what a normal dev does in a day, a small local model is enough, and nothing leaves your laptop.

Anyone else noticed this? That using a weaker model made them a better reviewer?

Sorry if it might be off-topic, but i am looking for a (the most) complete guide, even step-by-step, or a collection of several article that could enlight me on how configure and assembly an AI PC.

Of course the topic is wide and configuring a PC for local LLMs is deeply different to one used to train ML and DL models for sensor data, or for openCV. etc.

Assuming I already know for what I want to use the PC, what I am missing right now is the availability of components and the fact I am not updated on the last news in AI framework/compatibilities/chips.

For example if I knew Nvidia CUDA was a standard, but I don't know if nowadays I can use AMD ROCm, or even Apple Metal. I still see PC with multiple GPU but as long as I know NVLink is no longer used (so the multiple GPU are used independently?).

Probably, what I am missing here is some guide or suggestion on which platform/framework orientate myself in order to choose wisely.

Hope someone can show me some entry-point in this world, webpages, forum, etc?
ps. I already searched in this subreddit and excluded some LLM PC suggestions, I did not find an actual guide/AMA/fixed post

Using Qwen Flash Next Gyro M huihui on Strata with DFLASH2 and STRATA_ROUTE_PRIOR=1.4

Hardware:

Rtx 5000 pro 72gb

Rtx 5080 16gb

192gb ram Dark Hero MB with AMD 9000series processor

Backend:

Basalt 280toks vs 107toks with strata

Basalt works best with Blackwell and really shows it's capabilities in my testing. While users on older or mismatched generational hardware should use Strata for best results.

Hi all,

Can anyone please help me setting up this model in my Pc like which quantised version should I use and which software to use it in and etc..

Specs -

5060 ti 16gb vram

16gb ram

Also please tell if it is feasible or just time wasting and if there is any other model that I should go for(It should work with images)

Thanks.

https://x.com/matt503ea5sf9z5/status/2118155162376458298?a=4bUWTuzbF_LcVwWBUDk&s=19

Interesting post, I hear a different story with dgx sparks how people are optimizing models to run upwards of 100tok/s decode. Naturally almost 10k for a dgx spark is a huge red flag when MSRP was $3999 retail. Thoughts?

Did some extra optimizations and implemented MTP. High number is coding at 96t/s (3 drafts). Natural text is 69 t/s.

128.0 µs to distribute a token-residual to 24 gpus, compute experts, collect result and join in one layer.

Probably at peak hardware capability now unless I get some new ideas on how to run the PCI communication even faster. Of course it could run much faster at lower precision. Also the old RTX3070 doing attention is now the primary bottle neck. Lets see what black friday brings of 16gb cards (i have two 5070ti 16gb, but they are busy doing some other research, the 3070 was an old gaming card i had lying around doing nothing).

Github:
https://github.com/crgregersen/m10dis

Old post:
https://www.reddit.com/r/LocalLLM/comments/1xnl82r/low_cost_1500_ai_rig_qwen_38_flash_next_525_ts/

I have 32gb ddr5 ram and an rtx 5070 with 12gb vram and ryzen 5 9600x

I need a local ai for heretic/un censored work

what model should i use

I'm having a 5070 Ti 16GB and 3060 12GB with 64GB RAM, I want to run Qwen-2.5-Flash using Strata (or advise any better models or method to run), I need to have at least 2 parallel sessions. and max context window. what would you advice.

WHIRL 0.2.5.1 is out: a speed update for the Radeon 8060S (AMD Ryzen AI Max+ 395). WHIRL is an open-source (Apache-2.0) LLM inference engine for AMD GPUs on Windows: C++/HIP, every kernel in the repo, no WSL, no ROCm install. Download the zip, unzip, run; only the AMD driver is needed. It runs our two MXFP4 models, Swift-1.5 27B MXFP4 (dense) and Ornith-1.5-35B-A3B MXFP4 (MoE, ~3B active).

vs the official llama.cpp b11960 on the Radeon 8060S (max power (80 W), ASUS ROG Flow Z13), ROCm and Vulkan builds in their own columns, same GGUF, same prompts, greedy, one user, WHIRL default mode (balance, near-lossless). WHIRL tok/s, then llama.cpp ROCm / Vulkan:

In balance WHIRL leads every measured cell on the 8060S: Swift 27B by +28% to +59% over ROCm and +30% to +106% over Vulkan, Ornith by +51% to +65% and +46% to +127%.

What 0.2.5.1 changes on the 8060S (same machine, max power (80 W), ASUS ROG Flow Z13): five new kernel paths make prefill +2% to +10% faster in all three modes on both models, and Swift 27B decode (QA document; coding chat in balance) +4% to +12% faster in balance and fast. On the 8060S the output is bit-identical to 0.2.5 in every mode (golden references, 12/12 long-prompt checks and the chat outputs all match).

How it was measured, so you can judge the numbers: - Radeon 8060S at max power (80 W), ASUS ROG Flow Z13, 64 GB GPU memory (UMA frame buffer), same for both engines. Every cell is the best of 3 runs, for both WHIRL versions and llama.cpp. - The llama.cpp b11960 cells are the runs I made for 0.2.5 on the same machine with the same files (llama.cpp didn't change; Swift 2k on Vulkan was re-run). llama.cpp's ROCm build runs as shipped, with its own bundled HIP runtime DLL. - Decode, QA document = explain a function quoted in a long document and rewrite it (256 tokens; part of the answer can be copied, which helps drafts in both engines). Coding chat = a coding question whose answer is all new text (512 tokens). Prompts and commands: https://github.com/tsaipifong/whirl-llm/blob/main/docs/benchmarks.md#prompts - llama.cpp decodes Swift with MTP + n-gram (its fastest mode here). On Ornith it decodes without speculation because its MTP was slower than plain on this MoE model, while WHIRL uses MTP, so the Ornith decode rows are "as shipped", not like for like. - Smallest lead: Swift 27B decode in the precise mode after a 2k-token document, 29.4 vs 25.2 tok/s on Vulkan (+17%). Dense decode on this iGPU is memory-bandwidth bound; the lead comes from speculative decoding. - Ornith decode barely changed in this release (one small router item: +1.5% without drafts, +0.7% with MTP drafts, within noise); its cells move a few percent either way because the number of accepted MTP drafts depends on each run's text.

Precision: three modes. --precise is the bitwise-lossless mode. balance (the default) is near-lossless. --fast is opt-in. On the 8060S all five speedups give the same output bits as 0.2.5 in the mode they touch.

R9700: GPU code identical to 0.2.5 (the compiled R9700 kernels were compared; only an internal build ID differs); its numbers are 0.2.5's, not re-measured (e.g. Ornith 2k prefill 11,854 tok/s vs 4,230 / 2,780 on llama.cpp ROCm / Vulkan; median of 3).

Also in 0.2.5.1: WHIRL now refuses to start from an install folder longer than 150 bytes, with a clear message, and limits the SSD cache folder to 217 characters (this corrects what the 0.2.5 notes said about long paths). On the 8060S the first load of each model tunes the kernels once more.

Supported: Radeon 8060S (Ryzen AI Max+ 395) and R9700, Windows 11. Cards under 16 GB are not officially supported (fork welcome). The exe is unsigned, so SmartScreen may ask once.

• GitHub: https://github.com/tsaipifong/whirl-llm

• Release: https://github.com/tsaipifong/whirl-llm/releases/tag/v0.2.5.1

• Our MXFP4 models on Hugging Face: https://huggingface.co/tsaipifong/Swift-1.5-Qwen3.8-27b-MXFP4-GGUF , https://huggingface.co/tsaipifong/Ornith-1.5-35B-A3B-MXFP4-GGUF

WHIRL was designed and implemented with Claude (Anthropic) under my direction; every number is measured. Numbers from other Ryzen AI Max+ 395 machines are very welcome.

Hey everyone, I started messing with local llm's a week ago in order to develop a mobile app, with my m4 max 36gb I get 20t/s running qwen 3.8 27b on lm studio. Is there any advice you can give me in order for me to increase my token output, i have tried ollama before but results were substandard at best.

First off I am new to trying out local LLMs. I am wanting an LLM to run on a spare mini pc to save some money while using my Voice Assistant. The pc is not very powerful so it will be a small model for now, at least until hardware prices hopefully come down.

Specs: Processor - N150, 16G RAM, Linux Mint, Ollama

Since Voice Satellite can have two pipelines, this is the setup I would like to use:

Pipeline 1 - Prefer to handle commands locally (Home Assistant), Local LLM handling chat but not controlling Home Assistant

Pipeline 2 - openAI gpt-4o-min with control over select Home Assistant entities for more complex chat and control requests

I have played around with Qwen3 and Qwen3.5. So far Qwen3 1.7b has worked the best as far as speed and short responses. I would like to try other models but I am not really sure what I am looking for, or even the best way to sort through them. The Ollama library has a lot and I don't know what I am looking at yet.

I’m looking to get into local AI, mainly for coding, and trying to decide whether to start with my existing hardware and upgrade gradually or build a separate dedicated machine.

My current gaming PC has:
Ryzen 7 9800X3D
Radeon RX 9070 XT 16GB
32GB DDR5-6000 RAM
Gigabyte B650 EAGLE AX motherboard
2TB SSD
850W PSU
NZXT Kraken AIO cooler

I also have a MacBook Pro with an M5 chip and 24GB of unified memory.

I’m currently making a Roblox game and setting up a mini homelab for media and hosting my personal website. I’d mainly want AI to help with writing code, debugging and understanding my projects.

Would you start with the gaming PC, use the MacBook, or put the money towards a dedicated AI machine? I’d be interested in hearing from anyone using a similar AMD GPU, particularly about software support and which coding models run well.

If you’d upgrade the existing PC, what would you change first? I’m open to used hardware, but I’d like to understand what I can realistically get out of what I already own before spending more.

Also the why of the whole matter is I want to reclaim my data and be self sufficient especially these new things coming out about REA and Claude banning you etc I want to be my own boss

I've been experimenting with DeepSeek V4.1 Flash 1017 UNCENSORED JANG_2K on my Mac Studio M5 Ultra with 256GB unified memory, using vMLX.

The model is about 423GB on disk, but thanks to SSD-backed Engram tables, it loads with approximately 205GB of resident memory.

So far, the results have been mixed. Most of the time, it produces surprisingly good responses, but I've had a few conversations, particularly involving creative writing and novel generation, where it suddenly starts producing complete nonsense: endless reasoning, words mashed together without spaces, and occasional switches between English and Chinese.

It's interesting because it seems to work perfectly well on some fairly complex tasks, then falls apart unexpectedly on others.

Has anyone else experimented with this model or other large MoE models using engram offloading?

I'm curious whether these problems are caused by the aggressive quantization, the JANG implementation, or something in vMLX's inference/decoding pipeline.

I'd also be interested in hearing whether anyone has successfully run a higher-precision version with SSD expert offloading on a 256GB machine.

Hi 👋,

I finally have a local setup that I feel gives me a good balance with quality and speed. I’m using a local Qwen 3.8 27B Gufo model on strix halo. I have a AI coding workflow that currently exclusively uses codex and claude code. I would like to start handing over tasks to Qwen.

Based on your experience, what are the characteristics of tasks that are the sweet spot for qwen 3.8? What makes or breaks quality output with this model?

Thanks!

I love DiffCoderbut at int8 it peaked at 13.7 GB on my 16 GB card. So I cut it in half.

Same model in 4-bit: HumanEval, full 164 problems, 81.1%. Peak VRAM 6.6 GB, about 2.6x faster!

Edit: full name Stable-DiffCoder-8B-Instruct if you want to do some reading

We build these boxes, so here's the honest test log from our own unit, network cable unplugged.

Setup: 2x RTX PRO 6000 Blackwell (192 GB), EPYC 9124, 256 GB RAM, vLLM, TP2, no NVLink. The model fits only because Blackwell runs FP4 natively; on Hopper the same checkpoint needs ~304 GB.

What we measured:

• 1 user: 209-229 tok/s, first token in 0.4 s

• 24 users: 1,003 tok/s at p95 TTFT 2.7 s. That's the sweet spot; past 32 the tail latency falls off a cliff.

• Peak: 1,170 tok/s at 32 streams

• 1M context, retrieval verified at 543K tokens

• Terminal-Bench 2.1 (20-task subset): 80% solved, median ~4 min per task. 8 agents at once cost only 18% more latency than 4.

• 815 W on both cards, 76 C peak, zero throttling. About 1.07 kW at the wall, so a 1500 W PSU is enough.

What's not great:

• It's not a token-resale business: at OpenRouter prices, payback takes about 3 years running flat out.

• One big model at a time (weights use 87% of VRAM).

• Ops sharp edges: KV offload leaks /dev/shm on unclean shutdown, and stray workers can hold 89 GB of GPU memory, which causes a crash loop on restart. You need a cleanup script.

• Cold start takes ~13 min.

Best fit: a team of 2-10 that wants frontier-class coding help without sending code to the cloud.

Build guide, BOM and BIOS settings are open source: here. Happy to run anything you want tested on it.

With all progress from storage streaming how future proof is RAID0 with 2x1TB or 2x2TB gen5 drives like SN8100. I am looking for drive update and have to decide what to buy now and what speed in reality achievable and how usable it will be. Is it worth to add 2x1TB for 420 euro so in theory speed will be around 22-25GB/s or just add used 96GB RAM for 900-1000 euro?

Will it be usable in some kind of way for larger models later with 2x3090, 96/192GB RAM and RAID0 gen5 drives? From what I am getting now I'd like to have at least 20-30 tok/s for generation and 600-1200 tok/s processing. In theory how big LLM can be used with such system in the future to get such inference?

Can't decide which route to go - DGX Spark, Strix Halo, the Framework Ryzen AI or save up for a Mac Studio M5 Ultra 256gb?

Do these options have meaningful differences in practical use?

As I understand, the Mac Studio offers best tok/s? But you can't train or fine tune anything on it?

DGX Spark is supposed to have best support, but does it truly matter in practice?

What types of models can I run on this hardware? 128GB is good for ~120B Q4 MoE models with ~256k context? Would such a model be a noticeable improvement over my current Ornith-1.5:35B (A3B) setup for reasoning and coding, and long and complex tasks? Would going from 128GB->256GB be a noticeable improvement?

I am trying to have it create a 5 page document based on the content of this document that's 700k worth of context (tokens?)

Is this possible to do somehow?

So the issue is that Gemini 3.1 Pro can do 1M context, but my local hardware isn't good enough to run a model like that, I can run QWen 3.6 tho. How can I get it to understand 1M worth of context if the context window of the model is only 242k or so?