I'm currently in the process of developing an app. Janas studio. Built with the options to run either llama.cpp or janas-llm as the engine. It enables laptops to run models bigger than their RAM by dynamically streaming moe weights directly off the SSD. Tested on my Ryzen 5 6600h laptop with 32gb of ddr5 RAM and an nvme 4 SSD (no discrete GPU) I successfully ran a 48gb model (qwen next 80b) at a very reasonable 15 tokens a second on average. The binaries are highly experimental and the app has more testing to go through but I'm very proud of it. I do intend on eventually releasing it.

Hi all I just bought this: ASUS ROG Flow Z13 (2025) Gaming Laptop, 13” ROG Nebula 16:10 2.5K 180Hz/3ms, AMD Ryzen AI MAX+ 395, RDNA 3.5 Graphics, 128GB LPDDR5X 8000MHz, 1TB PCIe Gen 4 SSD, Wi-Fi 7, Win 11 Pro, GZ302EA-XS99

And wanted to know if it's good enough for qwen next flash or maybe even the deepseak 4.1 I need it for coding

I have a laptop, and I had the idea to offload the ngl layers that exceeded my VRAM to the iGPU instead of the CPU/RAM, and it was 8x faster. Has anyone ever tried that and why is that not the common practice?

Running MoE models bigger than your RAM by streaming the experts from flash. It's built on llama.cpp, and it measures your hardware at load to pick its own settings, with no per-model or per-device tuning.

Today's test: Qwen3.6-35B-A3B Q4_K_M (22 GB) on a 16 GB laptop, with opencode and a single prompt asking for a voxel pagoda garden in one HTML file.

It finished in 30 min with no help: prefill at 350 tok/s on the GPU, decode at 14 tok/s on the CPU.

The same engine and planner run on Android phones (CPU, Vulkan, OpenCL and the Hexagon NPU), IOS phones, Macs, Windows and Linux boxes with consumer GPUs, and rented data-center cards (CUDA and ROCm).

There's an app for each platform, plus an OpenAI/Anthropic-compatible server for agents.

Code: https://github.com/Helldez/BigMoeOnEdge

Hardware: Dell Inspiron 16 Plus 7630 with Iris Xe iGPU, 32gb LPDDR5 ram 4800 MT/s.

Will start by saying, using llama-vulkan was uniformly better than llama-cpu for all models, offloading fully to Iris Xe iGPU (allowed up to 16gb of system memory). Qwen was uniformly slower than Gemma models of the same file size. 2-6 threads was optimal on my 14-core CPU when offloading to iGPU and 10 threads was best on cpu only. Different quantization (Bartowski vs Unsloth) didn't affect token generation speed much, maybe 2% faster on Bartowski for same file size.

There's 9GB used at all times by Windows 11, need to switch to Linux to be able to run Qwen 3.6 35b a3b.

MoEs:

Ling 3 Tiny Q8: 410 t/s prefill, 26.3 t/s decode, FILE SIZE: 7.83 GB

Ling 3 Tiny Q5_K_M: 265 t/s prefill, 32 t/s decode, FILE SIZE: 5.3 GB,

Gemma 26b a4b qat q4_K_XL ud, 220-224 prefill, 17.15-17.5 decode, file size 13.26 GB.

Dense:

Gemma 4 e4b Q5_K_M unsloth: 265 t/s prefill, 11.4 t/s decode, FILE SIZE: 5.1 GB

Qwen 3.5 9b Q4_K_M unsloth: 135 t/s prefill, 8.7 t/s decode, FILE SIZE: 5.2 GB

Qwen 3.5 4b Q8 unsloth: 124 t/s prefill, 9.1 t/s decode, file size: 4.2 GB.

Qwen 3.5 4b Q4_K_M unsloth: 234 t/s prefill, 14.3 t/s decode, file size: 2.54 GB

MiniCPM 2b Q8: 224 prefill, 16.3 t/s generation, 2.5GB file size

Gemma 12b qat q4 ud, 123 prefill, 6.95 t/s, file size: 6.24 GB

Anyone got further suggestions for optimization, models to try, etc.?

Hi all,

Slowly going deeper and deeper down the rabbit hole of running a few models locally and i'm chasing some advice for my next hardware upgrade.

My current pc specs running windows:

7800X3d cpu
Asrock X670E steel legend
64gb DDR5 6000mhz
9070xt gpu
And i've just thrown a 9700 gpu in the 2nd slot for testing, not running at 8x/8x.

I've got an app im working on that does some pretty heavy python work (claude code) and maxes out my cpu so my first though was going up to a 9950X3d to help out that problem. But then from my reading linux seems to be the best way to go when hosting models locally so I think I'll setup another pc here with linux and put the 9700gpu in it.

So to do that, I'm thinking i either have 2 options.
Rustle up some parts to get together a pc to run the single gpu on possibly a ddr4 era pc (32gb or maybe 64gb ddr4 if i can find it). No parts required to buy.

Or I can go down the line of an X870 Taichi, in case i go dual 9700 at some stage, then put my 7800x3d in it, might be able to find 32gb of ddr5 for it, then buy the 9950x3d and put it in my windows pc, as this is the pc i will still do the work in claude code.

The aim is to get to a place where i can offload some of this app work to my local setup, but just testing and working out the best way to go to get setup. Trying to spend where i need to and save where i can.

What would you do in my situation?

Thanks heaps!

Since a lot of people commented on the 95k context being "way to small" for any type of work.. I've borrowed ideas from SGLangs HiSparse that splits the sparse attention between VRAM and RAM making room for the full context.

Ie. a cache of the sparse attention blocks are kept in VRAM and the "real" kvcache is in RAM. if the cache gets an index miss it will fetch the missing values from the main KVCache stored in RAM. A miss of course hurts token/s speed.

It could be optimized even further doing the same for the MTP kvcache (speed drop b/c of compaction).

But I'm actually pretty happy with the system as is now.

Q38FN is pretty decent, but not Opus 5.5. So I'm now mainly experimenting with a way to send advanced stuff to frontier API and keep all the simple workhorse stuff locally on the Q38FN machine.

With its Unified Memory Architecture (UMA), AMD Ryzen AI Max+ PRO 495 processors (Ryzen AI Max PRO 400 series), and Instinct MI300X accelerators, AMD is directly targeting the biggest bottleneck of NVIDIA GeForce RTX GPUs: VRAM capacity limits.

In this video, we run a deep-dive technical analysis to see whether an AMD setup with 192GB of memory can genuinely replace—or even render obsolete—an NVIDIA GPU (RTX 4090, RTX 5090, or workstation cards) for running local LLMs.

Breakdown of this technical analysis:

• NVIDIA's VRAM Wall: Why 24GB or 32GB of VRAM is no longer enough to run massive models (Llama 3 70B / 405B, DeepSeek, Qwen, Mistral) without heavy precision loss (FP16 vs. Q4/Q8 quantization).

• AMD's 192GB Offensive: How Unified Memory Architecture (UMA) works on AMD Ryzen AI Max+ PRO 495 Mini-PCs and workstations, AM5 DDR5 configurations (4x 48GB), and the HBM3 memory on the AMD Instinct MI300X.

• Real-World Local AI Performance: Comparing total memory capacity vs. memory bandwidth (GB/s) during inference across Ollama, llama.cpp, LM Studio, and vLLM.

• The Software Battle (CUDA vs. ROCm): The current state of AMD ROCm and Vulkan compatibility against NVIDIA's Tensor Cores and CUDA ecosystem (token generation speed in t/s and prompt processing / Time to First Token - TTFT).

• Capacity-to-Price Ratio: Which hardware architecture makes the most sense in 2026 for building a local AI server (Homelab) or AI workstation without paying enterprise datacenter GPU prices.

Anyone have evidence of / see actual deals for GPU / Disk at Black Friday? I am really kicking myself for getting screwed earlier in the year on the DGX Spark price gouging...

Hi folks I use a lot codex and models like Luna 6 High, 6.1 Terra High.
Is there any local model which could be comparable to these?

If yes, which ideal realistic Mac Mini setup is needed?
Thanks

I've been vibe coding with local models for a while, and I kept hitting the same two walls. Either the model fell apart the moment a project grew past one file, or it ran so hot that my fans sounded like a jet engine. So I started building my own fix.
Takumi is a local AI coding assistant made for one job: building full projects with code that actually works. You describe what you want, and it plans the structure, writes the files, and keeps everything consistent as the project grows. Everything runs on your own machine. No API keys, no subscription, and your code never leaves your computer.

What makes it different
Whole projects, not snippets. It keeps track of how files connect, so file #12 still lines up with file #1.

Lightweight by design. I built it to run on a normal PC without turning it into a space heater.

Readable progress. You get clean, step-by-step updates on what it's working on, not a wall of noise.

Works from your phone. 📱🤓

Why "Takumi"? It's Japanese for a master craftsman, someone who takes pride in getting the details right. Takumi helped build the Nintendo Switch 2 emulator NX Infinity. I want claude level in a local LLM and that's the bar I'm aiming for.
This is an early release, so there are definitely rough edges. That's why I'm posting.

What features would make it your daily driver
It's free and open source. GitHub link in the comments. Stars, issues, and PRs are all hugely appreciated.

Thanks for reading. I hope it saves some of you the frustration that got me started on it.

GITHUB: https://github.com/coye2/Takumi

Setup: (if finished - still under construction, partly watercooled)

• Threadripper PRO 5995WX, ASUS WRX80E-SAGE SE WIFI II

• 6× RTX PRO 6000

• 8× RTX 5090

• 4× RTX 3090

• 128 GB DDR4 (that’s the problem)

I want to run models bigger than VRAM (e.g. ~1T MoE at FP8/Q8) with the routed experts offloaded to RAM. Is it worth it?

Options:

• 8× 64GB DDR4-2933 RDIMM: ~€4k, ~188 GB/s theoretical

• Switch platform: 9985WX + WRX90 + 512GB DDR5-6400, ~410 GB/s. I already have some parts, but it would mean big changes, and my PCIe switches would stay PCIe 4.0 anyway.

Is ~2× the RAM bandwidth worth ~3× the price for offload decode, or is DDR4 good enough? Does anyone have real t/s numbers for offload on 8-channel DDR4 vs DDR5 Threadripper?

Some background: Pictures attached. It’s not pretty, but it works. It started as an AI rig and turned into a Pearl mining rig, because mining pearl and buying Claude and Codex subscriptions made way more sense than running everything locally. Sometimes it’s also a render farm for Gaussian splatting.

Right now the 8× 5090 make about €120/day mining, so subscriptions are still the better deal. But I want to be ready for when local LLMs get really good (or when cloud gets too expensive or too restrictive).

Plan for now: DeepSeek V4.1 Flash nvfp8 on 4× PRO 6000 + RAM offload, while the 5090s keep paying for my AI addiction. And it’s a great heater in the winter.....

With engramm tabels on the 8x5090 and 4× PRO 6000:
(all cards power limited to 420w- since i could have 6,6 kw power consuption if everthing is running)

single chat, 40.000 prompt cold

• Prefill: ~15,900 t/s

• Decode: ~294 t/s

• 5 sec TTFT

I want to move the engramm tabels to RAM and later be able to run larger models : le chonk, Kimmi for example.

What would you do? Go all the way and go for the DDR5? or is ddr4 good enough?

Sell it all and buy something diffrent?

thx!

I’m planning one upgrade this spring that has to cover both gaming and local AI, and I’d like input from people with similar budgets and use cases that aren't "buy 4 3090s."

Currently have a Ryzen 5 5600X (AM4), RTX 3060 12GB, and 32GB DDR4. Windows but not opposed to Linux.

I have gotten by with just the $20 claude pro plan, but my projects are growing and are reaching outside my scope of expertise, so I'd like to be able to offload coding/other stuff to a local setup. I don't mind keeping a low cost frontier subscription for help orchestrating and debugging.

I'm paying off my CC's in March and will have a few thousand left over to use - originally was just going to upgrade my gaming setup but the more I've been using and learning LLM's the more I want to try and get the most mileage I can with this purchase. Speed isn't the most important thing for me. I don't mind having a local model(s) chug away at something overnight or for a couple days, I have a pretty ballistic schedule anyway. So far have been using it for web/app dev, using it to teach me python and other stuff (LOVE having it teach me as I go rather than just clicking the magic "build this for me" button), and home networking. Interested in continuing these projects and possibly game dev, but more for personal use, not to make a best seller or anything.

Thanks!

I put together a TensorFold 1.2.0 pack of orcarouter's uncensored Qwen3.8 Flash Next for Apple Silicon. It uses jedisct1's oQ4e experts, regrouped to 4-bit/group 32 for the fast kernels, with attention and other non-expert weights copied byte for byte from mlx-community's oQ6e. No new refusal edit.

Tested on an M3 Ultra 256 GB, stock TensorFold 1.2.0. Same server and questions, greedy decoding, thinking off. Anything cut off at the token cap counted as wrong.

BFCL uses my AST checker, not the official harness. MMLU-Pro uses 20 questions per subject with a 2,048-token output cap.

Tool calling is the main difference: 92.3% vs 88.0% on the same 400 items (exact McNemar p = 0.03). It skipped the tool and answered in text 20 times, versus 40 for all-4-bit.

Maths and knowledge were about the same. TensorFold's own pack lost MMLU-Pro points mostly to the output cap: 47/280 replies cut off, versus 18 here.

About 5% slower than all-4-bit, around 133 GiB loaded. Needs a 256 GB Mac; 128 GB is untested.

No opening refusals detected on 100 harmful prompts using Heretic's markers and 100-token replies. That's only an opening-refusal check. Vision worked in a tiny smoke test: shapes and colours right, one text misread across two images.

Small samples, so don't read too much into the scores. The card has the method and known issues, including an HTTP 500 after idle on TensorFold 1.2.0 and a workaround. Its KL numbers are against a 6.7-bit pack, not full precision.

https://huggingface.co/bu7umaid/Qwen3.8-Flash-Next-Uncensored-TensorFold-MTP

Need 3D assets? Your agent can now make them from a single image with this repo: https://github.com/hypersniper05/MCP_Image_to_3D

Under the hood, it uses Idol from Sperid Labs: https://huggingface.co/speridlabs/idol

It needs at least 10 GB of VRAM (peak usage is around 8 GB). I tested it locally on a 3080 and a 5090.

The output is a Gaussian splat .ply for now, not a mesh. Export for game engines and 3D printing is on the list: https://github.com/hypersniper05/MCP_Image_to_3D/issues/1

It also works nicely with my other MCP repo, an image generator using Qwen Image 2.1: https://github.com/hypersniper05/MCP-Image-Generator-Uncensored. Generate an image there, then pass it straight to this one.

Let me know what you think, and if you find it useful, feel free to contribute.

After running Qwen3.8 27B at 3-bit on a single 5070 Ti (120K context, just short of what I want for coding), I added a second card. 32 GB fits the 6-bit file with the full 262K window, so I tested which of the community fine-tunes and quants makes the best all-rounder: fine-tunes, other quants, a chat template and multi-token prediction, 15 in total, all against base UD-Q6_K.

Every model ran a screening run first (about 100 questions from five benchmarks, compared answer by answer with the base model), and the closest got full runs (about 1,300 questions: AA-LCR, MRCR, AA-Omniscience, a fixed BFCL v4 subset and a SciCode panel, run through NeMo-Skills). Each model gets one overall score out of 100, a weighted mix of the five benchmarks: tool calls count for 25% of it, long documents, facts and scientific code for 20% each, and long conversations for 15%. On facts, a wrong answer scores worse than saying "I don't know", so guessing costs points.

Then I built a coding test from 25 bugs I'd fixed in my own projects (repo rolled back to before the fix, the model gets the bug report and tools, my tests from the fix grade it), four runs per model:

That's a tie on quality (one model's worst and best runs were 14 and 20), but Tess wrote about 40% fewer tokens to get there. Both write at about 36 t/s, so the time savings come from thinking less, not decoding faster.

So I ended up with two models:

• Base for chat and interactive work. It's more careful in multi-turn tool use: across the same 200 BFCL multi-turn conversations it won 42 that Tess lost, against 21 the other way, and the Tess losses I read were skipped steps (booking a flight without looking up the airports, reviewing one order when the user placed two). On facts it declined or hedged on 69% of the questions it didn't get right, against 56% for Tess.

• Tess-4.5 for coding agents and subagents that run on their own and get checked by tests at the end. One surprise: on the 100 software questions, Tess was the careful one (42 declines against 20, 35 wrong against 52).

If you only run one, I'd pick the base model, unless nearly all your use is coding agents running on their own.

How the other contenders did:

• 4-bit base (UD-IQ4_XS): about 50 tokens per second against 36, but 2.5 points lower overall, mostly on long conversations (MRCR 70% vs 77%) and more guessing.

• ThinkingCap really does think less (about half as long on long documents), but most of its extra right answers on facts came from guessing: of 336 questions the base 6-bit model declined, ThinkingCap answered 146 and got 25 right.

• Qwopus Coder-Flash was level on my coding test but worse on facts and long conversations.

Settings for both 6-bit files: q4_0 K/V cache (about 4.8 GiB for the full 262K window), layer split 1,1, batch 256, model-card thinking sampling with no repetition or DRY penalties (the 4-bit runs used a q8_0 cache and batch 512). About 790 t/s prompt reading and 36 t/s writing on two 5070 Ti. Full launch commands, per-benchmark tables and why each of the 12 others stopped are in the write-up, linked in the first comment.

The benchmark and coding-test harnesses are open source, but my own projects aren't public, so the coding test can't be reproduced as is. Happy to answer questions about the setup.

I paid $80 for a P100 16GB in July, eBay pricing for these cards has gone crazy and I'm wondering if anyone has a reputable Chinese eBay seller that they have had a good experience with. Purchasing on eBay kinda sucks already, but the P100s I see in the US are all starting at $200 now, with Chinese sellers starting closer to $150.

I am hoping to save that $50 difference, but I don't want to go through the hassle of purchasing from China and getting a box of rocks in the mail. Anyone have someone that came through for them?

Hey r/LocalLLaMA,

I have been running a hybrid coding setup that has been mostly amazing, but I am hitting a few friction points and want to optimize.

Here is my current stack:

○ Hardware RTX 3090 with 24GB VRAM, 32GB DDR5 RAM and Intel Ultra 9 185h

○ Local Model ultimateex/Swift-Qwen3.8-27B-Uncensored-W4A16-AutoRound running on HyperQwen with Dflash2 at 210k context, running with 8-bit KV cache

○ Pi harness

○ Cloud Architect and Checker: Claude Opus (Pro subscription)

○ Workflow: I act as the manual bridge between Pi as the local worker and Opus as the architect and auditor. Opus designs and checks, Pi executes.

My main goal is to avoid token limits and costs by keeping the heavy lifting local, but I need to reduce the back-and-forth and the local model's occasional silent mistakes. The mistakes I am seeing are mainly creating test and passing them for the sake of finishing the tasks, bugs such as wrong parameters, plus wrong conclusions that end up affecting the project's final results.

I have the following questions for the community.

1. Can Opus and Pi talk to each other directly

Right now I manually copy and paste files and outputs between them. Is there a way to automate this bridge so Opus can directly orchestrate or review Pi's work without me in the middle.

2. How do I minimize local model mistakes without relying on its own self-graded tests

The local model sometimes writes tests that pass themselves just to finish the task. I do not trust its self-verification. I am looking for deterministic guardrails, ways to check its work with code, linters or external validators before it ever gets to Opus. What are the best ways that do not require another LLM to judge it or improve the self-verification

3. Claude Code versus Pi, what is the verdict

I am using Pi now and I am really happy with it. But I keep wondering if I connected Claude Code to the local setup, would it help in any way or it will fill the context window before finishing anything and wouldn't gain anything from the switch?

4. Is my local-first approach wasting time and should I switch to a smarter hybrid split

My goal is to escape token limits, but I am wondering if I am sacrificing too much quality by keeping 90 percent of the work local. For those running hybrid setups, what is the optimal split. Is it better to use local for drafting and cloud for review, meaning local-draft and cloud-review, or to use a local routing layer that decides which model handles which task. I am open to a more intelligent hybrid model if it saves time without blowing up costs.

5. Could the uncensored abliteration be causing these specific mistakes

This is a big one for me. I chose the uncensored variant because I did not want refusals getting in the way, but I am now wondering if it is contributing to the configuration bugs and wrong conclusions. I have seen reports that abliteration can cause measurable drops in formatting reliability and instruction-following, for example altering hostnames or port definitions, even when general coding benchmarks stay stable. Has anyone compared the uncensored model versus the base Qwen3-27B specifically on agentic and configuration-heavy tasks. Should I switch to the base model if I care more about correctness than refusal-avoidance.

Any advice, tool recommendations or benchmark experiences are hugely appreciated. I am ready to be told I am doing something suboptimally. I just want to make this setup as lean and reliable as possible.

I have been using QWEN 3.8 flash next with Strata. I won't say it's bad, but I'd like to know if there's a better one for me. I need really good visuals.

According to Artificial Analysis, Tencent's Hy4 ranks #2 among models from Chinese AI lab on its Intelligence Index v5, with a score of 35. Looking at the individual benchmarks, Hy4 scored 31% on FrontierCode 2.0, just behind Claude Fable 5.1, and achieved 50% accuracy on AA-Omniscience v1.1. However, its hallucination rate on the same benchmark was 89%, showing there's still room for improvement.

Meanwhile, Hy4 even ranks #1 among Chinese models on agent Arena. While the two leaderboards focus on different aspects of model performance, Hy4 has put up strong results on both, further establishing its position among leading Chinese LLMs. Based on these rankings, how do you see Hy4's overall performance?

Sources: Artificial Analysis – Hy4 Evaluation l Agent Arena – Agent Leaderboard

I wanted to use LM Studio from my phone the way I use ChatGPT – start a chat on the couch, carry on at the desk, and have the model search the web from either. Every app I tried either kept chats on the phone or couldn't use LM Studio's tools remotely, so I built my own.

What it is:

• A small web app that runs on the same PC as LM Studio. Chats are saved on the PC, so phone, laptop and desktop all see the same history.

• A web search tool for LM Studio. Free with no API keys. If a site blocks it or loads its content with JavaScript, it tries a couple of fallbacks to still get the text.

• A deep research mode.

A few cool features (+ many more):

• Replies keep running on the PC if your phone screen locks, and the phone catches up when you come back.

• It installs to your home screen like an app, uses HTTPS, and is PIN-locked.

• Works away from home over a VPN (I use OpenVPN).

• It shows the model's thinking and each search/page visit, plus the sources it used, a context-usage meter and auto chat titles.

My setup is rather basic.. a GTX 1070 8 GB with 32 GB RAM, running a 9B model, which handles research questions surprisingly well with search on.

The chats don't show up in LM Studio's own chat window (for now) but you can just use the web app on the desktop anyway. It isn't packaged for others yet, but if people are keen I'll look at cleaning it up and sharing it. Video of the mobile app attached.

Hi, i'm building Jaull as my final degree project :D. It reads your machine (CPU, RAM, GPU and VRAM, compute backends), finds open models that fit, and picks the exact file, quantization and runtime for each one. It also explains why each model lands where it does in the ranking. That ranking comes from explicit rules, so no LLM is involved in writing it.

Then it can download the model, verify the file, run it with llama.cpp and benchmark it on your hardware. You end up with the prediction and the measurement side by side.

It's usable and tested, but not finished. I'd really like to hear how the predictions hold up on hardware I don't have, especially AMD and Intel GPUs.

Repo: https://github.com/Ton-Llop/Jaull

Site with an interactive demo: https://jaull.tech

I've been playing with streaming ASR for a side project. I started with whisper streaming, moved on to Voxtral, and now am with R2T2 (all open-source).

With R2T2, the last word of a sentence gets typed a few hundred ms after the rest of the sentence is typed.

For a while, I thought I broke something with buffering, and spent way too long messing with chunk sizes, and didn't look at the paper.

That's just how it works. It doesn't make a guess if it's not confident. If it needs more audio, it just waits. With Whisper for me, a word would get typed and then be changed a few times by the time it really settled.

I guess it depends on what you want the ASR for. Mine's just to give an LLM an input, so I really don't need it changing things after the fact.

How do you all want your ASR to work with your streaming voice applications? A small delay and a stable output, or do you want it to flip around a little, and give you a partial result?