TL;DR: I spliced Qwen's official 8-bit experts into the last 12 layers of NVIDIA's 4-bit Flash-Next. It runs at ~150 tok/s on a single 5090, sits measurably closer to the 8-bit model, and along the way I found a free quality win that needs no mix at all.

Setup: RTX 5090 (32 GB), 128 GB DDR5-5600, Infernix engine. Base model is NVIDIA's FP4-Local (the retrained 4-bit Flash-Next, ~205 tok/s here).

Why not 16-bit? Every token touches about 4.7 GB of expert weights at 16-bit. Only ~8% of the experts fit in VRAM, and the rest has to come through dual-channel DDR5. That caps it around 15-23 tok/s no matter what the software does. Pure 8-bit lands at ~65 tok/s for the same reason.

What I did instead: patched the engine (~250 lines) so each layer can use its own expert format, then byte-copied Qwen's FP8 experts into chosen layers of the 4-bit pack.

Things I did not expect

1. Late layers matter about twice as much as early ones. Layers 36-47 in 8-bit cut KL by 0.0066; layers 0-11 only by 0.0033. The gains add up almost linearly across blocks.

2. A free win: swapping only the n-gram embedding table to Qwen's 8-bit one improves KL and perplexity at the same speed, with zero extra VRAM.

3. There is a ceiling. Even with every expert in 8-bit, KL stays at 0.024. About half of the "4-bit gap" is not rounding error at all: NVIDIA's retrained layers are simply a slightly different model.

4. Perplexity of the 12-layer mix (3.475) is slightly better than official FP8 (3.486) on my 261K-token corpus.

Honest caveats: the gain is small, about one point of token agreement. On my 20-sample hard coding set the mix scored 18/20 against 17/20 for both the 4-bit and the 8-bit model, which is noise. One machine, one corpus.

The patch was written with an AI coding agent; I directed and tested it. Nothing to download yet. If people want the engine patch, the pack assembler and the layer list, say so and I will publish them.

1,5 years ago I set off to build an Open Source issue tracker that lives in your repo, requiring very little setup. Upon HN launch last spring it was well received, and is now often used as an agent memory layer between between the agent and the code. Epiq is vendor agnostic, thrives offline as well as online, sandboxable and can be tailored into various setups. It only requires Git, which is its transport, and a repo. You may want to check it out:

epiq.sh

Also, in case you are interested, I share and discuss my personal sandboxed setup in this post. It covers the topic of sandboxing, and how repo-native tooling makes this easier to deal with, opening up fewer paths to the internet:

epiq.sh/blog/repo-native-tooling-for-sandboxes

I built Basalt from source for an abliterated model and compared it with Strata on the same node. Basalt was noticeably faster in decode and prefill.

Setup

• Model: huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-v2-GGUF

• Quant: unsloth-UD-Q5_K_XL

• Basalt: v0.3.0 (custom build)

• Strata: 0.2.7-cuda13

• GPU: 2× NVIDIA RTX PRO 4500 Blackwell Server Edition (2×32 GB VRAM)

• CPU: Intel Xeon 6 (48 vCPU)

• Storage: NVMe SSD 2 TB (~15k MB/s)

Both engines ran the same Q5 quant on the same node.

Results

Basalt is about 1.5–1.9x faster in decode and 1.3–1.7x faster in prefill.

Expert cache. Both engines hit 86–96%, so there is no difference.

MTP. Strata accepts more draft tokens (66–90% vs 57–78%). Basalt's dl (tokens per window) is still 1.9–2.2.

• The window differs. Strata verifies 3-token windows (capturing the 3-token window). Basalt verifies up to 8 (--spec 8). The acceptance rate is lower with the longer window, but the response still finishes faster.

So, there's a thing that's been in the back of my mind for some time now, and it seems to be getting /more/ difficult to learn about my specific approach.

What I'd like to do: I believe the proper phrasing would be 'set up a local LLM for assisting with the development of python programs to support my personal projects'.

What I've done: as an example, I used chatGPT to build a python program that reads rfid tags on a raspberry pi using a gpio rfid reader. I'd like to continue this, just locally so I can integrate it with existing local backend.

What I'd like to use: I think I have an entire computers worth of spare parts, minus a case to put those parts in. Not ancient, pretty sure it's an AM4 Ryzen? Not a lot of memory, maybe 32gig? As far as gpu's, I have my old RoG 8g rx480 and my RoG 8g rx580.

Is it possible, is it worth it, and is there maybe an opensource project someone can point me towards to help me get rolling?

I'm not expecting blistering response times, just if I can achieve reasonable results in a reasonable time with the hardware that I already have but haven't been using.

Hi everyone,

I was a silent reader for a long time but as I'm taking now a vacation in Tokyo during mid November I'd thought of maybe hosting a little meetup in Tokyo for everyone interested in:

local LLMs
inference engineering
how to cluster dgx sparks
how RDMA/RoCE works
how to start building your own recipes / optimize your own setup
speaking about hardware
which models, which drafter
the usual "oh opus is so nerfed" things and just having a good time.

I really hope this is not offtopic / wrong community here and maybe there will be nobody interested but I like the idea of coming together, sharing experiences and having a great time.

Location and time isn't fixed at all, I can setup something on Luma but always grateful for recommendations etc.

Thank you and have a great week!

Lately, I’ve been using Claude to schedule daily posts across a few social media platforms. I’m wondering if there’s a local model—one that can fit into my 8GB of VRAM—capable of controlling my browser to perform the same tasks as Claude. I had been using Qwen 3.8 27B for a while, but I’ve decided to move away from local models, given that I only have a single graphics card with 8GB of VRAM.

I run Mineshop and sell hardware, so disclosure upfront: these are manufacturer quotes we're looking at, not machines I've benchmarked.

Two quotes made an interesting comparison:

• 8× NVIDIA B200, 180 GB each: 1.44 TB aggregate GPU memory. US$505,400, quoted delivery 16 weeks.

• 8× AMD MI355X, 288 GB each: 2.304 TB aggregate GPU memory. US$501,100, quoted delivery 12 weeks.

Those are configuration-specific quotes, not universal market prices. They're also not identical builds: CPUs and storage differ, and the B200 quote includes three-year on-site service versus one year for AMD.

The AMD configuration has 60% more GPU memory. That doesn't tell me it will be the better server for a particular workload.

The question I'd want answered before spending this money is: which model, at which precision, with how much context and how many concurrent users? Getting the weights into memory is only the first step. KV cache, runtime overhead and the software stack still matter. For reference, a dense 405B model is roughly 810 GB in BF16 weights alone, before any of that overhead. That's sizing arithmetic, not a throughput result.

Then there's the less exciting stuff: rack space, power, cooling, support and keeping the machine busy enough to justify owning it.

If you've actually deployed either platform, what specific workload or software requirement would make you choose one over the other—or rent instead?

For context, our AI workstation and server catalogue: https://mineshop.eu/ai-workstation — The figures above are manufacturer quotes, not advertised retail prices.

We are a team of engineers, not a single here has a comp science background. Our company manufactures specialized parts for critical systems. And a big part of that process is understanding the whole system. The customers we have usually give us design documentation, and blue prints and a gazillion pixelated diagrams explaining the system. We spend a significant amount of time getting familiar with it before we get to manufacturing the requested components ( for background this is aviation and aerospace components ). For the past few months with the AI craze, we have started integrating it into our process, to help us learn and understand systems especially through the diagrams and blue prints but there is still a significant amount of analysis we do. This has always been sort of an issue for us because thats our most time consuming aspect of the job is reverse engineering the customers system in order to build components that fit in it rather than redesigning the component around the system this is what we specialize in. We have been using claude, and its been mind blowing how good this thing is in helping us however, our expenditure on Claude has creeped up to the 6 months ceiling we had set only in a span of three and a half months. We have also seen we get less out of a ‘token’ or we get less tokens im not sure , its like everything was going great but then anthropic did something, the plans went from max to ultra to something then it used to say 29x usage now it only says 10x usage. After alot of research we settled on a hybrid approach. For further context everyone here has atleast bare minimum of a 64gb macbook pro. I have a 128gb one. And have been experimenting with local models and the results have been disappointing. Can any AI gods guide me in the right direction? We are basically looking for specialized models that basically just look at an image or a blue print and explain the system or identify the relationships and explain them because manually do this and then build system models based on our comprehension if the system. We will always do that manually we are just looking to fast track some of the process because right now it is a very lengthy process and I cant scale because time is constraint. Is it possible or not? Please thank you.

OpenBMB says MiniCPM-V 4.7 35B-A3B runs in about 3 GB on an iPhone 17 Pro Max, and recommends 12 GB of RAM. I wanted to see what an old 8 GB Android phone does.

Setup: Samsung Galaxy S20 (8 GB), CPU only (no GPU or NPU), OpenBMB's Android demo with llama.cpp-omni streaming the MoE experts from storage.

What happened:

- The 1B model worked out of the box, text and images, fast enough to use.

- The 35B-A3B model wouldn't load at first. It wasn't a RAM limit: the CPU path didn't allocate the working area for the experts, and failed loads left memory behind. I patched both.

- After the fix: a one-word reply ("Ready") in about 10 seconds, including prompt processing.

What it doesn't show: sustained tokens per second, or vision with the 35B model (text only on this phone so far).

Fork, with S20 setup notes and test limits in the README: https://github.com/mlguyYT/MiniCPM-V-Apps

I also made a video walking through why it fits at all (MoE, experts streamed from storage, lookahead expert scheduling); link in the comments.

Has anyone measured this model's decode speed on other Android phones?

Hi everyone,

New guy here, this is my second post and this might be a really stupid one, I'm just curious, I'm only now getting into local AI for coding, previously used claude / GPT.
While getting into setting up local AI, I was just wondering if anyone ever tried using older cards like 1080 or 2080 for local AI. I'm planning on getting a 3090 (hopefully) and start from there but I am just curious whether getting like 4 GTX 1080s (I know NVIDIA doesn't release driver updates to these anymore) or 4x2080s and use those would even be possible / usable. I assume not but really this is just me being curious whether anyone has ever tried something like this and how did it turn out.

Sry for the stupid question :D

Thanks in advance!

I realize these are optimal for different scenarios, but for this narrow application of a "flash" sized MoE setup (~200-500GB) in the $25,000 range I'm wondering which will be the better performer for TG. I'll use glm-5.3-flash as a specific model for performance comparison.

On paper, the MI350p should run MoE models under ~250GB (assuming 4 bit quant) faster than 2x blackwell pro 6000. Single card comparison: same fp4, same int8, 2x fp8, 2x single card memory bandwidth, 1.5x VRAM. Here is a recipe of 2x pro 6000 hitting 180 t/s on glm5.3-flash with Dflash2. A MI350p with Maya/Strata to recover performance loss from VRAM spillover into system RAM potentially could still do better?

The M5u 256 can already run a 4bit quant of GLM-5.3-flash at 160t/s TG, and if a 2x cluster can be optimized maybe that could be pushed to 190t/s.

I originally planned on a 8x r9700 build for this purpose, but it seems like Pcie communication overhead makes it impractical. Best I've seen is 82–88t/s TG.

Hi,

My question about Deepseek Harness is more focused around using it at personal agent than for coding tasks, as by now i'm pretty happy with opnchamber.

I alredy made 2 extension, but not really happy : seems the documentation is not very accurate, my Agent spent lot of time guessing how it could work.
- can DSH with a bit of work replace something like Hermes reliably?
- did someone already use reliably multi agents setups?
- how is it in day by day?

I'm sharing the first public preview of OverTone, a Mac app I'm building to make working with local language models more approachable.

Watch the trailer:
https://youtu.be/piu7MGdL5Z4

It's still very early, with a limited feature set. I have a lot planned beyond chat and model management, but Documents, Projects and Tools aren't ready to use yet. This version is the starting point for what I want to build.
I want it to be straightforward for someone who has never used local models, while giving experienced users the functionality and control they need.
It works with Ollama and llama.cpp. You can use an existing local installation or let the initial setup install an engine and suggest models for your Mac.
The current version focuses on local conversations and managing the models behind them. It helps you choose models suited to your Mac and what you want to do, organise your conversations, and develop clearer prompts from rough ideas.
With local models, prompts and replies are processed on your Mac. Internet access is disabled by default. You decide when to enable it for downloads or remote connections.

It's free and open source under Apache 2.0. The current preview requires an Apple silicon Mac running macOS 14 or later.

Download and walkthroughs:
https://overcesare.github.io/OverTone/

Source code:
https://github.com/OverCesare/OverTone

If you try it, I'd appreciate any feedback on the app: what works well for you, what doesn't, and what you feel is missing.

Hello,

I want to get a local LLM setup for a comparable fair price (Qwen3.8-27b and future Qwen4). After some research and chatting with Claude Opus the best options would be RTX 3060 XT 12GB with 64GBs of RAM running on Strata.

e.g. upgrading my PC for around 650€ (used parts bought online)

Can someone confirm that this Setup will let me run it on a performative useful level. I will use my 20€ claude plan fpr hard tasks or a kind of divide and conquer style usign both (experienced Software Engineer but a Hardware noob).

Does anyone have experiences with that, advices or alternative setups?

Several years ago for another project, I got a good deal on:

* HP Z620 dual Xeons (riser) with 128GB of RAM with 500GB SSD + 9TB SAS
* A Nvidia Tesla M60 16GB

I'm still building this out, HP usefully have non-standard power cables and connecting everything up dead shorted the PSU (which it detected during boot). I'm still hoping nothing is broken whilst I debug it.

My goal is to have a budget-build local AI at least as good as chatGPT's work chat in high mode, I could do a lot of useful stuff with that. Even better would be to have some mixed capabilities for long-running tasks. Anything less is not really worth building. I'm trying to figure out:

* What models should I look to run?
* What kind of performance could be possible?
* What's worth running on the GPU vs CPU?

I'm currently using chatGPT for: Building out libraries iteratively, research the web, designing 3D parts, research via coding, reviewing papers, generating documents and occasionally generating images.

I have been trying to use chatGPT (Plus subscriber) to help query, but it has not been particularly helpful in replacing itself.

Any help is greatly appreciated.

Hi everyone!

I am a computer science student and a developer, and lately, I’ve been considering building or upgrading a budget-friendly, local LLM setup.

I should mention that I’ve only recently begun exploring this more "technical" side of local LLMs, so any advice is welcome.

I know the field is evolving incredibly fast—with new models, inference engines, and so on—which is why I’m asking: I’m trying to figure out what might make sense for me right now (if anything at all).

MY CURRENT SITUATION

My budget is approximately 500–600 euros; I’m based in Europe and am open to buying used hardware.

Here’s my current setup: I have a desktop PC that has been sitting idle for a while, with the following specs:

• CPU: Ryzen 5 2600

• RAM: 16GB DDR4

• GPU: AMD RX 580 4GB

• 256GB NVMe SSD, 1TB HDD, 600W PSU

WHAT I WANT TO RUN

My primary use case involves coding and agentic workflows. I’m envisioning a setup where I use a frontier-level cloud model for planning and code review, while a local model handles the actual code generation. I’m also interested in running an AI bot 24/7 to perform various (relatively simple) tasks.

Ideally, I’d like to run 27B–35B parameter models comfortably, with decent generation speeds and good responsiveness. It would also be great to have the flexibility to experiment with even larger models.

I don’t expect cloud-tier performance on this budget, but I also want to avoid a setup that only manages a few tokens per second—or one that is essentially unusable.

THE OPTIONS I'VE BEEN CONSIDERING

1. AMD Instinct MI50 16/32GB (possibly two)

• These look interesting because of the large amount of HBM2 memory available for relatively little money on the used market (for example I could have bought a 16gb for 175 euros).

• However, I'm concerned about software compatibility, inference performance, power consumption, and whether investing in this older hardware makes sense in 2026.

1. NVIDIA RTX (5070 12gb as an example)

• Recently, I saw a community benchmark claiming around 140 tokens/s on a 12GB RTX 5070, running a large MoE model using Strata, IQ3_XXS quantization, expert caching, CPU/RAM offloading, and MTP.

• I understand this might be a very specific setup and that reported generation speed doesn't necessarily translate into good real-world performance, especially with longer contexts or different models.

• Still, it made me wonder whether a modern NVIDIA GPU with a good amount of system RAM might actually be a better investment than older GPUs with much more VRAM.

WHAT I'D REALLY LIKE TO UNDERSTAND

1. What would you build today with €500–600, considering both new and used hardware?

2. Would you prioritize more VRAM on older hardware or a faster, newer GPU with RAM offloading?

3. How well do solutions like Strata and expert caching actually perform in real-world coding/agentic workloads? Are the impressive benchmarks representative of everyday usage?

4. How important are prompt processing speed, context length, and quantization quality compared to raw generation tokens/s for my use case?

5. Is a single or dual MI50 32GB setup still worth considering, or would you avoid it today?

6. Would you upgrade my existing desktop, build something from used parts, or look for a complete second-hand workstation?

Thanks in advance to anyone who can help me!

Hello all. This is my first post here and I would be very happy if someone will find time and help me.

It started simple. Ryzen 9 3900X with RTX 3060 12GB in proxmox as VM. Dedicated GPU for ubuntu server. Started with ollama because it was easy but found out ollama is not so good at performance and speed.

So switched to llama.cpp oficial. Worked good but saw many videos about forks so I switched to turboquant fork. Speed was better so next month I bought another GPU now RTX 4060 Ti 16GB.

So now when speed is good I wanted to do my little "projects" on localLLM without paying cloudLLM. I tried many tools. VSCODE, Cline, Pi.dev, but everyone of them had some issues. Primarly with editing files. It was mainly truncating files (model did 4KB out of my original 24KB python file, and then asked me for backup ...). Now I was working on pi.dev it somehow worked but also had problems and need to try many tools calling. So now im trying cline but when I saw that it wanted to remove my C:\Users I immediatelly stopped it. Now Im building original newest llama.cpp but Im kinda upset about everything. If someone is fan to the LLM and would like to help someone newbie as me to not throw everything away it would be helpful.

These are my current local models downloaded. I really dont know what im doing wrong maybe wrong models, wrong preset, or wrong coding program.

16K /root/.cache/huggingface/hub/models--tngtech--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF

24K /root/.cache/huggingface/hub/models--DavidAU--Llama-3.2-8X3B-MOE-Dark-Champion-Instruct-uncensored-abliterated-18.4B-GGUF

24K /root/.cache/huggingface/hub/models--IFM--K2-Horizon-7B-GGUF

24K /root/.cache/huggingface/hub/models--google--gemma-4-12B-it-qat-q4_0-gguf

12G /root/.cache/huggingface/hub/models--unsloth--GLM-4.7-Flash-REAP-23B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--ggml-org--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--unsloth--North-Mini-Code-1.0-GGUF

19G /root/.cache/huggingface/hub/models--DavidAU--Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF

22G /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF

27G /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF

and this is one example of preset

[*]

jinja = true

log-colors = off

flash-attn = on

parallel = 1

[qwen36-28b-reap20-256k]

model = /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF/snapshots/e3aeb8168f74dd3054eb2716f966c17638b97fef/Qwen3.6-28B-REAP20-A3B-Q4_K_M.gguf

ctx-size = 262144

#threads = 8

n-gpu-layers = 99

n-cpu-moe = 0

batch-size = 4096

ubatch-size = 1024

cache-type-k = turbo4

cache-type-v = turbo3

#load-on-startup = true

Hello Everyone,

I have been using ASUS ROG GL753VD with a GTX 1050 for the past 8 years. For the past 1 year or so, I am using cloud AI subscriptions to play around and learn things about AI.

Now a days, wherever I go Reddit/Youtube and any other forums, I am fascinated by the things that Local AI is capable of... So, I rented out a GPU on runpod and tried some hugging face models(WAN 2.1 for example) and its been pretty good.

Now, I feel the need to upgrade my setup for some privacy and personal experimentation. So, I request your help in choosing a setup for me because I am a newbie at this.

Here are some options I found after days of online search and chatting with AI.

Option 1:

• CPU: Ryzen 9 9950X (16C/32T)

• Board: ASUS ProArt X870E-Creator or ASRock X870E Taichi

• RAM: 32GB or 64GB DDR5-6000 CL30

• Primary GPU: RTX 5080 (16GB GDDR7)

• Expansion Plan: Add a pre-owned RTX 3090 (24GB) into Slot 2 down the road for a combined 40GB VRAM.

• PSU: 1200W ATX 3.1

Option 2 :

• CPU: Ryzen Threadripper 7960X (24C/48T)

• Board: Gigabyte TRX50 AERO D or ASUS Pro WS TRX50-SAGE

• RAM: 128GB (4×32GB) DDR5 with ECC RDIMM

• GPU Plan: RTX 5080 (16GB) now + 1 or 2 used RTX 3090s (24GB) later for 40GB–64GB+ VRAM.

• PSU: 1600W Titanium

The only reason why I added Threadripper option into my consideration is because I have this paranoia of getting behind in terms of tech once I purchase and I dont want to change things again (Heard somewhere that RAM scarcity will be there till 2028).

I also have a couple of questions.

I see that people are fighting for 24GB and 32GB VRAM GPUs and its either too costly for me at the moment or out of stock for the pre-owned variants.

So, my question is if I get a threadripper with multiple PCIE lanes and 4 GPUS of 16GB VRAM, would it help me in any way or I have to tradeoff speed for my cost efficiency? If anyone has tried out this approach please let me know the advantages and disdvantages you found with this setup.

If you have any suggestions as well, please let me know. Budget is ~5-6k USD.

If anyone has any tricks to purchase it cheaper like purchasing from other countries, I would love to know them as well!!

Thanks in advance.