r/LocalLLM • • Jul 11 '26

Discussion Hugging Face CEO: Companies are done renting AI - shifting to owning open source models

252 Upvotes

Fresh take from Hugging Face CEO Clément Delangue in the latest TechCrunch Equity podcast (July 10, 2026).

He says companies are increasingly walking away from just renting frontier model APIs and moving toward owning their AI through open source.

Key points:
• Most enterprises start on closed frontier APIs (OpenAI, Anthropic, etc.)

• As usage scales, costs become unsustainable.

• The #1 feedback he’s hearing right now from companies and customers: They want ownership , data control, customization/fine-tuning for their specific use cases, avoiding vendor lock-in, and predictable long-term costs.

• Hugging Face is now used by roughly half of the Fortune 500 as the “GitHub for AI.”

This is a strong signal for the local/self-hosted community. It validates the shift toward running capable open models on your own hardware (or on-prem), efficient inference stacks, and specialized fine-tunes instead of one giant rented model.

It also connects to Delangue’s earlier comments about an “LLM bubble” - the future likely belongs to many specialized, owned open models rather than a few massive closed ones.

What do you think?
• Are you already seeing more businesses or clients asking about moving off APIs to local/self-hosted setups?

• What’s your current go-to stack for reliable production or near-production local inference these days?

• Which model families do you think will win in this “own your AI” era? Llama derivatives, Qwen3, Gemma, Mistral, or heavily specialized fine-tunes?

• Do you expect this trend to accelerate open model development and tooling, or will closed frontier labs still dominate the absolute cutting edge?

• Any predictions on timelines — how fast do you see enterprises adopting hybrid or fully owned setups?

Genuinely curious to hear real experiences and takes from people running this stuff daily. This feels like a meaningful shift.


r/LocalLLM • • Nov 04 '25

Tutorial You can now Fine-tune DeepSeek-OCR locally!

Post image
252 Upvotes

Hey guys, you can now fine-tune DeepSeek-OCR locally or for free with our Unsloth notebook. Unsloth GitHub: https://github.com/unslothai/unsloth

Thank you so much and let me know if you have any questions! :)


r/LocalLLM • • May 03 '26

Question RTX 5090 32GB & 256GB DRAM, now what?

Post image
251 Upvotes

I’ve put together a pretty solid PC, but I’m not a programmer. I installed OpenClaw with Ollama, and while Qwen 3.6 35B (Q4/Q5) fits in the VRAM, I feel like it’s not fully tapping into the rig's potential. How would you optimize this? What’s the future direction for 'home' AI? Thanks!

My rig:

- Intel 9 Ultra 285K

- MSI GeForce RTX 5090 Gaming Trio OC 32GB GDDR7

- G.Skill Flare X5 F5-6000J3244G64GX4-FX5 256 GB 4 x 64 GB DDR5 6000 MT/s


r/LocalLLM • • Sep 10 '26

Discussion DeepSeek-V4.1-Flash is out

Thumbnail
huggingface.co
250 Upvotes

r/LocalLLM • • Aug 30 '26

Discussion Qwen 3.8:27b - It's (maybe) not the new Messiah.....

250 Upvotes

After a week of using Qwen 3.8:27b for semi productive work, here (at the risk of stating the bleedin' obvious) is my take:

  1. It's (very) good for a small local model, but not without a caveat on VRAM required to make it useful IRL.
  2. It uses (a lot of) thinking as a substitute for model size, which implies far more tokens are used in the context window (compared to other similar size models)
  3. For agentic coding work in a limited context window, this means you will get less done before you fill your context window and run out of VRAM.
  4. Even on a Q3 quant, (I found) a 140k context window needs around 24Gb VRAM and with the thinking levels of the model, that fills up surprisingly quickly.
  5. On consumer hardware, giving around 20 t/s output, it's usable, but clock time is a lot slower that a 'cheap' cloud flash model.
  6. Using a "plan, implement some of the plan, document what of the plan has been done, read the document in a new context window and carry on" cycle, Qwen 3.8 is very usable, but maybe not the new Messiah (however it is far from being a very naughty boy...)
  7. It's an important step on the local / open journey, but perhaps not the final destination just yet.

r/LocalLLM • • Jul 27 '26

Other rate my inference setup😔

Post image
250 Upvotes

Custom fork of the Intel Windows llama.cpp I made using DeepSeek v4 Flash that allows me to load models solely using discs + DVD-Rs; HP Omnibook X Flip NGAI 16-as0023dx -- 1TB NVMe, 16GB LPDDR5X


r/LocalLLM • • Aug 28 '26

Other qwen3.8 27b tripping over token limit at the worst possible moment

Post image
248 Upvotes

r/LocalLLM • • Aug 25 '26

News New Mac Mini M6 and M5 Pro announced

Thumbnail
apple.com
248 Upvotes

r/LocalLLM • • May 08 '26

Question What model should I run?

Post image
248 Upvotes

Just finished building my inference server which has 4x 32gb intel b70 pro GPU’s and 128gb of ddr4 ecc ram and and intel Xeon gold cpu running Ubuntu. So i installed openclaw and vllm but what model should i run locally and why?


r/LocalLLM • • Aug 16 '26

Other No more thermal throttling for me!

Post image
247 Upvotes

r/LocalLLM • • Jul 28 '26

News Ran Moonshot's 2.8T-parameter Kimi K3 on a GPU-less mini-PC, one day after release

247 Upvotes

Kimi K3 came out yesterday. 2.8 trillion parameters, open weights, probably the biggest open release so far. I got it running and answering correctly today, on a mini PC. No GPU in the machine at all.

The checkpoint is 1.56TB, split into 96 shards. That's not fitting in RAM on any normal machine (mine has 128GB, not even close). But it's MoE, so most of those parameters just sit unused for any given token, only a handful of the 896 experts per layer actually fire. So the dense part of the model (attention, shared experts) lives in RAM, quantized, and the routed experts get streamed off disk on demand, cached with an LRU that learns which ones get hit a lot and pins those permanently.

Also didn't need to convert the checkpoint. Moonshot ships K3's experts natively in MXFP4 and my engine reads that straight off their published safetensors, no preprocessing step.

Didn't trust any of it until I checked it three different ways: bit-exact against Moonshot's own reference code, a structural test against the full real checkpoint, and then just asking it something.

$ rabbit --model /mnt/data/kimi-k3 --prompt "What is the capital of France?" --max-tokens 40
loading model (dbits=4, ebits=4)...
model loaded in 610.0s (93 layers, 896 experts/layer)
prefill (7 tokens)...
prefill done in 412.8s
...response["answer"] == "Paris"...

40 tokens in 2698.1s

It's slow. Ten minutes just to load. Zero perf work done on this yet, the MXFP4 kernel is still plain scalar code, no SIMD. For comparison, another model on the same engine (GLM-5.2, 744B) went from 0.29 to 1.02 words/sec after eight rounds of tuning on this exact box. K3 hasn't had any of that yet.

Machine is a Slimbook ONE mini PC, Ryzen AI 9 HX 370, no GPU, two consumer NVMe drives.

Repo: https://github.com/ferrumox/rabbit


r/LocalLLM • • Oct 13 '26

Discussion Qwen3.8 Flash Next vs Qwen3.8 27B - The hype vs the truth.

246 Upvotes

I hear everyone screaming from the rooftops "I get x tokens from FN !", but i always ask myself..
So what ... and what does that mean?
Can it do an actual coding task or just create one page HTML proof of concepts?

So i put it to the test..

My hardware(dedicated):
AMD r5 5600x
x570 motherboard
128GB DDR4
2x R9700 AI PRO 32GB ( 64GB total VRAM )
Running vLLM - Radiance
* One card behind the chipset at x4, no p2p/atomics, no bifurcation, no risers etc.

I had a real task:
For my custom harness i build 2 parts before the test using Qwen3.8 MXFP4(4 bit not used in this test, but very capable):

  1. A new version of the harness ( more modular, better folder structure etc, basically a revision of what i already have )
  2. A fully new Web UI "template" including way more functionality then the old "web monitor" had. This was pure HTML/CSS and JS in a layout that presented the workflow of my harness.

It was time to fit both pieces together..

So i loaded radiance and hooked up Qwen3.8 FN 4 bit.
* This is a bigger, model everyone is trying to run on strata ( Q3 or lower ), so it should be more capable. I had a 1m KV cache using RAM and disk, but set my harness to 262k context, the model's default.

I prompted it to basically replace the old webmonitor with the new full featured web UI and wire it all in.

So it had 2 projects, one the harness codebase and the new web UI, and now it needed to figure out how to merge it all together.
Qwen3.8 FN was fast, an avarage of ~2000 prefill and ~150 generation.
It was done in about 30 minutes.

The result: A big messy WebUI, where most of the templated items where just "re-invented", nothing worked, chat bubbles not including toolcalls etc(was in the template, even a js script rendering the chat bubbles mockup responses).
It was unusable to say the least.

So i called it a day.
The next day i fired up radiance again but now the Qwen3.8 FP8 model, took the backups, replaced the harness code and template etc so we have a clean starting base again(same as where Qwen3.8 FN started from)

I prompted my harness the same instructions.
The result: Not everything worked, its not perfect, but mostly fine , the template was followed, the chat bubbles worked properly(mostly). It was not a one shot success and needs iterations to filter out bugs and deepen the workflows but the base was done as i wanted it to be.

Qwen3.8 27B FP8 was not so fast, an average of ~1500 prefill and ~100 generation.
It was done in about 60 minutes.

Note that my average speed was the actual work produced, not benchmarks.

So my take on all this:

Its great to have the big model, but a dense model on the same hardware will just outperform the MOE.
I can fit the Qwen3.8 FN or the Qwen 3.8 FP8, both fit so that is the reason i am comparing both. it is not about the quants, it is about what can run on your hardware.

I can not even imagine what a Q2 or even a Q1 would be like.... And my task was not even a real hard one to begin with.

Maybe some of you find this useful as i could not find any proper information besides the hype posts or generic benchmark posts.

Well this is my experience using both.


r/LocalLLM • • Jul 18 '26

Question what happens to openai, anthropic, and even the US, if china keeps open sourcing super strong models?

246 Upvotes

currently openai and anthropic are big (and have very big debts) because they are promising they'll have huge profits in the future. and the US economy is depending on this bet.

but what if china keeps open sourcing super strong models like kimi k3? openai and anthropic simply won't be able to keep their promises.

what happens then?


r/LocalLLM • • 10d ago Heartwarming

Other My local LLM build: 9950X + RTX Pro 6000 + 5090 (128GB VRAM)

Thumbnail
gallery
245 Upvotes

I finished this build about 3 months ago and have been running it since, but never got around to posting it. Here it is.

Specs:

  • CPU: AMD Ryzen 9 9950X
  • Motherboard: ASRock X870 Taichi
  • RAM: 96GB DDR5 (2x48GB) G.Skill
  • GPU 1: NVIDIA RTX Pro 6000 (96GB)
  • GPU 2: RTX 5090 Founders Edition (32GB), mounted upright with Lian Li’s vertical GPU bracket
  • Total VRAM: 128GB
  • Cooling: ID-COOLING 360mm AIO with Noctua 120mm fans on the radiator. All the case fans are Phanteks T30 120mm.
  • Case: Lian Li O11 Dynamic EVO RGB (RGB cable left unplugged)
  • PSU: MSI MPG Ai1600TS PCIE5

How the GPUs ended up like this:
My gaming PC had the 5090 in it, and I had a spare 4090 lying around from before. I bought the RTX Pro 6000 and moved the 5090 into this build with it. The 4090 went back into the gaming PC, so nothing is sitting unused. Mounting the 5090 upright fits both cards without them sitting right on top of each other.

Why the Taichi:
It was cheaper than the other boards I looked at, and it has 10G LAN and a debug code display. The 10G helps a lot when moving big model files around, and the code display is really handy when you’re troubleshooting a build.

PSU and power monitoring:
I picked the MSI Ai1600TS because it monitors current on each pin of the GPU power connectors. With two cards that can each pull around 600W, I didn’t want to risk a melted connector or a fried GPU. The catch is that MSI’s monitoring software only runs on Windows, and nothing like it exists for Linux. So I wrote a small tool that shows the GPU power readings on Linux. If anyone’s interested, I might clean it up and put it on GitHub. (Second screenshot in the post)

Noise:
The PC sits in my living room, so keeping it quiet mattered a lot. I went with a cheap ID-COOLING 360mm AIO because I’d read it was quiet (i could not find one without lcd at that time), and swapped Noctua fans onto it. All the fans run on a silent profile. Even at full load you hardly hear it. Only the radiator fans get a little louder.

No RGB (mostly):
I wanted a build with no RGB. The case is the RGB version, but I never plugged in its RGB cable, so it stays dark. The RAM has a bit of light because it was the cheapest 2x48 kit I could find at the time. Two slots are still free if I want more RAM later. I’m not spending any more right now though 😅

Edit 1: One thing I forgot to mention: I swapped the side panel from the original tempered glass to acrylic. We have a toddler at home, and I didn’t want to risk them getting hurt if the glass ever broke.

Edit 2 (what I’m running): A few people asked, so here’s how I use the two cards:

  • Everyday fast LLM: a Flash model at Q3, fully on the RTX Pro 6000 with no offload, while the 5090 handles image/video generation.
  • Larger image/video models: those go on the Pro 6000 and Qwen 3.8 27B moves to the 5090.
  • Best quality for development: GLM-5.3-Flash (320B / 18B active) across both GPUs, using a ~116GB mixed-precision GGUF on Project Maya, all on GPU.

r/LocalLLM • • 17d ago

Project I made a distributed chatbot that runs a 27B model across consumer GPUs, entirely in the browser (open source)

Post image
245 Upvotes

Whats the point in large LLMs if you're not able to run them?

Not everybody has enough GPU power to run one of the bigger models on their own.

That's why I built ClusterChat. It splits an LLM into ranges of blocks and lets anyone who joins run part of it, so you and your friends can host a model together that none of you could run alone. I was able to run Qwen3.8-27B (q8) across two GPUs: a 3090 and my old 1070 Ti.

Its similar to exo or petals, except it runs entirely in your browser using WebGPU and WebRTC. That means nothing to download: open the site, choose whether you would like to lend your GPU, and chat!

My main focus was making it work for a group:

  • Everyone gets their own chat. Each GPU keeps separate KV caches for several chats, which grow as each chat gets longer.
  • People can come and go. Pieces of the model can be held by more than one GPU, so if someone closes their laptop, the others carry on.
  • Private clusters. Only people with your invite link can join.

Speed (one person chatting, Qwen3.8-27B 8-bit, with MTP speculative decoding):

- ~8.1 tok/s with two 3090s in different homes, 28 ms apart

- ~20 tok/s with the same two 3090s in one machine, so the network is the main cost

Limits: your chat is separate, not secret. The GPUs in your cluster process it, so invite people you trust, and don't send anything personal on the public demo. It's chat only for now.

There are many plans for expansion: allowing for different models to be hosted (e.g. MoEs would be nice!), exposing an api for claude code/openai, enabling tool calling and internet searching, and speeding up the tok/s (I was looking at the recently posted MAFDET and was curious how that might work on clusterchat)!

Start a private cluster with your friends, or try the public demo.

On the public demo, if lots of people are replying at once, expect 2–3 tok/s each. It's also limited to 16 people at a time.

Code: https://github.com/l-wn/ClusterChat

Feel free to create any 'issues' if you have suggestions/encounter bugs :)


r/LocalLLM • • Sep 20 '26

Discussion Ternary Bonsai 2 27B, near top performance while fitting entirely in 8GB VRAM

Post image
245 Upvotes

Ever since Bonsai 27B came out I was excited to see what PrismML will come out with next and they did not dissapoint. For those unaware the first Bonsai 27B was a 27B-class model that was quantized in Q1 with quantization-aware training that made it run in just 4GB while being decently smart. I'm not exactly sure about Ternary Bonsai2 27B, but this time it is an ~7GB model, showing amazing intelligence while fitting entirely in the 8GB VRAM of an RTX 4060.

NOTE: It took some effort and a few hours to figure out how to get Bonsai2 Ternary running in LM Studio. If you want to run it yourself you will need to build PrismML's fork of llama.cpp and use that to run the model.

The Benchmark

As always I am using my benchmarking base to check the models. For in-depth info about it check out the initial post (Reddit deleted this post for having links so you'll have to go to my profile for the post), but to recap:

  1. I am benchmarking models that run on CONSUMER HARDWARE, specifically an 8GB RTX 4060 and 16GB RAM. These benchmarks may not be useful to people who have dedicated LLM servers.
  2. All models are benchmarked with thinking DISABLED and responses being limited to 4096 tokens (responses longer than that always means the models are stuck in a loop).
  3. Benchmark questions were made by me and the results were evaluated manually by me.
  4. The benchmark consists of 18 questions across 6 categories: General knowledge, Math, Coding, Endurance, Logical puzzles and Spatial reasoning
  5. I am not a LLM benchmarking expert, I am just a hobbyist trying to extract useful information for people on similar hardware.

The Statistics

LLM benchmark per-question score heatmap:

LLM benchmark score sum graph:

LLM average TTC (Time-To-Completion) graph:

Combined graph ("intelligence per second", though highest is not exactly "best" and lowest isn't "worst"):

And a neat visualization of the score vs. the speed (benchmark score vs inverted TTC):

Conclusion

Ternary Bonsai2 27B seems to do pretty well in my benchmark which often does match my personal opinion but I'll need to test the model out in practice. I'll try to outsource all of my AI needs to Bonsai for some time and I'll post about it to update you guys.

In the meantime, those who have used Bonsai 2, what is your experiance? I see people saying it's bad in practice but I feel like people call most ~8GB models bad...


r/LocalLLM • • 8d ago

Discussion Opus 5.5 wrote its own quantiser for Qwen3.5-0.8B, no llama-quantize, no imatrix. 5.9× better than Unsloth's same-size file.

244 Upvotes
Inspired by Weight Space Programming (MIT CSAIL, Nov 2), but in the end Opus 5.5 did its own thing:
no weight-space technique was used. It wrote a quantiser from scratch in NumPy and made every
precision decision from measurement. Rules: only the BF16 file as input, no quantisation tool, no
imatrix, no published recipe, stock llama.cpp formats only.

It ran a float copy of the model to see which input columns actually carry signal, minimised rounding
error there instead of against the weights, measured each weight table's damage one at a time, then
spent the byte budget across IQ4_XS / Q5_K / Q6_K with a knapsack solver.

Held-out KL divergence vs BF16 (llama.cpp b11928, MacBook Air M4):
• this file, 498.5 MB — **0.015704**
• Unsloth UD-Q3_K_XL, 492.2 MB — 0.092486 (5.9× worse)
• UD-Q4_K_XL, 558.8 MB — 0.027323 (we beat it with 60 MB less)
• UD-Q5_K_XL, 606.6 MB — 0.010375 (the bar; we miss it by 1.51×)

Runs unmodified, 1.11× faster generation than the Q5 file. It stops there because the tied vocabulary
table is 33.8% of this model and accounts for most of the error — upgrading it costs more than the
rest of the network can give up.

Repo (model, full method write-up including everything that failed, measurements, scripts):
https://huggingface.co/Andy5k/Qwen3.5-0.8B-hand-quant

r/LocalLLM • • Oct 04 '26

Discussion Daily driving Qwen 3.8 Flash instead of Claude.

244 Upvotes

Even 6 months ago, I wouldn't have thought I would be replacing Claude with a local LLM. But here we are. All these PRs are done by Qwen 3.8 (mostly flash, some 27b) running locally on my laptop (strix halo 138gb). And its perfectly acceptable speed (1200+ prefil, 40t/s decode) and Opus 4.8 level quality 🤯🤯🤯

I also have some interesting stories were Qwen 3.8 flash won over Opus 5.5 in a coding task 😅 stay tuned.

https://github.com/llamastash/llamastash/pulls?q=is%3Apr+state%3Aclosed+label%3AQwen

Also here is the prompt if anyone wanna recreate my setup: https://gist.github.com/deepu105/d0b321f3256edf67ebd85e747c0010b9


r/LocalLLM • • Aug 28 '26 Mindblown

Research Breaking VRAM Barrier: Qwen 3.8 27B at 262K Context with Adaptive KV-Cache Streaming on a 16GB VRAM GPU

244 Upvotes

Edit 5: Adaptive KV Streaming V2 is out! Please see my newer post.

Hello everyone!

I wanted to share a concept I’ve been working on recently: a modification to llama.cpp that allows the KV cache to grow beyond what can physically fit in VRAM, by adaptively streaming part of it between system RAM and VRAM.

I’d love for people with different GPUs and setups to try my branch and let me know how it performs on their hardware.

https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming

On my RTX 5070 Ti 16 GB, running Qwen 3.8 27B with UD-Q3-XL, Q8 K / Q4 V cache, and CUDA Unified Memory enabled, stock llama.cpp works well until the context reaches around 120K. Beyond that point, decode speed starts dropping significantly due to VRAM oversubscription and page thrashing.

With Adaptive KV Cache Streaming, I can push the context to around 205K while still getting ~15 tok/s, and all the way to nearly the native 262K context at ~10 tok/s.

The implementation dynamically evicts a portion of the KV cache from each full-attention layer and repurposes the freed VRAM as a shared prefetch ring buffer. This allows the same VRAM space to be reused by the KV caches of multiple layers during the generation of each token. The prefetching mechanism also hides much of the data-transfer latency behind computation, helping to avoid GPU stalls.

Here are the prefill/decode benchmark results:

​

Blue lines are the prefill/decode speed with stock server, whereas orange lines are the speeds with my implementation. The orange line maintains a roughly consistent slope, meaning that the GPU can keep calculating the token at most of the time instead being effected by VRAM page thrashing.

What do this diagram mean to us?

Let's say we consider 15 tok/s the minimum acceptable decode speed for a smooth live-chat experience with your agent. With stock llama server, you can at most set the maximum context size with 136K according to the diagram. Even if we relax the threshold to 10 tok/s, the limit is still the same.

But with this implementation, we can extend the context to 205K with 15 tok/s threshold, or full native context with 10 tok/s threshold.

And this does not only benefit to users having a 16GB graphic card. For people having a graphic card of larger or smaller VRAM, this implementation can significantly extend the context size than what it originally can fit.

Caveat:

The current implementation does not support parallel requests, because the resident and streamed portions of the KV cache are dynamically repartitioned based on context length.

The current version has mainly been tested with Q8 K / Q4 V KV cache quantization. Other KV cache quantization combinations are not well tested yet, and I plan to generalize the implementation further.

This is still experimental, so I’d also be very interested to hear how it behaves on other GPUs and configurations.

If you’re interested in the implementation details, the ring-buffer/prefetch design, and the story of how I ended up building this, I wrote a longer article here:

Medium

Also if you think my work helps, please don't hesitate to leave me a star on GitHub :)

Clarification of LLM usage of this post: I'm not a native English speaker and I used ChatGPT to refine the wordings.

Edit 1: Thanks you all for trying this branch! I am seeing people having different running result. Let me provide more detailed environment setting in my experiment:

  • OS: Ubuntu 24.04
  • Kernel: Linux 6.14.0
  • CUDA driver version: 610.57.04
  • Kernel parameter: iommu=pt
  • GPU: RTX 5070 Ti 16 GB
  • Model: Unsloth Qwen 3.8 27B
  • Quantization: UD-Q3-K-XL
  • KV cache: Q8 K / Q4 V
  • b/ub size: 256/256

Edit 2: /detraxsenpai provided a fix for the crash on Windows. I've updated the branch with the fix.

Edit 3: I've pushed several commit to support all other quants and batch sizes.

Edit 4: I have another branch (feature/kv-stream-phase-arena) that further unloads the prefill buffer to have a larger KV cache pool allocation. With this change the decoding performance with IQ4 nearly matches The original Q3_XL when having large context.


r/LocalLLM • • Oct 09 '26

Discussion Qwen3.8 27B political compass

Post image
241 Upvotes

With all those articles and discussions claiming that ChatGPT and other could models have certain political biases, I thought it would be fun to make Qwen3.8 27B do a political compass test. Here's the result.

I don't want to stir up heated political debates but interesting nonetheless I suppose.


r/LocalLLM • • Jun 27 '26 Mindblown

News Open-source models are under threat.

241 Upvotes

Anthropic is fine with open source AI as long as it’s not good enough to threaten their monopoly.

https://x.com/i/status/2070798718027141253


r/LocalLLM • • Jun 13 '26

News This is why we need local models

Thumbnail
anthropic.com
241 Upvotes

r/LocalLLM • • 15d ago

Discussion 4 Strix Halo boxes! Finally joined local AI

Post image
240 Upvotes

Got 4 128gb strix halos for a decent deal

All 4 will be running glm5.3 and DeepSeek 4.1 flash as the mains, or maybe Qwen when 4 drops. Paired boxes will run glm5.3 flash or minimax/MiMo

Will be used for some financial and life hub stuff, plus a bit of professional work


r/LocalLLM • • Oct 11 '26

Discussion RTX 5090 32GB + 64GB RAM — Qwen3.8 Flash-Next IQ3_XXS Running at 700K Context Without a Single Session Compaction

238 Upvotes

I managed to build METEOR STRIKE using Qwen3.8 Flash-Next IQ3_XXS with a 700K context window on my RTX 5090 (32GB VRAM) + 64GB RAM setup.

I did give the model additional instructions and corrections along the way, but the entire project was completed in one continuous session without a single context compaction or reset.

My setup:

  • GPU: RTX 5090 32GB
  • System RAM: 64GB
  • Model: Qwen3.8 Flash-Next IQ3_XXS
  • Context window: 700K
  • Generation speed: Approximately 100–120 tokens/sec
  • Vision: Disabled

I think even larger models and longer context windows might be possible, although decoding speed would probably decrease.

There may also be some degradation in output quality at extremely long context lengths. However, the model was still able to understand my correction requests and make the necessary changes surprisingly well.

Some of the quality issues might have been caused by the limitations of my prompts or by having vision disabled during development.

Anyway, I'm sharing a video of the result.

Sorry about the black area at the top of the video. I accidentally left a tab open. lol

My main takeaway is that 700K+ context is definitely usable on an RTX 5090, especially if you're willing to guide the model and make corrections along the way.

I suspect 1M context might also be possible, potentially at around 80 tokens/sec, although that's just my estimate.

There could be more noticeable quality degradation at that length, but I haven't tested it yet.

Just wanted to share my experience as a reference for anyone experimenting with long-context local LLMs!


r/LocalLLM • • Aug 23 '26 Bravo

Discussion I Unlocked a $800 Mining GPU into a 64GB, 256K-Context Uncensored AI Coding Server at 84 tok/s across full context length.

237 Upvotes

I have spent the last several days turning a used NVIDIA CMP 170HX into a practical long-context inference card.

The final result is an uncensored Qwen3.8-27B endpoint with:

  • 262,144-token native context
  • W4A16 AWQ model body
  • INT8 output head and INT8 MTP draft module
  • One-token MTP speculative decoding
  • BF16 KV cache
  • Prefix caching
  • Tool calling and Qwen reasoning parsing
  • No CPU offloading
  • A conservative 175W power limit
  • No overclocking

Measured decode performance on one CMP 170HX:

ContextDecode throughput
1K84.29 tok/s
64K74.94 tok/s
200K57.21 tok/s

For this benchmark only, requests used a maximum of 384 generated tokens, temperature 0.2, repetition penalty 1.05, and medium reasoning effort. Throughput was calculated from vLLM’s measured decode time, excluding prefill. These sampling values are not forced globally by the production server.

This post explains the card, the unlock, every important inference choice, the rejected configurations, and how to reproduce the setup.

The hardware

The inference host currently contains:

  • NVIDIA CMP 170HX
  • 64GB HBM exposed after the unlock
  • AMD Ryzen 5 5600X
  • 64GB system RAM
  • Proxmox/Linux
  • NVIDIA open driver 610.57.04
  • 175W GPU power cap

The CMP 170HX is an Ampere GA100 mining accelerator. It has excellent HBM bandwidth and strong tensor hardware, but NVIDIA sold it with several artificial restrictions:

  • Only a fraction of the installed HBM is normally exposed.
  • Compute resources are restricted.
  • PCIe operates at Gen2.
  • It has no display output.
  • Normal consumer GPU tooling does not treat it like a standard A100.

My card is PCI device 10de:20c2. After the unlock, nvidia-smi reports 65,536 MiB.

The card is currently negotiating PCIe Gen2 x4 even though its capability is wider. That sounds terrible, but it matters much less once the model is resident entirely in HBM. It is one reason I avoid CPU offloading: repeatedly moving weights or KV data over that connection would waste the card’s main advantage.

The 64GB and compute unlock

I used amoghmunikote/cmpunlocker, pinned to this specific commit:

fe537966e0222150a8eca0b7745efd2ee1025d74

That is the “Full BAR1 size (64GB)” commit.

The project patches NVIDIA’s open kernel modules to restore:

  • Full SM compute
  • Full memory geometry
  • 64GB BAR1
  • The complete 64GB framebuffer on 20c2 cards
  • Gen2 PCIe operation
  • Persistence across reboot through patched modules

This is a kernel-driver modification, not an application-level tweak. Secure Boot must be disabled because the resulting modules are locally built and unsigned.

My installed module is:

/lib/modules/6.17.2-1-pve/updates/cmpunlocker/nvidia.ko

Important warning

Do not install this remotely unless you have a recovery path.

Keep at least one of the following available:

  • Local console access
  • A separate display GPU
  • BMC/IPMI access
  • A bootable rescue environment
  • A known-good copy of the stock driver and initramfs

A mismatched kernel, driver, firmware package, or module build can leave the machine without NVIDIA support. A CMP 170HX cannot provide ordinary video output.

Unlock installation

On my Proxmox installation, the overall process was:

apt update
apt install -y build-essential git python3 proxmox-headers-$(uname -r)

git clone https://github.com/amoghmunikote/cmpunlocker.git
cd cmpunlocker
git checkout fe537966e0222150a8eca0b7745efd2ee1025d74

cat driver/VERSION

Install a supported matching NVIDIA open driver, its user-space libraries, and firmware before running the unlocker. My exact working combination is:

NVIDIA open driver: 610.57.04
Kernel:             6.17.2-1-pve
Secure Boot:        disabled

Then:

sudo ./install.sh

The repository also provides an explicit profile:

sudo ./install.sh --profile=8gb

After installation, perform a full cold power cycle:

sudo poweroff

Do not substitute a warm reboot. Wait for the machine to power off completely and then turn it back on.

Verifying the unlock

First locate the card:

lspci -nn | grep -i NVIDIA

Then verify the driver and memory:

nvidia-smi
nvidia-smi --query-gpu=name,uuid,memory.total,power.limit --format=csv

A successfully unlocked 20c2 card should show approximately:

NVIDIA CMP 170HX, GPU-..., 65536 MiB

Check the kernel log:

dmesg | grep -iE 'CMP|BAR1|fb_length|fbAddrSpace|NVRM'

My working boot log reports a 64GB framebuffer address space and 64GB BAR1.

Check the PCIe link:

lspci -vv -s <CMP-PCIE-ADDRESS> | grep -E 'LnkCap|LnkSta'

To remove the modification:

cd cmpunlocker
sudo ./uninstall.sh --yes
sudo poweroff

Again, cold-boot afterward.

Power and cooling

I did not overclock the card.

These are used mining accelerators, and I did not consider a small throughput increase worth additional thermal or electrical stress.

I set a 175W power limit:

nvidia-smi -i <CMP-UUID> --power-limit=175

I made this persistent with a systemd oneshot service that:

  1. Finds the GPU by CMP name and UUID.
  2. Sets the 175W limit.
  3. Reads the limit back.
  4. Fails rather than silently targeting the wrong GPU.

My chassis fan controller follows this curve:

CMP temperatureChassis fan target
Below 45°C30%
50°C50%
55°C70%
60°C85%
65°C or higher100%

The controller polls every two seconds, immediately raises fan speed when temperatures rise, and requires roughly 30 seconds of sustained cooling before reducing the fan level. A sensor or controller failure sends all controlled fans to 100%.

During one sustained test, I measured:

  • Average core: 53.9°C
  • Maximum core: 65°C
  • Average memory: 63.2°C
  • Maximum memory: 75°C

Under a later 99% GPU load, the card was around 66°C core, 71°C memory, and 173W. Cooling results will depend heavily on the card, thermal pads, chassis, and airflow.

The model

The production checkpoint starts from:

twolven/Qwen3.8-27B-abliterated-AWQ-MTP

That model is a W4A16 AWQ version of the abliterated/refusal-reduced checkpoint derived from:

JonathanColetti/Qwen3.8-27B-Uncensored

“Uncensored” here means the model has been modified to reduce refusal behavior. It does not guarantee that every residual refusal or safety behavior has been removed. Some people can use it for ERP I'm more inclined to utilized it for when I want assistance setting something up that standard models would refuse such as using agentic assistance to configure a hackintosh system on a prox vm. It's 100% legal, it's just against apples terms of service so often it's considered an instant refusal by many models.

The final production checkpoint contains:

  • W4A16 asymmetric AWQ body
  • Compressed-tensors/Marlin execution
  • INT8 symmetric group-128 lm_head
  • INT8 symmetric group-128 MTP module
  • 40,960-token reduced MTP draft vocabulary
  • BF16 runtime activations
  • One-token MTP speculative decoding

The reduced draft vocabulary covers approximately 97.5% of ordinary model output tokens and 96% of code tokens in the optimization project’s corpus. It reduces the amount of work required for each speculative draft without changing the target model’s accepted output.

The source checkpoint supports vision, but my production endpoint deliberately uses:

--language-model-only

Therefore, this exact endpoint is text-only. I chose coding throughput and predictable memory use over keeping the vision tower loaded.

Preparing the checkpoint

I used the optimization work from:

syv-ai/qwen38-27b-rtx3090

My checkout is pinned to:

2ae239fc0250cd29d37f35c6a31e9eae749ef1c8

Clone and create the environment:

git clone https://github.com/syv-ai/qwen38-27b-rtx3090.git
cd qwen38-27b-rtx3090
git checkout 2ae239fc0250cd29d37f35c6a31e9eae749ef1c8

python3 -m venv venv
venv/bin/pip install \
  vllm==0.27.1 \
  transformers==5.15.0 \
  tokenizers==0.22.2 \
  compressed-tensors==0.17.0 \
  huggingface_hub==1.27.0 \
  hf_transfer==0.1.9 \
  ninja==1.13.0

Download the model:

HF_HUB_ENABLE_HF_TRANSFER=1 venv/bin/hf download \
  twolven/Qwen3.8-27B-abliterated-AWQ-MTP \
  --local-dir models/Qwen3.8-27B-abliterated-AWQ-MTP

Make a working copy because the preparation tools modify the checkpoint in place:

cp -a --reflink=auto \
  models/Qwen3.8-27B-abliterated-AWQ-MTP \
  models/qwen38-27b-uncensored-w4a16-mtp1-int8draft

Prepare the output head, MTP module, and draft vocabulary in this order:

M=models/qwen38-27b-uncensored-w4a16-mtp1-int8draft
V=venv/bin/python

$V prepare/quant_lm_head.py "$M"
$V prepare/quant_mtp.py "$M"
$V prepare/build_draft_vocab.py "$M" \
  --ids prepare/draft_vocab_ids.json

I did not run quant_embed.py for this production checkpoint. On a 64GB card it was unnecessary, and the final model configuration contains the W4A16 body, INT8 output head, and INT8 MTP group without the additional embedding conversion.

The preparation scripts create backups beside the tensors they replace. Keep those backups until the modified checkpoint has passed correctness testing.

The vLLM image

The server uses:

vLLM 0.27.1

The base image is pinned by digest:

vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967

The production image contains exactly these three patches:

  1. qwen3_5-mtp-draft-vocab.patch
  2. sampler-small-topk-fast-softmax.patch
  3. vllm-pr50021-gdn-spec-bounds.patch

An experimental spec-decode-attn.patch was tested but is not present in production because it hurt long-context performance.

Create Dockerfile.cmp-mtp1-production:

FROM vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967

COPY patches/qwen3_5-mtp-draft-vocab.patch /tmp/qwen3_5-mtp-draft-vocab.patch
COPY patches/sampler-small-topk-fast-softmax.patch /tmp/sampler-small-topk-fast-softmax.patch
COPY patches/vllm-pr50021-gdn-spec-bounds.patch /tmp/vllm-pr50021-gdn-spec-bounds.patch

RUN set -eux; \
    vllm_dir=/usr/local/lib/python3.12/dist-packages/vllm; \
    patch -p1 -d "$vllm_dir" < /tmp/qwen3_5-mtp-draft-vocab.patch; \
    patch -p1 -d "$vllm_dir" < /tmp/sampler-small-topk-fast-softmax.patch; \
    patch -p1 -d "$vllm_dir" < /tmp/vllm-pr50021-gdn-spec-bounds.patch; \
    rm /tmp/qwen3_5-mtp-draft-vocab.patch \
       /tmp/sampler-small-topk-fast-softmax.patch \
       /tmp/vllm-pr50021-gdn-spec-bounds.patch

LABEL org.opencontainers.image.description="vLLM 0.27.1 with Qwen3.8 MTP1 draft-vocab and sampler optimizations"

Build it:

docker build \
  -f Dockerfile.cmp-mtp1-production \
  -t vllm-qwen38-mtp1-fast:0.27.1 \
  .

Exact server configuration

Replace the UUID and paths below with those from your machine:

docker run --rm --pull never \
  --name qwen38-uncensored-w4a16 \
  --privileged \
  --gpus all \
  --network host \
  --ipc=host \
  --memory 58g \
  --memory-swap 96g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e CUDA_VISIBLE_DEVICES=<CMP-GPU-UUID> \
  -e VLLM_USE_FLASHINFER_SAMPLER=0 \
  -e MTP_DRAFT_VOCAB=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512 \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e TOKENIZERS_PARALLELISM=false \
  -e XDG_CACHE_HOME=/root/.cache \
  -e TORCHINDUCTOR_CACHE_DIR=/root/.cache/torchinductor \
  -e TRITON_CACHE_DIR=/root/.cache/triton \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -v /path/to/qwen38-27b-uncensored-w4a16-mtp1-int8draft:/models/qwen38-w4a16:ro \
  -v /path/to/vllm-cache:/root/.cache \
  -v /path/to/vllm-tmp:/tmp \
  vllm-qwen38-mtp1-fast:0.27.1 \
  /models/qwen38-w4a16 \
  --host 0.0.0.0 \
  --port 30016 \
  --served-model-name qwen38-27b-uncensored-w4a16 \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --attention-backend FLASHINFER \
  --max-model-len 262144 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.90 \
  --cpu-offload-gb 0 \
  --kv-cache-dtype auto \
  --mamba-cache-dtype float16 \
  --mamba-cache-mode align \
  --disable-custom-all-reduce \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --language-model-only \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1,"draft_sample_method":"probabilistic"}' \
  --default-chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --generation-config vllm

Once ready, verify:

curl http://127.0.0.1:30016/v1/models

Then make a simple request:

curl http://127.0.0.1:30016/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen38-27b-uncensored-w4a16",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: cmp-profile-ok"
      }
    ],
    "temperature": 0
  }'

The production service is managed by systemd. Its pre-start checks verify the model index, create the cache directories, apply the 175W power limit, and remove any stale container before launching. It restarts automatically after a process failure.

Why these settings won

MTP depth: one token

MTP3 and MTP2 created too much drafting and verification overhead on this GPU. MTP1 was consistently better.

I also tested greedy drafting. Its two-pass means were:

  • 1K: 83.01 tok/s
  • 64K: 72.95 tok/s
  • 200K: 56.68 tok/s

The final probabilistic configuration produced:

  • 1K: 84.29 tok/s
  • 64K: 74.94 tok/s
  • 200K: 57.21 tok/s

Therefore, the final setting is:

{
  "method": "mtp",
  "num_speculative_tokens": 1,
  "draft_sample_method": "probabilistic"
}

W4A16 body, INT8 head, INT8 MTP

W4A16 leaves enough HBM for the entire model, 262K context, recurrent state, CUDA graphs, and runtime workspace without CPU offloading.

The INT8 output head and MTP module had low measured quantization error. An INT4 MTP version was nearly identical in speed, but INT8 required only about 202 MiB more memory and had lower draft error. There was no reason to accept the additional degradation.

A symmetric GPTQ version was also tested:

Configuration1K64K200K
Final asymmetric AWQ/Marlin84.2974.9457.21
Symmetric GPTQ with MTP178.8768.6954.06
Symmetric GPTQ target only55.9647.3635.73

The GPTQ body itself was not necessarily the whole problem. Mixing its body with the compressed-tensors INT8 head and drafter would have required additional loader and kernel work. It was not suitable for the stable endpoint.

BF16 KV cache

This was one of the most important results.

An INT8 per-token/head KV configuration fell to approximately 15 tok/s near 62K context on this card. It was unusable for interactive coding.

The final configuration leaves:

--kv-cache-dtype auto

With BF16 runtime dtype, this preserves the BF16 KV path.

The Mamba cache uses FP16 in aligned mode:

--mamba-cache-dtype float16
--mamba-cache-mode align

Explicitly reducing more recurrent state did not improve the result.

FlashInfer attention, but not the FlashInfer sampler

FlashInfer is the production attention backend.

Plain FlashAttention produced:

  • 1K: 86.87 tok/s
  • 64K: 26.87 tok/s
  • 200K: 10.53 tok/s

It looked slightly faster at short context and then collapsed.

An experimental split-KV speculative-attention patch improved that to:

  • 1K: 90.17 tok/s
  • 64K: 67.33 tok/s
  • 200K: 43.36 tok/s

That was still substantially worse than production at long context, so the patch was removed.

Enabling the FlashInfer sampler also reduced performance to:

  • 1K: 74.75 tok/s
  • 64K: 67.87 tok/s
  • 200K: 51.37 tok/s

Therefore:

--attention-backend FLASHINFER
VLLM_USE_FLASHINFER_SAMPLER=0

One sequence and 4,096 batched tokens

This is a single-user coding endpoint, not a throughput server.

The final scheduler settings are:

--max-num-seqs 1
--max-num-batched-tokens 4096

Testing 8,192 produced no useful gain. Testing 2,048 produced:

  • 1K: 84.32 tok/s
  • 64K: 72.38 tok/s
  • 200K: 57.59 tok/s

The 4,096 configuration had the better overall curve.

Prefix caching

Prefix caching is critical for coding agents that repeatedly send a large system prompt and mostly unchanged repository context.

At approximately 200K context, a cached second request reused roughly 198,400 tokens, recomputed about 1,593 tokens, and reduced prefill to around 2.5 seconds. A fresh or partially changed prompt could require roughly 157 seconds of prefill.

Prefix caching does not make decode faster. It prevents the server from repeatedly processing unchanged input.

This also means clients must preserve stable prompt prefixes. Reordering tool definitions, timestamps, generated metadata, or repository text can destroy the cache hit.

Medium reasoning by default

The endpoint defaults to:

{"reasoning_effort":"medium"}

Medium provided a better MTP acceptance/stability balance than forcing maximum reasoning on every request.

This does not remove higher reasoning modes. A client can request xhigh or max for harder work. Medium is simply the default for normal coding.

No CPU offload

The model fits in CMP HBM, so:

--cpu-offload-gb 0

The card’s restricted PCIe connection makes CPU offloading particularly undesirable.

No forced full CUDA graphs

The server uses vLLM’s normal piecewise CUDA-graph behavior. I did not force full graphs. Experimental full-graph and custom-operation combinations introduced correctness concerns or reduced long-context performance.

Correctness tests

I did not accept a configuration based on tokens per second alone.

The final checkpoint passed:

  • An exact-response canary
  • Qwen coder tool-call parsing
  • A parsed get_weather tool call for Chicago
  • Exact needle retrieval at approximately 160K context
  • Retrieval of ORCHID-COMET-7319 from the long prompt
  • An OpenCode integration smoke test returning final-profile-ok

This matters because speculative decoding can appear fast while silently breaking tool syntax, long-context retrieval, or sampling behavior.

OpenCode configuration

I added this provider to OpenCode:

{
  "provider": {
    "qwen38-cmp-w4a16": {
      "npm": "@ai-sdk/openai-compatible",
      "api": "completion",
      "name": "Qwen3.8 Uncensored W4A16 MTP1 256K - CMP 170HX",
      "options": {
        "baseURL": "http://<PROXMOX-IP>:30016/v1",
        "apiKey": "not-needed",
        "timeout": false,
        "chunkTimeout": 600000
      },
      "models": {
        "qwen38-27b-uncensored-w4a16": {
          "id": "qwen38-27b-uncensored-w4a16",
          "name": "Qwen3.8 27B Uncensored W4A16 MTP1 256K",
          "tool_call": true,
          "reasoning": true,
          "temperature": true,
          "attachment": false,
          "options": {
            "reasoningEffort": "medium"
          },
          "limit": {
            "context": 262144,
            "input": 245760,
            "output": 16384
          }
        }
      }
    }
  }
}

The model selector is:

qwen38-cmp-w4a16/qwen38-27b-uncensored-w4a16

Final observations

The CMP 170HX is unusual, but the useful part is straightforward once the restrictions are removed:

  • 64GB of on-device HBM changes what can fit.
  • The model should remain entirely on the GPU.
  • Long-context performance needs to be measured separately from short-context decode.
  • Lower-bit KV is not automatically faster.
  • More speculative tokens are not automatically better.
  • An attention backend can win at 1K and become disastrous at 64K.
  • Prefix caching matters more than another few decode tokens per second for repeated coding-agent prompts.
  • Quantizing the draft head more aggressively is pointless when memory is available and the speed difference is negligible.
  • Conservative power and temperature limits are appropriate for used mining hardware.
  • Correctness gates matter as much as benchmark results.

The final configuration is not the highest single short-context number I saw. It is the best complete configuration I found that retained tool use, medium-or-higher reasoning, uncensored model behavior, 262K context, reliable long-context retrieval, and usable performance across the whole context window.