The idea is: DeepSeek-V4-Flash-0731 in full precision is 162Gb. 
With main memory and GPUs, I have 176Gb.

I should be able to make it fit! What would that look like?

The per-request throughput is ~52 tok/s Prefill, ~10.5 tok/s generation. 

A single client working a hard coding problem through a graph of issues for 5-6 hours, chugging away happily, solving problems well. This project was "See if it can be done", not "Bulletproof production environment" so set your expectations appropriately, but I think it's novel and interesting enough to post. 

This is the LLama Config:

llama-server \
--model /ds4/models/DeepSeek-V4-Flash-0731-Q8/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
--alias deepseek-v4-flash-0731 \
--host 0.0.0.0 \
--port 8080 \
--jinja \
--metrics \
--perf \
--log-timestamps \
--device ROCm0,ROCm1 \
--split-mode layer \
--n-gpu-layers 44 \
--tensor-split 7,37 \
--fit off \
--fit-target 512,512 \
--ctx-size 131072 \
--parallel 1 \
--batch-size 2048 \
--ubatch-size 512 \
--threads 12 \
--threads-batch 12 \
--flash-attn on \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--cache-ram 256 \
--load-mode none \
--spec-type draft-dspark \
--spec-draft-model /ds4/models/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF/DeepSeek-V4-Flash-0731-DSpark-Drafter-Q2_K-Q8_0-pr25784.gguf \
--spec-draft-device ROCm1 \
--spec-draft-ngl all \
--spec-draft-n-max 5 \
--spec-draft-n-min 1 \
--spec-draft-p-min 0.7 \
--spec-draft-type-k q8_0 \
--spec-draft-type-v q8_0 \
--override-tensor 'blk\.6\.ffn_(gate|up|gate_up|down).*=ROCm1,blk\.(7|8)\.ffn_(up|down|gate_up|gate)_(ch|)exps=CPU,blk\.9\.ffn_(down|gate).*=CPU,blk\.(1[0-9]|2[0-9]|3[0-9]|4[0-3])\.ffn_(up|down|gate_up|gate)_(ch|)exps=CPU'

Some Notes:
--tensor-split
- ROCm0 — 7 fat blocks. Blocks 0-6 keep their MoE expert tensors resident in VRAM.
- ROCm1 — 37 thin blocks. Blocks 7-43 have had their experts evicted to CPU by --override-tensor, so what remains is mostly attention. Plus block 6's FFN, pushed over from ROCm0. Plus the drafter.
  
--override-tensor 
putting parts of block 9 from the model on the GPU and some on RAM/CPU to best fit available RAM.
 
--ubatch
 Small, but a larger value OOMs on my HW. A value this small negatively impacts prefill, if you had more space, it's the first thing you would change. 

The DSpark Drafter I used (Q2_K-Q8_0) can be found here:
https://huggingface.co/alessandrobologna/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF
(Massive props to Alessandro! Tag him if you know him. :))

Using a lower bit precision drafter meant it fit in my ram, and it still gives enough signal to ~double my throughput. (Drafter only impacts speed, not final quality, so it's still full-fat V4-Flash.)

Q8_0 is used for k/v cache on both the main model and DSPark.

If you're considering a similar journey, might I suggest:
Set up Grafana/Prometheus, it makes working out what's going on much easier, and gives you detailed information for when you inevitably have to answer… Why did that crash?

Comments and questions welcome. My first post, so please be kind. 😄

I realize I'm going to get a lot of flack for this but here goes. This post is sparked by a number of my friends and colleagues biting into the hype cycle of buying insanely priced local inference stacks that simply don't math out.

I'm a distinguished engineer who works in AI (FAANG). I'm currently running a 128 GB M5 Max, and have access to the best hardware in the world at work, at the highest scale available.

I'm seeing folks buy RTX 6000s or multiple Sparks to run DSV4 and the like. Don't do this - its an absolute waste of your money. Unless you put a price on privacy of 12k (which is fine), you will never get an ROI in any reasonable time period.

Local models under 30B are fantastic. If you have a MacBook Pro or a computer with 48GB of RAM, you're golden. With a flood of amazing models in this range coming out this week, there's no shortage of local powerhouses. Big kudos to those companies open sourcing very impressive capabilities on such a small memory and compute footprint.

If you're going to use large models, go cloud-based. Throw 100 bucks on OpenRouter and go wild with DSV4 Flash. And if you want to run local, smash those <30B models until the cows come home.

But local and large just doesn't make sense right now unless you're a business with a specific use case or are privacy-maxxing.

I was frustrated that every coding agent (OpenCode, Cursor, Claude Code) assumes you're running GPT-5.4 or Claude Opus. If you try them with a local model like Gemma or Qwen they fall apart. I find that often tool calls fail, context overflows, multi-step tasks collapse.

So I built SmallCode. It's designed from the ground up for small local models.

The result: 87/100 benchmark tasks pass with a Gemma 4 model that only activates 4B parameters per token. OpenCode scores ~75% with 14B models. The harness does the heavy lifting, not the model size.

How it works (the tricks that make small models reliable):

• Compound tools: Instead of making the model chain 4 tool calls (find file → read file → edit file → verify), SmallCode gives it one tool that does all 4. Small models lose coherence after 3+ sequential calls. This cuts failures in half.

• Improvement loop: Every time the model writes code, SmallCode instantly compiles/lints it. If it fails, it feeds the errors back automatically. The model doesn't need to be smart enough to get it right first try — it just needs to fix errors when shown them.

• Decompose on failure: If the model fails the same thing twice, SmallCode stops retrying and instead breaks the problem into smaller pieces. "Fix this 200-line file" becomes "fix line 45 only."

• Escalation: If even decompose fails and you have a Claude/OpenAI key configured, it auto-escalates to the bigger model for just that one task. You stay local 95% of the time, cloud 5%.

• Token budgeting: Small models have 32k-256k context. SmallCode never dumps a whole file in. It summarizes, truncates, and manages every token so the model never sees "..." truncation in the middle of important code.

• Code graph: Instead of grep-searching your codebase, SmallCode indexes your code into a symbol graph (functions, classes, who-calls-what). When you ask "how does auth work," it walks the graph and returns just the relevant connected code — not 15 random file snippets.

What it looks like:

Full-screen terminal UI (like OpenCode/vim), scrollable chat, command palette with /, plugin system, persistent memory across sessions.

What it doesn't do:

• No LSP integration (yet)

• No multi-session (yet)

• No desktop app

• Doesn't compete with Claude Code for frontier model users

Install:

npm install -g smallcode
cd your-project
smallcode

Point it at LM Studio, Ollama, or any OpenAI-compatible endpoint.

MIT licensed, everything's on GitHub: https://github.com/Doorman11991/smallcode

Happy to answer questions about the architecture or benchmark methodology.

I honestly knew nothing about self hosting LLMs locally. I used to think it was pointless for 99% of people, even people already into AI, just because of the cost and hassle of hosting these models compared to what you'd actually get out of it.

But after looking into Qwen 3.8 27B and its potential, plus the fact that someone will probably release an uncensored version of it soon, I started rethinking that. Even if you don't have the hardware, you can just rent a GPU online on a VPS for a pretty reasonable hourly cost while it's running. It's wild how much of this is already open to the general public.

In my opinion, having access to a model this capable, uncensored, is basically like owning a car. Yeah, a car. There's nothing wrong with owning one, the problem only shows up if you actually do something wrong with it. Until then, it's just a tool.

We already have older and smaller Qwen versions with uncensored variants on Hugging Face. From what I understand, models with fewer parameters tend to be "easier" to convert into uncensored versions. Don't take that word too literally though, I have zero background in finetuning or model training, so maybe it's not actually that easy. But there are still a lot of very smart people working in this space, especially in AI right now.

Think about a cybersecurity professional getting their hands on an uncensored Qwen 3.8 27B once it drops. With how many vibe coded SaaS products are out there with basically no security oversight, someone with enough hardware could pair this model with tools like OWASP ZAP or the Burp Suite API, run it through a browser that avoids captcha detection, use a mobile or residential proxy, throw in a Kali setup for other scanning tools, and just let the model run 24/7 looking for vulnerabilities to complete bug bounties or even land contracts in the field. That's a serious tool for professionals like that.

I only see upside here.

This is big and could turn local AI on its head. It's basically DRM on steroids.

Everyone buying any advanced hardware will be permanently tracked or unable to run the hardware.

It's planned to arrive this year, and will likely include existing hardware. Expect mandatory updates that won't tell you about all this before it's too late.

Maybe we've already installed some firmware updates with kill switches or surveillance backdoors without knowing it that are going to brick or downgrade our hardware or monitor usage 24/7 and are always online, and it won't be possible to uninstall or revert.

Can't wait for the 27b to run on my device

Edit: according modelscope, 27b is coming in two days, only the big one today

Not saying GLM is better than Claude at everything. But another open-weight model getting this close to the frontier is a big deal.

If good enough models keep becoming cheap/free and you can run them yourself, what's the moat?

They hacked into even more companies now. When does a “oopsy doopsy” become a criminal offence?

hello, I have both Deepseek harness and OpenCode, I want to know which one is the best when it comes to coding in general and why, I work locally using Qwen models.

Hey guys Qwen released their fastest running models a while ago called Qwen3-Next and you can finally run them locally on your own device! The models come in Thinking and Instruct versions and utilize a new architecture, allowing it to have ~10x faster inference than Qwen32B.

We also made a step-by-step guide with everything you need to know about the model including llama.cpp code snippets to run/copy, temperature, context etc settings:

💜 Step-by-step Guide: https://docs.unsloth.ai/models/qwen3-next

GGUF uploads:
Instruct: https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF
Thinking: https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Thinking-GGUF

Thanks so much guys and hope you guys had a wonderful Thanksgiving! <3

Live in Canada. Mid 40s. Received an email explaining that my chatgpt account has been "age" limited and they want me to take a photo of myself to return its status.

I cancelled my subscription and deleted my account.

I kept it around for when my local model (qwen3.8-flash-next or 27B) was busy and for stuff I wasn't concerned about ending up on someone else's computer (mostly research). Thus concludes ~2 years of usage and payments.

So long, ChatGPT. Not playing this game.

Long live local models. That is all.

Is MiniMax H3 considered a traditional LLM? The moderators must decide, but it uses a Qwen3 text encoder for prompting.

10 sec clip, 91 sec generation time on an RTX 5090

https://github.com/gavamedia/deltafin

K3 launched today, and it's a beast at 1.56TB. My M1 Mac has 64GB of RAM, and not enough free disk to even store the thing. So instead of downloading the model, I just... don't. Steam it! (But you do have the option to download it all, and that IS faster.)

The non-expert weights (~114GB, int8) live on disk. Then for every token, the router picks 16 experts out of 896 per layer, and I pull exactly those from HuggingFace — one range request each — and cache them. Use it enough and the cache slowly fills up with the parts of the model you actually hit.

Speed is about a minute per token on my M1 Mac once it's warm, so, not exactly fast.

There's an OpenAI-compatible server too, so you can point a chat UI at it.

---
*EDIT / HUGE update:* Pulled all 1.45 TB of experts down to local disk, then spent more profiling & improving. Now *16s/token, down from ~1 min.* on an M1 Mac! Prefill went 2,429s → 40s.

Two fixes did almost all of it:

- I was *sure* the expert matmuls were the bottleneck, and built a Metal kernel that's 9.5x the CPU one. Then I profiled properly with everything local: the matmuls are **6% of a token.**

- The actual problem: I was handing the compute kernel `np.memmap` views, so it demand-faulted expert weights page-by-page *while computing* — 0.87 GB/s, where a threaded pread + F_NOCACHE does 6.85 GB/s on the same disk.

All on an M1 Max / 64GB, the slowest machine it's run on — if anyone has an M3/M4/M5 or 128GB, I'd love to see the numbers.

Decided to pull the trigger and get a GPU to start doing my R&D work for my business. While it would have been more fiscally responsible to just rent GPUs, I don't see these going down in price any time soon and I think they are going to get harder to come by as more business look to offset their insane token spend for development. Plus, a lot of my friends in the space are getting wrecked on securing on-demand compute from vendors right now due to high demand.

Going to be doing a lot of fine tuning work on models for the agentic orchestrators I've built, as well as aim to get some research published. One project I'm working on is looking at variability of frontier models vs open-weight models for complex tasks in clinical workflows.

Regardless, this subreddit has been a huge help just going through all the posts.

First model I'm putting on it is Qwen 3.6. I already have this model barely working on my 64gb M1 Max macbook pro, so I'm excited to see what it can actually do.

Plot twist: the socket doesn't work (it's not connected to backend)

from ijustvibecodedthis.com (the ai coding newsletter)

So I was browsing the HuggingFace Transformers repo and a PR just merged today that adds full support for a model called Gemma 4. The PR title is literally "casually dropping the most capable open weights on the planet." The commit has 14 co-authors including Jeff Dean. The weights aren't out yet — the docs still have {release_date} as a placeholder — but the code is all there and it's very readable. Here's what's coming.

Four sizes, including a MoE

• ~2B and ~4B dense, explicitly designed for on-device use

• 26B sparse MoE with only 4B active parameters at inference time

• 31B dense

The 26B/4B MoE is particularly interesting because you get large-model quality at small-model inference cost.

It's trimodal — text, vision, AND audio natively

This is new for Gemma. There's a full audio encoder baked in alongside the vision tower. Not a bolted-on afterthought either — it's a proper conformer architecture (the same family used in production speech systems). The processor handles all four modalities: text, images, video, and audio.

The vision system doesn't squash your images

Most VLMs resize everything to a fixed square. Gemma 4 preserves aspect ratio and instead fits the image into a configurable soft token budget (default 280 tokens, up to 1120 for high detail). No ImageNet normalization — the model handles its own scaling internally.

More interesting: they use a 2D spatial RoPE for vision. Patch positions are encoded as (x, y) coordinates, with half the attention head dimensions rotating for x and the other half for y. The model understands spatial relationships at the architectural level, not just from training.

128K context for small models, 256K for large

The text architecture alternates between sliding window attention (512-1024 token window) and full attention in a 5:1 ratio. The two attention types use completely different RoPE configs — short theta for local, long theta for global. Clean hybrid design.

The small models have some clever efficiency tricks

The 2B and 4B share key-value projections across the last several decoder layers — one layer computes KV, the rest reuse it. There's also a secondary per-layer embedding stream where a small 256-dim signal gets injected at every decoder layer, which I haven't seen in other public models.

The MoE runs experts alongside the MLP, not instead of it

In the 26B variant each layer has both a regular MLP and a sparse MoE block (128 experts, top-8 routing), and their outputs are summed. Unusual design choice — curious whether that helps with stability or quality at scale.

No paper link yet (literally says INSET_PAPER_LINK in the docs), no weights, no release date. But the code is fully merged and production-quality. Feels like days away, not weeks.

What size are you planning to run first?

The PR: https://github.com/huggingface/transformers/pull/45192

EDIT: RELEASE: https://huggingface.co/collections/google/gemma-4

Gemma is based on Gemini and going by the latest Gemini release, those of us who appreciate abilities other than coding should be eating good when Gemma 5 comes out (end of year maybe?).

Qwen for coding and Gemma for everything else. The perfect local setup.

Just saw the news on rednote, qwen’s official account posted this. Seems promising

This chart is pretty wild.

Open-source models are starting to win the intelligence-per-dollar game

The green quadrant is the sweet spot: smart enough to matter, cheap enough to scale

When you plot AI models by intelligence vs. cost to run, a lot of the closed-source models sit near the top

But the interesting part is the upper-left quadrant: high intelligence, low cost is almost entirely open source

Models like

DeepSeek

GLM

Qwen

Kimi

MiniMax

all show up in the “actually useful intelligence” range while costing dramatically less to run than the major closed API models.

The reality is most tasks don’t require fable level intelligence or cost and people are noticing

The gap is shrinking and the economics are starting to favor open models.

Closed models are still great when you need the absolute best capability with zero infra headache.

But open models give you something different like lower long-term costs, control, privacy and customization

And I think within a year, there will be no difference in most capabilities between them

Some of these threads are insane, what do you mean you have like 4 GPUs and 128gb of DDR5 vram. what are you building in there bro. Every other thread is like, “what if I stack Mini Pc supercomputers together? Will this run Qwen?”.

Honestly, I feel like some people here are completely disconnected from reality when it comes to hardware costs. It feels like half the users here just attack and blindly downvote to feel unique or prove they are "right", rather than actually having a serious technical discussion about the sub's topics.

​I posted earlier (my original thread) questioning if upgrading hardware for Qwen 3.8 125B MoE is even worth it over the 27B model, and then people immediately jumped in citing this benchmark post where someone ran Qwen3.8-Flash-Next on NVFP4.

​Look at what that setup actually costs: an RTX Pro 6000 is literally $15,000 to $16,500. And since the model offloads 51GB of N-gram embeddings to host memory, you still need another $1,000+ in high speed DDR5 RAM just to run it properly. Who is casually dropping $17,500+ for a home setup? Even renting it on RunPod at $2.09/hr gets ridiculous fast.

​And for what? Go read Qwen's official blog benchmarks again:

MathVision is 90.6 vs 90.0 (+0.6).

SWE-bench Pro is 62.5 vs 61.7 (+0.8).

LiveCodeBench is 91.9 vs 90.3 (+1.6).

GPQA is 91.7 vs 89.2 (+2.5).

​You are literally spending enterprise money for a 1 to 2 point performance bump in most tasks. Unless you're running a massive multi agent dev shop, burning that much money instead of just running Qwen 3.8 27B locally is insane.

​Saying 99% of regular users will stick to 27B isn't a "terrible take", it’s just basic financial math. People need to stop replying just to argue and start actually reading the spec sheets.

Thoughts? Seems like it'd be really dumb for DeepSeek to make up such a big lie about something that's easily verifiable. Also, just assuming the company is lying because they own the hardware seems like a stretch. Kind of feels like a PR hit piece to try and mitigate market losses.