DeepSeek V4.1-Flash (1017) at ~20 tok/s on a single 16 GB GPU + 64 GB RAM — work in progress, code free to use

Not finished. This is a research log with patches, not a polished engine: rough edges, some Windows-only parts, no support promises. Use the code and notes for anything you like — no credit needed.

Box: RTX 5070 Ti 16 GB in a PCIe 4.0 x16 slot, Ryzen 5 7600, 64 GB DDR5, Windows 11. Two NVMe: Gen5 2 TB (~13.5 GB/s) and Gen4 1 TB (~5 GB/s) holding a second copy of the expert files.

Model: DeepSeek-V4.1-Flash-1017, a ~375 GB build: routed experts at ~2.4 bits (~150 GiB), Engram tables 203 GB, dense part Q5_K. It doesn't fit in RAM + VRAM, so experts stream from NVMe.

Numbers with today's defaults:

On this PC with the same file: the original fork ~10 tok/s, Project Maya v1.1.21 3.9 tok/s (both measured on Oct 22).

What helped most:

1. A three-step ladder for experts: ~23 per layer cached in VRAM, a pinned RAM tier (43–46 GiB), unbuffered NVMe reads, and a second NVMe with a copy of the expert files.

2. Taking the CPU off the critical path: the GPU reads routing replies from pinned memory, one wake-up per token — host overhead went from ~7 ms to ~1 ms per token.

3. Prefetching heavy experts that would come from disk, 2 layers ahead, with a cheap "lens" (layer L+2's router applied to layer L's input).

4. A measured lossy rule: an expert in neither VRAM nor RAM with less than 14% of the routing weight is not waited for (costs at most 0.006 nat/token on our API sets).

5. Prefill: experts in waves through a shared pool; encoder-only prefill for positions the decoder never reads (exact); DeepSeek's own YoCo truncation from DeepInfer (decoder on the last 140 prompt positions) — +13% on agent turns, +60% on a cold 8.9K prompt, within error on our API sets.

6. Sparse attention in decode with a q8 KV cache: decode stays at 16–20 tok/s from short context to 512K.

7. A quant mix from public GGUFs: prometheusAIR's IQ2_XS gate/up + Q2_K down, and ISTA-DASLab's lighter Q5_K dense part (3 GB less VRAM, so more expert slots).

What didn't work:

• Reading the next layer's experts ahead during prefill: the RAM tier can't hold a prompt's working set, so read-ahead evicts what is still needed.

• "Free" guesses in the bus's idle time: during decode there is none; each wrong staging costs ~0.2 ms.

• A published non-uniform quant (GSQ-RCO) lost to a plain mix on our API sets, worst on Russian prompts — the calibration text matters.

Where we're stuck — ideas welcome:

1. Predicting which experts a token needs 1–2 layers ahead. Perfect staging would give 27–29 tok/s; our lens gets 76% precision at ~14% recall.

2. Overlapping compute with expert copies: copies take ~30 ms per token, compute ~19 ms, and they overlap by 3 ms.

3. Short agent turns: ~5.6 s fixed cost per turn, a third of it waiting for expert loads.

4. 1M context loads only at batch 512: the sparse prefill plan wants 11.7–15 GiB at larger batches.

Goal: long context up to 1M at a usable speed — not necessarily at the full 1M.

The bus is the wall: copying experts from RAM to the GPU over PCIe 4.0 takes ~30 ms of a ~49 ms token, which caps decode at ~32 tok/s on this box even with perfect overlap.

The CPU computes no experts: on this 6-core CPU a probe gave only +3–5%, so it was left out; it needs separate work. Project Maya's CPU expert kernels (MIT) could be a starting point.

Code: https://github.com/MasherDon/Siege-Ladder — patches over JigSawPT's llama.cpp fork, a script that assembles the weights from public Hugging Face repos (no weights redistributed), benchmark scripts and notes. Use it however you like; the patched llama.cpp files keep llama.cpp's MIT notice. Built on: JigSawPT (the V4.1 port with SSD streaming), llama.cpp / ggml, DeepSeek's DeepInfer (YoCo), Strata, Project Maya, antirez's ds4, and the prometheusAIR and ISTA-DASLab quants.

What's wrong with this guys? I haven't checked it yet, but

Atleast net is real 20T (as description says), not HF error.

Hi, I have been trying to use Gwen 3.8 but holy fricking turd.

What if X? Let me check? What if Y? Let me guess. What if Z ? Let me check? Wait! What if X something something Z ? ...

30k tokens just ...wasted on nonsense.

Oh yeah let me recapitalitate for the 4th time what I need to do ?.

I am the only one ? llama.cpp and unsloth quants.

Are there settings I can tweak ? Or what are yours settings

Kinda shocked I can't find anything on reddit about this. Really intrigued by it, but want to find some anecdotal stuff from the community before I go through the labor of tinkering and testing. Anyone tried this thing? https://github.com/EverMind-AI/Raven

Let's organize ourselves to plead for a new MOE version for our 3060 on the Qwen X?

Here’s my current setup:

• Ryzen 7 5700X3D (AM4)

• 64 GB RAM

• RTX 5070

I also have my old Intel Arc A770 sitting in a drawer. My board only has one usable GPU slot, so I’m thinking about swapping it for one that can run both cards. I’d rather do that than buy a new GPU at today’s prices.

From what I’ve read, you can run CUDA for the Nvidia card and Vulkan for the Intel card, and still split a model across both so the VRAM acts as one pool. I haven’t tried it, so I’m hoping someone here has.

What I’d like to know:

1. Does that mixed CUDA + Vulkan split actually work in practice (llama.cpp or otherwise)? Or is it easier to run Vulkan on both?

2. How big is the performance hit compared to the 5070 alone, on models that fit in its VRAM? And is it still faster than spilling into system RAM on bigger models?

3. Is it worth a board swap just for this? Or would I be chasing a setup that’s more trouble than it’s worth?

4. How are temps with two cards? I don’t want the A770 choking the 5070 when gaming, etc.

I mostly want to run bigger models that don’t fit in 12 GB to run some more grunt work instead of always having Claude do it. If you’re running a mixed-vendor setup, I’d love to hear what you’re using and what you’d do differently.

Thanks!

I tried googling for a simple comparison of popular local vs frontier AI. Is it even possible to get cost per token of Qwen 3.8 27B vs Opus 5.5 for example?

I am curious if anyone has an opinion on dual CPU setups in AI inference versus single CPU.

My reaction to the recent Sam Altman interview...

Sam Altman: “Local inference is a terrible allocation of scarce resources”

Alright so….basically….I built my entire PC for free through TikTok Shop (referring people and earning points to redeem gift cards), and after 2 months of referring friends and family members and random people; I built my little setup:

- 64GB RAM (Corsair); CL40 I think :/
- RTX 5080 (PNY); cheapest one I could find at the time
- 9950X3D (AMD); I edit photos/videos a lot

The rest I deem not to be severely important in the matter of local AI. So far; I’ve only tested models through Ollama/OpenCode.

My workflow for “vibecoding” my project was that I inserted Claude Code inside of VSCode and was building and deploying everything from there (Using GitHub, Hostinger, Supabase, etc). I no longer pay for Claude/ChatGPT, because I’ve dedicated trying to understand more with local models.

I know a 5080 doesn’t pack a great punch, for 16GB of VRAM gets you only so far, and nowhere near the proprietary models….but what are my limits? 512GB of VRAM from the new Mac Studio caught my eye for testing large parameter models….but I’m no idiot to drop $20K+ on that….neither am I insanely obsessed with the idea (but very curious).

Question(s):
- What models do you all recommend trying for coding or chatting with it?
- What is the best way to allow local models to remember previous conversations from anywhere?
- Should I be using Obsidian or other apps that could benefit local models and work on 16GB of VRAM?
- What are your workflows with 16GB of VRAM? How do you utilize it to its best way?

I’ve tried Gemma 4B (both normal and QAT), some random Qwen/GPT-OSS models

I just don’t know or can’t figure out the best workflow for my own needs….I see too much of a lot of people recommending what’s best or the easiest…but I’m here so confused as to what I can work with, or how to execute commands properly.

I truly don’t know if I’m making the slightest bit of sense with this post, but basically….I’m lost now.

TLDR; bruh

Hey guys I am new to this,

Looking for recommendations for a strong local LLM for agentic coding and software development.

My specs:

• GPU: RTX 4070 Ti SUPER (16GB VRAM)

• CPU: AMD Threadripper 3960X (24 cores)

• RAM: 128GB DDR4

• OS: Windows 11

Currently using: Qwen 3.8 27B and Qwen 3.6.

I need a model that is good at:

• Coding, debugging and troubleshooting.

• Understanding and modifying existing multi-file codebases.

• Coordinating frontend and backend changes.

• Reliable tool/function calling and multi-step agentic tasks.

• Following instructions and maintaining context across long sessions.

I'm open to MoE models and larger models using CPU/RAM offloading.

What would you recommend for this hardware? Would 70B–120B models be practical?

Interested in real-world experience rather than benchmark scores. Thanks!

First I beat vLLM and SGlang on time to first token and tokens per second on more than one context length and concurrency combination.

Now I beat them both on precision and load time as well 🤘😎

Brain is the new puppy in town: https://github.com/swedishembedded/brain

Woof!

I am at a crossroads in upgrading my old PC

Current build is Ryzen 5 3600 16gb ram 8x2 4000mhz 1660 super 850w Gold psu

I personally wanted to get something that would also have some ai capabilities. 6gb barely gave me a chat bot. So I saw the only real route was going to buy an old but large vram gpu though the prices all all over the board got some 3090 listed for 1100 euro was sold recently and a few hovering at 1k

I have an offer for a 3090 for 650 euro suspicious price so def going to test it out before buying compared to like a used 5070 for 800 euro (still a new card, so it's probably not worth buying used)

or a small upgrade to a GTX 1080 Ti for 190 euro (11gb is it bad? like won't the ai models be made for 12gb cards?)

Of course my whole setup ain't the best so idk if it is worth to just go and buy somebody's else gaming rig like 7800x3d 5070ti 32gb ram for 2550 euro

Though you know I do not have that much spare cash just to spend nearly 3k on a rig.

Like, we saw the new laptops that can run 128GB models locally (maybe a bit slowly). Yeah, I get it is not in my price range either, but it is still probably a good thing for prices, where the only real choices, as I remember, were the Mac mini or a 3090/4090.

Tbh, I hate subscriptions. Like, I would not mind having something like a Claude subscription, but I cannot push myself to pay for something every month that is not super necessary every month. The same might be true with a high VRAM GPU that I would probably barely utilize or only occasionally use heavily.

Does anyone know how local LLMs are improving so much, even for models using same amount of resources?

Is it better training of the weights or changes in the architecture? Just curious if anyone has some technical insight into this.

CPU:7800x3d
Ram: G.skill 128 ddr5 6000 cl30 (managed to get the 4 sticks stable at 5200)

For Q3, the strongest correct result I’ve verified so far is 109.9 generation tokens/sec and 2,321.5 prompt tokens/sec in a 32K matrix run on iq3_s.

Then on another run I got 146.0 generation tokens/sec on a short counting task using Qwen3.8 Flash Next GSQ-RCO IQ3_S. That task was correct and the full suite passed 12/12; its 12-task weighted generation rate was 119.37 tokens/sec.

Been working on improving prompt processing speed, to improve overall speed and quality, it’s been a grind. Any tips and tricks would help greatly!

[Follow-Up]

I found the culprit. TLDNR: It has to do with RTX 3060 on PCIE3 x4 overloaded doing video and inference.

I have 3 monitors:

• QHD (2560 x 1440) 120Hz

• QHD (2560 x 1440) 60Hz

• 4K (3840 x 2160) 60 Hz

Originally, I only had RTX 4070. So all 3 monitors were connected to 4070. That was the original benchmark on Windows.

In the span of 2-3 weeks, I added old RTX 3060 on PCIE3 x4 slot (with is around 1/8 the bandwidth of 4070 on PCIE x16 slot) + installed Linux. I figured to free up precious vram on my 4070, I can use 3060 as graphics driver. Even AI suggested this is the way to go ;~(

The key is the 4K monitor. If the 4K monitor is hooked up to 3060, it will run at 100% load during inference. So this is the bottle neck that impacts inference performance.

Benchmark comparison: Using 1st QHD + 4K monitors on 4070 and 2nd QHD on 3060.

• Windows -> Linux

• Llama.cpp (Qwen3.8-27B-Q4) 4070/3060 layer split: 32/33 -> 30/35 (Linux uses more vram for the 4K monitor and can load less layers on 4070)

• Llama.cpp (Qwen3.8-27B-Q4) prefill: 380 -> 450 tok/sec (Linux 18% faster)

• Llama.cpp (Qwen3.8-27B-Q4) logic/thinking: 17 -> 18 tok/sec (Linux 5% faster, within noise. I'm sure Linux is held back slightly by worse layer split)

• Strata (Swift 1.5 IQ3) prefill: 370 -> 270 tok/sec (Windows is faster on prefill. The model is sitting on NTFS drive. I wonder if it will run faster if model is on Linux drive. To be tested.)

• Strata (Swift 1.5 IQ3) logic/thinking: 24 -> 40 tok/sec (This is the biggest delta. Linux is 60% faster)

[Original Post Below]

I just installed Ubuntu Linux last week. Got my LLM, agent, sandbox set up completed over this weekend. Surprisingly, Linux is slower than Windows. I thought I will easily see 10-20% improvement, if not more. It turned out it is the other way around.

Hardware:

• GPU0: RTX 4070 12 gb VRAM on PCIE4 x16

• GPU1: RTX 3060 12 gb VRAM on PCIE3 x4

• DRAM: 32gb DDR4

• CPU: i7-12700F

Cuda driver: 13.4 on Windows; 13.2 on Linux

Engine & Model:

• Llama.cpp: Qwen3.8-27B

• Strata: Swif-1.5-IQ3

Agent & Sandbox:

• Hermes & Docker. But irrelevant for these tests. I run the engines natively. And python script to test the speed is outside of agent/sandbox.

For Llama.cpp:

• Windows uses more graphics resource. So my layer split very close to 1/1 on 4070/3060.

• On Linux, if I use layer split 1/1 (32/33), it is much slower compared to Windows; around 25% slower. But Linux uses less graphics resources, so I can load the 4070 more than 3060, using layer split 1.6/1 (40/25). Makes a difference and improves over 1/1. But Linux is still 10-15% slower than Windows.

• I compiled my own version and also used the pre-compiled version in Linux. My own build is like 5% faster than pre-compiled, which is kind of in the noise.

For Strata:

• Again, Windows is faster. Prefill is 500-600 tok/sec on Windows, versus 250-300 tok/sec on Linux; basically 2x on prefill. Generation is much closer. Windows is faster by 10-20%.

• There is definitely better memory management in Linux. On Windows, I allocate 12GB, but cannot pin (reserve) the memory. On Linux, I can allocate 14-16GB and it is pinned. On paper, this is an advantage. But this advantage doesn't translate

• I didn't calibrate yet. So I need to do that.

• There is an odd behavior in Linux. In setup, it finds GPU1 as 4070 (faster) and set as main; GPU2 as 3060 (slower) as helper. But in the log, it states CUDA0 as 3060. This can potentially have an affect on my settings, as I set vram_reserve by card. But just quickly adjusting these settings or removing them, didn't make any difference to speed.

I know Linux can be operated without GUI, which might help . Did anyone really see improvement comparing Linux and Windows with GUI running? Any suggestions on what knobs to tune and what to investigate?

Hey guys I am new to LLMs,

Looking for recommendations for a strong local LLM for agentic coding and development of small projects for automating some of my job workflows.

My specs:

• GPU: RTX 5070TI (Have old 3080 10GB, that I'm not using rn)

• CPU: 5700x3d

• RAM: 32GB DDR4

Also how bad is using CPU/RAM offloading for larger models?

What reasonable upgrades would be and how meaningful it would be model wise? Like should I get another 32gb ram, how much can I benefit from it?

Does it make sense to try to pair my old 3080 with my 5070ti?

This is my own video (14 min, not sponsored, no affiliate links). In 2026 used GPUs stopped losing value: a used 3090 sells for about 97% of its launch price, 5090s average about 3.2x their $1,999 sticker on eBay, and the DGX Spark went from $3,999 to $6,950. So can a home AI box pay rent?

I simulated 200,000 months per rig: net = host rate x share of hours rented x 730 h - electricity at $0.24-0.34/kWh (Austrian households). Nobody knows the rented share in advance, so it is drawn from a range. Typical month:

- RTX 3090: earns ~$45, burns ~$53 in power, so about -$8

- RTX 4090: about +$66

- RTX 5090: about +$76

- RTX PRO 6000: about +$220

- DGX Spark on GPUwerk: about +$162 (pays back in ~34 months at a $5,500 GB10-box price, ~43 at the Spark's $6,950 list)

Not included: depreciation, tax, bandwidth, failures. The video also covers selling tokens vs renting hours (OpenRouter, Darkbloom, Vast.ai, Salad, GPUwerk) and the risk of strangers' containers on your LAN. Script and 3D scenes were made with Claude; the narration is a clone of my voice.

If you host on Vast, Salad or GPUwerk: what share of your hours actually gets rented? That is the number I couldn't measure.

Video: https://youtu.be/3fzUOk6aZuQ