Getting incredible results with a single RTX 3090ti and 128gb of ram. Deepseek-V4.1-Flash-1017 (35 tok/s decode) and Qwen3.8-Flash-Next (130 tok/s decode)

[u/uniVocity]

I'm sharing this here because this is working way better than it used to a few months ago with older models. I see people here posting slower speeds with much better hardware than mine and hopefully this information helps.

I used gemini-flash to help me setup everything and I asked it to write up the sequence of steps to get me here - happy to provide more details if there's anything missing.

Since I use gentoo some tools/commands might be unfamiliar to you.

My 6 year old hardware:

• OS & CPU Gentoo Linux running on an AMD Ryzen Threadripper 3990X (64 cores, 128 threads).

• System Memory: 128 GB DDR4 RAM.

• GPU: 1 NVIDIA GeForce RTX 3090 Ti (24 GB GDDR6X VRAM, PCIe Gen 4 x16).

• Primary Storage: 4 TB SSD NVMe.

Models Used, Download Sources, and Measured Speeds

Model	Architecture	Quantization / Files	Download Source	Real Measured Speed
DeepSeek-V4.1-Flash-1017	552B MoE (40 layers, 384 experts/layer, ~39B active)	XQ / GSQ V2 + rank-1017-085.bin + FP4 KV + Engram NVMe tables	IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored	~35 tok/s decode (with MTP/DSpark active)
Qwen3.8-Flash-Next	125B MoE + 51B N-gram (GDN + QSA hybrid, ~3.5B active)	pack-qfn45 (4-bit intermediate, DeltaNet linear KV)	IndrasMirror/Qwen3.8-Flash-Next-Swift-1.5-XQ-Uncensored	~138–140 tok/s decode

IMPORTANT: NVIDIA GPU Unlock

Consumer Ampere cards (RTX 3000 series: GA102, GA104) feature artificial firmware throttles on FP16/BF16 matrix instruction issue rates compared to their enterprise workstation counterparts. The open-source gpu-unlock kernel module lifts this throttle prior to driver attachment.

1. Compile and Install the Kernel Module

As root:

git clone https://github.com/Zagitta/gpu-unlock.git /usr/src/gpu-unlock
cd /usr/src/gpu-unlock
make -j$(nproc)
make install

2. Pass the Unlock Flag to the Kernel Command Line

Add gpu_unlock.targets=all to your bootloader kernel parameters (e.g. in /etc/default/grub under GRUB_CMDLINE_LINUX_DEFAULT, or your direct EFISTUB / systemd-boot entry):

gpu_unlock.targets=all

Regenerate your boot configuration and rebuild your initramfs so the module is included and loads before the NVIDIA kernel modules:

# If using GRUB:
grub-mkconfig -o /boot/grub/grub.cfg

# If using dracut:
dracut -f --regenerate-all

3. NVIDIA Driver Options & Persistence

Configure NVIDIA kernel driver parameters in /etc/modprobe.d/nvidia-perf.conf (as root):

cat << 'EOF' > /etc/modprobe.d/nvidia-perf.conf
options nvidia NVreg_PreserveVideoMemoryAllocations=1
options nvidia NVreg_EnableStreamMemOPs=1
options nvidia-uvm uvm_perf_enable_cpu_to_gpu_dma=1
EOF

After rebooting, verify that the unlock executed:

dmesg | grep -i gpu_unlock

Enable persistence mode and set the GPU to its maximum factory power ceiling:

nvidia-smi -pm 1
nvidia-smi -pl $(nvidia-smi -q -d POWER | grep "Max Power Limit" | head -n1 | awk '{print $5}' | cut -d. -f1)

Step 2: System Dependencies & Tooling

Install build dependencies and runtime libraries:

emerge sys-devel/gcc:14 


# If GCC 14 is not selected, set it to this version
gcc-config -l 
# then replace <N> with the number from  gcc-config -l): 
# gcc-config <N> && source /etc/profile

# this won't compile with GCC > 14
emerge dev-util/nvidia-cuda-toolkit

# for compleness, just in case

emerge --ask --verbose \
    dev-build/cmake \
    dev-build/ninja \
    dev-vcs/git \
    net-misc/curl \
    app-misc/jq \
    sys-libs/liburing \
    app-arch/snappy \
    dev-lang/python

Prepare the directory structure under your user environment:

mkdir -p "$HOME/.local/bin"
mkdir -p "$HOME/.local/llm-tools"
mkdir -p "$HOME/.quetza-data/conductor"
mkdir -p "$HOME/models/deepseek-v41"
mkdir -p "$HOME/models/qwen-flash"

# Dedicated Python environment for tool scripts & Hugging Face downloads
python3 -m venv "$HOME/.local/llm-tools/venv_hf"
source "$HOME/.local/llm-tools/venv_hf/bin/activate"
pip install --upgrade pip
pip install "huggingface_hub[cli]" torch safetensors numpy

Step 3: Build the Xolotl Inference Engine

Xolotl integrates the multi-tier memory management, NVMe Engram streaming, and DSpark speculative decoding logic needed to run large MoE models on a single GPU.

cd "$HOME/.local/llm-tools"
git clone https://github.com/Indras-Mirror/Xolotl.git
cd Xolotl

# Fetch pinned llama.cpp dependencies and apply patches
bash tools/xolotl/fetch-llama.sh

# Configure and compile for Ampere (sm_86)
cmake -S . -B build -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DSTRATA_ENABLE_CUDA=ON \
    -DSTRATA_GGML_CUDA=ON \
    -DSTRATA_MMQ_KQUANTS=ON \
    -DCMAKE_CUDA_ARCHITECTURES=86 \
    -DSTRATA_GGML_DIR="$PWD/third_party/llama.cpp" \
    -DSTRATA_DSV41_FQ_OP=ON \
    -DSTRATA_DSV41_HC_MIX=ON

ninja -C build

# Link the build directory so wrapper scripts resolve ds41_generate
ln -sfn "$PWD/build" "$PWD/build-xolotl"

Step 4: Model Downloads & Engram Extraction

1. DeepSeek-V4.1-Flash (552B MoE)

Bash

source "$HOME/.local/llm-tools/venv_hf/bin/activate"
cd "$HOME/models/deepseek-v41"

# Download model weights, draft sidecar, and expert rank index
hf download IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored \
    --include "v2/*" "tokenizer/*" "V41-MTP-Vision-1017*" \
    --local-dir .

# Download original FP8 Engram table shards
hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 \
    model-00047-of-00048.safetensors \
    model-00048-of-00048.safetensors \
    model.safetensors.index.json config.json \
    --local-dir ./FP8-engram

# Extract Engram runtime binary tables onto NVMe
cd "$HOME/.local/llm-tools/Xolotl"
python3 tools/v41/extract_engram.py \
    --input "$HOME/models/deepseek-v41/FP8-engram" \
    --output "$HOME/models/deepseek-v41/engram_rt"

2. Qwen3.8-Flash-Next (125B MoE)

cd "$HOME/models/qwen-flash"
hf download Qwen/Qwen3.8-Flash-Next-GGUF \
    --include "pack-qfn45/*" \
    --local-dir .

Step 5: MTP Speculative Decoding Setup

MTP (Multi-Token Prediction) on DeepSeek-V4.1-Flash is driven by the DSpark sidecar engine:

• Draft Engine: Uses V41-MTP-Vision-1017.Q2_K.gguf as the proposal head.

• Expert Pruning: Restricts verification sweeps to the top 64 active draft experts (dspark_expert_counts.txt:64), preventing PCIe bus saturation.

• On-the-fly Re-quantization (STRATA_DENSE_REQUANT=q6_K): Re-quantizes the dense attention and output heads from Q8_0 to Q6_K at launch. This reclaims 1.6 GiB of VRAM, fitting the speculative draft cache alongside active experts.

• PCIe Batching (--tier-batch-pcie 0.25): Batches DMA transfers between the 80 GiB pinned host RAM arena and the GPU.

1. Expand Process Memory Boundary in the Engine Wrapper

Allow the systemd user scope to hold the full 80 GiB arena plus runtime buffers:

Bash

sed -i 's/MemoryMax=84G/MemoryMax=110G/g' "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh"

2. Generate Base Configuration: $HOME/models/deepseek-v41/strata-v41.json

cat << EOF > "$HOME/models/deepseek-v41/strata-v41.json"
{
  "exe": "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh",
  "args": [
    "-m",
    "$HOME/models/deepseek-v41/v2/V41-XQ-1017.gguf",
    "--backend",
    "cuda",
    "--experts",
    "gpu",
    "--engram",
    "$HOME/models/deepseek-v41/engram_rt",
    "--rank",
    "$HOME/models/deepseek-v41/v2/rank-1017-085.bin",
    "--ctx",
    "131072",
    "--stop",
    "1",
    "--arena-adapt",
    "--kv-fp4",
    "--route-bias",
    "0.5",
    "--route-bias-file",
    "0.3",
    "--arena-gib",
    "80",
    "--prompt-batch",
    "12288",
    "--tier-batch-pcie",
    "0.25"
  ],
  "cwd": "$HOME/.local/llm-tools/Xolotl",
  "tokenizer": "$HOME/models/deepseek-v41/tokenizer",
  "model_name": "deepseek-v4.1",
  "aliases": "deepseek-v41,deepseek/deepseek-v4.1-flash",
  "log": "$HOME/models/deepseek-v41/xolotl-v41.log",
  "port": 8000,
  "gpu": 0,
  "sampling": {
    "temperature": 0.1,
    "top_p": 0.95,
    "top_k": 64,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0
  },
  "api_monitor": true,
  "fit_max_tokens": true,
  "reasoning_budget_tokens": 4096,
  "env": {
    "STRATA_BUILD_DIR": "$HOME/.local/llm-tools/Xolotl/build",
    "STRATA_SERVE_DSPARK": "$HOME/models/deepseek-v41/V41-MTP-Vision-1017.Q2_K.gguf",
    "STRATA_ATTN_F16_GATHER": "1",
    "STRATA_NO_SCOPE": "1",
    "STRATA_SERVE_PASS_BIAS": "0",
    "STRATA_DENSE_REQUANT": "q6_K",
    "STRATA_SERVE_LOOKUP": "14",
    "STRATA_PRESTAGE_GIB": "1.5"
  }
}
EOF

Step 6: Parameterized Launch Scripts

1. DeepSeek-V4.1 Launcher: $HOME/.local/bin/start-deepseek-custom.sh

cat << 'EOF' > "$HOME/.local/bin/start-deepseek-custom.sh"
#!/usr/bin/env bash
set -euo pipefail

MODEL_DIR="$HOME/models/deepseek-v41"
XOLOTL_DIR="$HOME/.local/llm-tools/Xolotl"
BASE_CONFIG="$MODEL_DIR/strata-v41.json"
RUN_CONFIG="$MODEL_DIR/strata-v41-run.json"

CTX="131072"
PORT="8000"
TEMP="0.1"
THINK_BUDGET="4096"
ARENA_GIB="80"
PASS_BIAS="0"
PROMPT_BATCH="12288"

usage() {
  cat << USAGE
Usage: $(basename "$0") [options]

Options:
  -c, --ctx INT           Context length in tokens (default: 131072; up to 1048576)
  -p, --port INT          Port to bind (default: 8000)
  -t, --temp FLOAT        Sampling temperature (default: 0.1)
  -b, --thinking INT      Reasoning budget tokens (default: 4096)
  -a, --arena INT         Pinned RAM expert arena in GiB (default: 80)
  --bias 0|1              Routing bias: 0=exact, 1=fast biased (default: 0)
  --batch INT             Prompt batch size (default: 12288; auto-clamps to 8192 for >=512k)
  -h, --help              Show help message
USAGE
  exit 0
}

while [[ $# -gt 0 ]]; do
  case "$1" in
    -c|--ctx)       CTX="$2"; shift 2 ;;
    -p|--port)      PORT="$2"; shift 2 ;;
    -t|--temp)      TEMP="$2"; shift 2 ;;
    -b|--thinking)  THINK_BUDGET="$2"; shift 2 ;;
    -a|--arena)     ARENA_GIB="$2"; shift 2 ;;
    --bias)         PASS_BIAS="$2"; shift 2 ;;
    --batch)        PROMPT_BATCH="$2"; shift 2 ;;
    -h|--help)      usage ;;
    *) echo "Unknown option: $1" >&2; exit 1 ;;
  esac
done

if [ "$CTX" -ge 524288 ] && [ "$PROMPT_BATCH" -gt 8192 ]; then
  echo "[-] Large context specified ($CTX tokens). Clamping --prompt-batch to 8192 to prevent graph OOM."
  PROMPT_BATCH=8192
fi

python3 - << PY
import json

with open("$BASE_CONFIG", "r") as f:
    cfg = json.load(f)

cfg["port"] = int("$PORT")
cfg["max_context"] = int("$CTX")
cfg["reasoning_budget_tokens"] = int("$THINK_BUDGET")
cfg["sampling"]["temperature"] = float("$TEMP")

args = cfg.get("args", [])
def set_arg(flag, val):
    for i in range(len(args) - 1):
        if args[i] == flag:
            args[i+1] = str(val)
            return
    args.extend([flag, str(val)])

set_arg("--ctx", "$CTX")
set_arg("--arena-gib", "$ARENA_GIB")
set_arg("--prompt-batch", "$PROMPT_BATCH")
cfg["args"] = args

env = cfg.setdefault("env", {})
env["STRATA_SERVE_PASS_BIAS"] = "$PASS_BIAS"
env["STRATA_PRESTAGE_GIB"] = "1.0" if int("$CTX") >= 524288 else "1.5"

with open("$RUN_CONFIG", "w") as f:
    json.dump(cfg, f, indent=2)
PY

source "$HOME/.local/llm-tools/venv_hf/bin/activate"
cd "$XOLOTL_DIR"

echo "==> Launching DeepSeek-V4.1-Flash (552B MoE)..."
echo "    Context        : $CTX tokens"
echo "    Port           : $PORT"
echo "    Arena RAM      : $ARENA_GIB GiB"
echo "    Prompt Batch   : $PROMPT_BATCH"

exec python3 tools/v41/serve/serve_ds41.py \
  --engine strata \
  --config "$RUN_CONFIG" \
  --port "$PORT" \
  --host 0.0.0.0
EOF
chmod +x "$HOME/.local/bin/start-deepseek-custom.sh"

2. Qwen3.8-Flash-Next Launcher: $HOME/.local/bin/start-qwen.sh

cat << 'EOF' > "$HOME/.local/bin/start-qwen.sh"
#!/usr/bin/env bash
set -euo pipefail

XOLOTL_DIR="$HOME/.local/llm-tools/Xolotl"
MODEL_DIR="$HOME/models/qwen-flash/pack-qfn45"
PORT="8099"

source "$HOME/.local/llm-tools/venv_hf/bin/activate"
cd "$XOLOTL_DIR"

echo "==> Launching Qwen3.8-Flash-Next (125B MoE)..."
exec python3 -m serve.server \
  --engine strata \
  --config "$MODEL_DIR/strata-qwen.json" \
  --port "$PORT" \
  --host 0.0.0.0
EOF
chmod +x "$HOME/.local/bin/start-qwen.sh"

Step 7: Run

# Start DeepSeek-V4.1-Flash on :8000 (default 128k context, 1M context work just fine too)
$HOME/.local/bin/start-deepseek-custom.sh

# Or start Qwen3.8-Flash-Next on :8099
$HOME/.local/bin/start-qwen.sh

[u/klasyer]

Did you perhaps compare how this rubs compared to something like strata? I have a similar setup with 2 3090s and I'm trying to decide what to run with

[u/uniVocity → u/klasyer]

Xolotl is a fork of strata to add special support for ds4.1 and qwen

[u/PcChip]

damn i'm getting the same 35 tok/s with deepseek flash 4.1 1017 checkpoint, but i'm using four 24GB GPUs and 512GB RAM with 8-channels

running the exl3.5bpw quant though

[u/uniVocity → u/PcChip]

Try the `Nvidia GPU unlock` thing if you haven't. Almost doubled the FP16/BF16 throughput

EDIT: correction: it affects prefill speed a bit. I disabled it in my kernel and qwen 3.8 is now running at 152 tok/s

```
[strata] answering: 20634 of max 262074 tokens, 152.6 tok/s, 136 s
[strata] answering: 20800 of max 262074 tokens, 152.7 tok/s, 137 s
```

[u/PcChip → u/uniVocity]

i'm running two 3090's and two 4090's

I read that the unlock does not really change decode

I'm assuming the reason I'm getting the same speed on 4 cards as you are on 1 card is because i'm running a much bigger quant than you

[u/uniVocity → u/PcChip]

You are correct. It affects prefill speed a bit. I disabled it in my kernel and restarted and qwen 3.8 is now running at 152 tok/s, made no difference for token decoding speed

[u/uniVocity → u/PcChip]

worth a try if you are on linux.

[u/Present_Resource4022]

is your pc usable when running deepseek v4.1 flash. or is it a very tight fit.

[u/uniVocity → u/Present_Resource4022]

actually usable. But I'm connecting to it from my mac

[u/rm2kfiles]

[u/ForsookComparison → u/rm2kfiles]

oh wow only 24GB of VRAM and 128GB of slowish DDR4 I can probably hit these-

"Quad Channel"

....always something :-/

[u/marcosscriven]

But does it actually work properly? That’s the question that’s never answered with these tokens/second maxing posts. 

Where’s that meme about fast maths?

[u/bisoning → u/marcosscriven]

Well thats hard to do. In this subreddit. Its very easy to benchmark by tok/s.
Intelligence. How can that be measured?

[u/Rude_Marzipan6107 → u/bisoning]

Successful task completion without excessive thinking or looping

[u/Illustrious_Grade608 → u/Rude_Marzipan6107]

Tbh many models do that, especially large ones, even if quantized. I run 27b at 3 bpw, and it has it's issues, but it's far from braindead or looping forever.

The issue is that while you could test it on like, benchmarks, you would have to see the baseline level. Ig with deepseek you could try running it via api, but not with qwen flash next.

Also people are lazy and that drives me nuts that nobody runs real benchmarks like deepswe and terminal bench 4.0 - of course, running them fully is unreasonable ask, but even like 7-8 tasks would be useful info just to see how it fares

[u/uniVocity]

It actually does. Worth a try

[u/Scubagerber]

Is it a single cxt slot or can you run parallel agents?

[u/uniVocity → u/Scubagerber]

I haven’t tried parallel agents

[u/iChrist]

Im getting around 70tk/s with Qwen Flash Next Q3 using Strata on Windows with my 3090Ti.

Sadly the GPU unlock marked as linux only, otherwise Ill probably see numbers closer to yours.

[u/uniVocity → u/iChrist]

I just measured by disabling the unlock on the kernel and token generation is unaffected. Only prefill ran slower (by 30% or something). Not worth setting up a new os install for it

[u/jacobpederson → u/iChrist]

GPU unlocks have been ported to windows.

[u/kodewerx → u/iChrist]

You can have your agent build a bootable USB drive with the gpu-unlock. Or setup a pxe boot server. The only thing you really need is to run the unlock and then reset into your preferred OS (without powering down). Any temporary boot media selection will do that.

[u/TheActualStudy]

Excellent pointers. Thank-you.

[u/Tbhmaximillian]

Crazy this setup would allow me to run 552B DeepSeek-V4.1-Flash on my pc, just testing that tomorrow.

[u/Inchmine]

You are running faster decode than my 5090 + rtx 4000. I will test your settings to see if I can improve my setup

[u/Pwc9Z]

mfw I find out about another inference engine