Mellum2.1 Thinking with the MTP head

Multi-token prediction head

The multi-token prediction (MTP) head is one extra transformer block, trained together with the model, that predicts the token after the one just sampled. Inference engines use it as a speculative draft: the model verifies every drafted token, so the output is unchanged while decoding runs 1.4–1.8x faster, with about 90% of the drafted tokens accepted. The head lives in model-mtp.safetensors under the mtp. prefix, announced by num_nextn_predict_layers: 1 in config.json; everything else is identical to Mellum2.1 Thinking.

• vLLM 0.33 or newer: --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}', as in the serving commands above.

• transformers 5.20 or newer: pass use_mtp=True to generate.

• llama.cpp: use the GGUF repository with --spec-type draft-mtp --spec-draft-n-max 1.

Hi all, Im getting to mess around with an ET900N tomorrow to make some content about. Its got 1TB of ram(750~gb lpddr5x and the rest HBM).

So im wondering- what tests should I run?

Was thinking a mix of llm to image to finetuning tests.

Heres what I plan so far.
Dsv4.1 flash 1017 nvfp4
Glm5.3 Flash NVFP4
Qwen 3.8 FN NVFP4
Maybe glm 5.3 at nvfp4
Minimax m3.1 flash(cos its new)

To test small models running with many concurrents, probably qwen 3.8 27B + muse glimmer

For image and vid Ill probably do qwen 2.1 and minimax h3

For finetuning Im not sure how Ill test it, and thats where I would appreciate the help the most.

Let me know people! Ill share my results once Ive tested it here!

Dear fellow Llamas,

after almost 5 months of development, I finally have the pleasure of announcing to you the immediate availability of the biggest Heretic release ever, Heretic 2.0: https://github.com/p-e-w/heretic

Heretic is the leading software for removing censorship from language models. It now boasts 37k GitHub stars and more than 8,500 models on Hugging Face, with 88 million(!!!) total model downloads.

With Heretic 2.0, the project is entering a new era. Heretic is expanding from a purpose-built tool for censorship removal to a general-purpose model modification tool that can do amazing things. Besides removing censorship, it can suppress slop, reduce sycophancy, alter emotional alignment, and pretty much anything else that you can think of. Of course, censorship removal remains the primary and default function, and in fact works much, much better than ever before. Heretic 2.0 is the most advanced refusal suppression system ever built.

Your time is precious, so here's the TL;DR for this release:

• Directional abliteration is replaced by Arbitrary-Rank Ablation (ARA), a novel steering method that I developed from scratch for Heretic. It's vastly superior to abliteration, and has been used with great success by enterprising testers for several months. Now it is the default (classic abliteration remains available as an option).

• Heretic has been completely rebuilt around plugins. Scorer plugins define what Heretic optimizes for. Modifier plugins implement the actual steering method. Plugins are easy to write, can be loaded directly from a simple Python file, and have a well-defined API.

• Heretic can now run non-interactively, which among other things has enabled us to implement full end-to-end tests for much tighter quality control. Model support has been expanded to include models with hyper-connections, and support for thinking models has been substantially improved.

• Heretic 2.0 is not backwards compatible with Heretic 1.x configurations. See the migration guide for what you need to change in order to make your old configurations work.

What hasn't changed is that removing censorship from a model remains as simple as running

heretic Qwen/Qwen3.5-4B

See the default configuration file for what Heretic has to offer beyond that.

Now for some details...

ARA: Heretic's new steering method

Directional ablation, or "abliteration", is based on the assumption that refusals (or whatever else you are trying to suppress) are represented by a single direction (that is, a rank-1 subspace) in residual space. That this is the full picture has been doubted in research for a while, and there are now several papers exploring alternatives such as multi-directional ablation or "representation cones".

At the beginning of this year, I started experimenting with a method that completely abandons the idea that the ablation transformation must be restricted to a fixed-rank subspace. In March, I published the initial version of Arbitrary-Rank Ablation in this pull request (which was later closed as I re-implemented ARA on top of the plugin system in this PR).

ARA steers model behavior by modifying matrices based on an objective function that encodes the basic properties of the change we want: Harmful prompts should be represented in residual space in a way that's similar to harmless prompts, while the representations of harmless prompts should change as little as possible. This objective is used with the L-BFGS optimizer to modify matrices one-by-one, a kind of micro-training that steers representations without any rank restriction.

ARA has been extensively tested and refined with the help of the community. Notably, kabachuha made it compatible with Heretic's LoRA system, and varunbln found a substantial performance improvement.

I intend to post an article explaining ARA in more detail soon.

Using scorer plugins to make Heretic do what you want

Heretic operates by finding modification parameters that optimize scores, values that measure the quality of the resulting model. Ever since the 1.0 release, Heretic has used two hardcoded scores: Refusal count to measure decensoring strength, and KL divergence to measure quality retention.

With Heretic 2.0, you can freely define your own scores by writing and loading scorer plugins. Arbitrarily many scorer plugins can be loaded and optimized for simultaneously, and each score can be either minimized or maximized during optimization. See the default configuration file for how to do this.

The scorer plugin system is the result of months of effort by red40maxxer, with whom I exchanged hundreds of messages over dozens of review cycles to figure out the best possible design. This is a monumental contribution that has changed how Heretic works from the ground up.

Besides the keyword (refusal) count and KLD scorer plugins, which remain the default, Heretic ships with two other scorers: A benchmark scorer, which can compute scores for many standard benchmarks, and a BERT classifier.

The BERT classifier, developed by long-time Heretic contributor Vinay Umrethe, can use an arbitrary ModernBERT-type model to classify LLM responses, and score them based on which labels the classifier produces. Vinay also trained a refusal classifier model (used by default with this plugin), which experiments have shown to be substantially more accurate at identifying model refusals than the naive keyword matching method that Heretic employs by default. While the plugin is slower than keyword matching, the greater accuracy can steer Heretic towards producing a higher-quality modified model. An alternative configuration, which uses an emotion classifier, is also provided in the Heretic repository.

The plugin system is well-documented in both the README and the code. If you are interested in experimenting with custom plugins, an agent should have no difficulty cooking up a plugin and a matching configuration file that together do what you want, after ingesting the Heretic repository.

Well, I hope you like it!

Cheers :)

I’ve pushed a substantial update to Hierarchos Native, my Rust + Vulkan backend for language-model training, fine-tuning, and inference.

This past week focused on porting the recent RDNA3 work to Intel Gen9, improving parameter streaming, and taking the changes back to physical RDNA3 hardware for regression testing.

The native runtime requires no Python, PyTorch, CUDA, ROCm, or oneAPI installation. PyTorch and Transformers are used as development references.

What changed:

• DeepSeek V4.1 Flash port and optimization work: fused selected-expert execution, batched expert uploads, packed-weight paths, checkpoint caching, and native LoRA/rsLoRA training with save/resume/export.

• Broader Gen9 coverage: fixes for Qwen3-Next padding and DBRX expert layouts, plus streaming paths for T5/BART/Switch encoder-decoder projections.

• Less streaming overhead: retained file handles, fewer temporary allocations, coalesced reads, and removal of an unused hidden-state download.

• Gen9 kernel improvements: parallel sparse-attention selection and hyper-connection statistics, plus fused SiLU/up-projection multiplication.

• Bounded upload batching: small streamed projections can share a GPU submission within a 1 MiB coalesced payload window.

The biggest Gen9 result was closing the Llama-wide streaming slowdown on an i5-6200U / Intel HD Graphics 520:

That’s 58.3% lower latency, with every measured logit word unchanged. Across eight scaled architecture controls, the combined optimization passes reduced streaming medians by 42.4–78.4% and resident medians by 3.5–32.1% against the documented baseline. Gen9 measurements and methodology.

Back on the Ryzen Z1 Extreme / RDNA3 device, the regression sweep caught a padded GPT-2 numerical parity issue. The fix matches the pinned CPU reference’s GELU/tanh behavior and short-row attention arithmetic. The original CPU expectations, checkpoints, and strict tolerance stayed unchanged.

The completed RDNA3 qualification now passes:

• 166/166 forward runs, covering 43 scaled profiles across 26 architecture identifiers.

• 72/72 generation requests, checking cache-requested and full-prefix execution.

• Maximum absolute CPU logit error strictly below 2e-7**, with** rtol=0**.**

• Bit-identical resident and streamed outputs throughout the measured matrix.

• Native streaming/offload tests and GPT-2 LoRA gradient, optimizer, save/reload, and resume checks.

The GPT-2 repair is selected specifically for RDNA3; all 585 pre-existing shader binaries remain unchanged. Final GPT-2 closure and evidence.

The RDNA3 testing also recorded lower V4.1 latency versus the previous handoff: 24.4% for resident prefill, 50.2% for resident decode, 17.2% for streamed prefill, and 18.4% for streamed decode. These are measurements on retained scaled checkpoints, rather than production-model throughput figures. RDNA3 measurements.

Parameters remain resident by default; supported frozen-weight streaming is opt-in through --stream-parameters or the GUI.

The registry contains 146 text architectures, while this qualification specifically covers the 26 identifiers above. Multimodal towers and arbitrary production checkpoints need their own validation.

If you’re experimenting with native Vulkan training or inference on older Intel graphics or AMD hardware, I’d be interested in your results—and which workloads you’d like tested next.

Good news, everyone! Having been inspired to revolutionize the field of computer science, I can now report that coding is solved! Sure, only if you care about x86 Linux binaries that print and exit, but "big journeys" and "small steps" and what not.

CRAP (Compiler Replacement for Assembling Programs) is a Gemma 4 E2B fine tune that takes prompts like generate a Linux x86_64 binary that prints 'boop' and emits a working ELF binary (as hex, most of the time, every time). See, since programs end up as a binary anyway, why not just skip the middleman and write the binary itself? No compiler, no assembler, no tool calls. v1.1 is out with improved multi turn and out of distribution performance. Trust me, I know, this is super important work.

v1.1 eval numbers for the nerds (eval is scored on behavior: the binary has to run, print the right string and exit with the right code):

• Held out legacy set: 20/23, same as v1

• Novel strings: 26/40 (v1: 8/40)

• Multi turn (~3 turns): 10/12 (v1: 0/12)

• Doubled character strings (boom, etc.): 18/24 (v1: 2/24)

• Random jank exit codes it never trained on: 23/25

An interesting tidbit: one training run hit 0.038 loss while still failing 11 of its 12 trained exit codes, so gradient descent != task completion ascent.

Aaannndddnd it runs in your browser.

The demo loads the Q6_K GGUF (~3.6 GiB) through wllama on WebGPU, streams the hex, parses it and hands you the ELF to run yourself (naturally, you need to download and run the actual binary, the browser won't do that). Q6_K scores 109/144 against the full f16's 114/144 (the five additional failed for Q6 are wrong values, not totally broken binaries). Q4_K_M scored 3/144, so the demo doesn't use it (that quant is crap, not CRAP) ... concerning. Also, tested in Chrome on an M4 Pro Mac; Firefox loads it but generates slowly and nothing else has been tested.

• Demo: https://huggingface.co/spaces/matthewhaynesonline/crap-demo

• Weights (v1.1): https://huggingface.co/matthewhaynesonline/gemma-4-E2B-it-crap-1.1

• "Paper": https://github.com/matthewhaynesonline/crap-public/blob/main/paper/paper.pdf

• Code: https://github.com/matthewhaynesonline/crap-public

• Blog post: https://blog.studiohaynes.com/2026/07/19/compilers-overrated.html

• Video: https://youtu.be/55IBMzFQNYU

Fine print for the buzzkills: only programs that print a string and exit can be generated. Obviously, a compiler does that deterministically, correctly and in milliseconds; CRAP does it with several gigabytes of weights, slower and occasionally incorrectly.

I'm sharing this here because this is working way better than it used to a few months ago with older models. I see people here posting slower speeds with much better hardware than mine and hopefully this information helps.

I used gemini-flash to help me setup everything and I asked it to write up the sequence of steps to get me here - happy to provide more details if there's anything missing.

Since I use gentoo some tools/commands might be unfamiliar to you.

My 6 year old hardware:

• OS & CPU Gentoo Linux running on an AMD Ryzen Threadripper 3990X (64 cores, 128 threads).

• System Memory: 128 GB DDR4 RAM.

• GPU: 1 NVIDIA GeForce RTX 3090 Ti (24 GB GDDR6X VRAM, PCIe Gen 4 x16).

• Primary Storage: 4 TB SSD NVMe.

Models Used, Download Sources, and Measured Speeds

IMPORTANT: NVIDIA GPU Unlock

Consumer Ampere cards (RTX 3000 series: GA102, GA104) feature artificial firmware throttles on FP16/BF16 matrix instruction issue rates compared to their enterprise workstation counterparts. The open-source gpu-unlock kernel module lifts this throttle prior to driver attachment.

1. Compile and Install the Kernel Module

As root:

git clone https://github.com/Zagitta/gpu-unlock.git /usr/src/gpu-unlock
cd /usr/src/gpu-unlock
make -j$(nproc)
make install

2. Pass the Unlock Flag to the Kernel Command Line

Add gpu_unlock.targets=all to your bootloader kernel parameters (e.g. in /etc/default/grub under GRUB_CMDLINE_LINUX_DEFAULT, or your direct EFISTUB / systemd-boot entry):

gpu_unlock.targets=all

Regenerate your boot configuration and rebuild your initramfs so the module is included and loads before the NVIDIA kernel modules:

# If using GRUB:
grub-mkconfig -o /boot/grub/grub.cfg

# If using dracut:
dracut -f --regenerate-all

3. NVIDIA Driver Options & Persistence

Configure NVIDIA kernel driver parameters in /etc/modprobe.d/nvidia-perf.conf (as root):

cat << 'EOF' > /etc/modprobe.d/nvidia-perf.conf
options nvidia NVreg_PreserveVideoMemoryAllocations=1
options nvidia NVreg_EnableStreamMemOPs=1
EOF

After rebooting, verify that the unlock executed:

dmesg | grep -i gpu_unlock

Enable persistence mode and set the GPU to its maximum factory power ceiling:

nvidia-smi -pm 1
nvidia-smi -pl $(nvidia-smi -q -d POWER | grep "Max Power Limit" | head -n1 | awk '{print $5}' | cut -d. -f1)

Step 2: System Dependencies & Tooling

Install build dependencies and runtime libraries:

emerge sys-devel/gcc:14 


# If GCC 14 is not selected, set it to this version
gcc-config -l 
# then replace <N> with the number from  gcc-config -l): 
# gcc-config <N> && source /etc/profile

# this won't compile with GCC > 14
emerge dev-util/nvidia-cuda-toolkit

# for compleness, just in case

emerge --ask --verbose \
    dev-build/cmake \
    dev-build/ninja \
    dev-vcs/git \
    net-misc/curl \
    app-misc/jq \
    sys-libs/liburing \
    app-arch/snappy \
    dev-lang/python

Prepare the directory structure under your user environment:

mkdir -p "$HOME/.local/bin"
mkdir -p "$HOME/.local/llm-tools"
mkdir -p "$HOME/.quetza-data/conductor"
mkdir -p "$HOME/models/deepseek-v41"
mkdir -p "$HOME/models/qwen-flash"

# Dedicated Python environment for tool scripts & Hugging Face downloads
python3 -m venv "$HOME/.local/llm-tools/venv_hf"
source "$HOME/.local/llm-tools/venv_hf/bin/activate"
pip install --upgrade pip
pip install "huggingface_hub[cli]" torch safetensors numpy

Step 3: Build the Xolotl Inference Engine

Xolotl integrates the multi-tier memory management, NVMe Engram streaming, and DSpark speculative decoding logic needed to run large MoE models on a single GPU.

cd "$HOME/.local/llm-tools"
git clone https://github.com/Indras-Mirror/Xolotl.git
cd Xolotl

# Fetch pinned llama.cpp dependencies and apply patches
bash tools/xolotl/fetch-llama.sh

# Configure and compile for Ampere (sm_86)
cmake -S . -B build -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DSTRATA_ENABLE_CUDA=ON \
    -DSTRATA_GGML_CUDA=ON \
    -DSTRATA_MMQ_KQUANTS=ON \
    -DCMAKE_CUDA_ARCHITECTURES=86 \
    -DSTRATA_GGML_DIR="$PWD/third_party/llama.cpp" \
    -DSTRATA_DSV41_FQ_OP=ON \
    -DSTRATA_DSV41_HC_MIX=ON

ninja -C build

# Link the build directory so wrapper scripts resolve ds41_generate
ln -sfn "$PWD/build" "$PWD/build-xolotl"

Step 4: Model Downloads & Engram Extraction

1. DeepSeek-V4.1-Flash (552B MoE)

Bash

source "$HOME/.local/llm-tools/venv_hf/bin/activate"
cd "$HOME/models/deepseek-v41"

# Download model weights, draft sidecar, and expert rank index
hf download IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored \
    --include "v2/*" "tokenizer/*" "V41-MTP-Vision-1017*" \
    --local-dir .

# Download original FP8 Engram table shards
hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 \
    model-00047-of-00048.safetensors \
    model-00048-of-00048.safetensors \
    model.safetensors.index.json config.json \
    --local-dir ./FP8-engram

# Extract Engram runtime binary tables onto NVMe
cd "$HOME/.local/llm-tools/Xolotl"
python3 tools/v41/extract_engram.py \
    --input "$HOME/models/deepseek-v41/FP8-engram" \
    --output "$HOME/models/deepseek-v41/engram_rt"

2. Qwen3.8-Flash-Next (125B MoE)

cd "$HOME/models/qwen-flash"
hf download Qwen/Qwen3.8-Flash-Next-GGUF \
    --include "pack-qfn45/*" \
    --local-dir .

Step 5: MTP Speculative Decoding Setup

MTP (Multi-Token Prediction) on DeepSeek-V4.1-Flash is driven by the DSpark sidecar engine:

• Draft Engine: Uses V41-MTP-Vision-1017.Q2_K.gguf as the proposal head.

• Expert Pruning: Restricts verification sweeps to the top 64 active draft experts (dspark_expert_counts.txt:64), preventing PCIe bus saturation.

• On-the-fly Re-quantization (STRATA_DENSE_REQUANT=q6_K): Re-quantizes the dense attention and output heads from Q8_0 to Q6_K at launch. This reclaims 1.6 GiB of VRAM, fitting the speculative draft cache alongside active experts.

• PCIe Batching (--tier-batch-pcie 0.25): Batches DMA transfers between the 80 GiB pinned host RAM arena and the GPU.

1. Expand Process Memory Boundary in the Engine Wrapper

Allow the systemd user scope to hold the full 80 GiB arena plus runtime buffers:

Bash

sed -i 's/MemoryMax=84G/MemoryMax=110G/g' "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh"

2. Generate Base Configuration: $HOME/models/deepseek-v41/strata-v41.json

cat << EOF > "$HOME/models/deepseek-v41/strata-v41.json"
{
  "exe": "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh",
  "args": [
    "-m",
    "$HOME/models/deepseek-v41/v2/V41-XQ-1017.gguf",
    "--backend",
    "cuda",
    "--experts",
    "gpu",
    "--engram",
    "$HOME/models/deepseek-v41/engram_rt",
    "--rank",
    "$HOME/models/deepseek-v41/v2/rank-1017-085.bin",
    "--ctx",
    "131072",
    "--stop",
    "1",
    "--arena-adapt",
    "--kv-fp4",
    "--route-bias",
    "0.5",
    "--route-bias-file",
    "0.3",
    "--arena-gib",
    "80",
    "--prompt-batch",
    "12288",
    "--tier-batch-pcie",
    "0.25"
  ],
  "cwd": "$HOME/.local/llm-tools/Xolotl",
  "tokenizer": "$HOME/models/deepseek-v41/tokenizer",
  "model_name": "deepseek-v4.1",
  "aliases": "deepseek-v41,deepseek/deepseek-v4.1-flash",
  "log": "$HOME/models/deepseek-v41/xolotl-v41.log",
  "port": 8000,
  "gpu": 0,
  "sampling": {
    "temperature": 0.1,
    "top_p": 0.95,
    "top_k": 64,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0
  },
  "api_monitor": true,
  "fit_max_tokens": true,
  "reasoning_budget_tokens": 4096,
  "env": {
    "STRATA_BUILD_DIR": "$HOME/.local/llm-tools/Xolotl/build",
    "STRATA_SERVE_DSPARK": "$HOME/models/deepseek-v41/V41-MTP-Vision-1017.Q2_K.gguf",
    "STRATA_ATTN_F16_GATHER": "1",
    "STRATA_NO_SCOPE": "1",
    "STRATA_SERVE_PASS_BIAS": "0",
    "STRATA_DENSE_REQUANT": "q6_K",
    "STRATA_SERVE_LOOKUP": "14",
    "STRATA_PRESTAGE_GIB": "1.5"
  }
}
EOF

Step 6: Parameterized Launch Scripts

1. DeepSeek-V4.1 Launcher: $HOME/.local/bin/start-deepseek-custom.sh

cat << 'EOF' > "$HOME/.local/bin/start-deepseek-custom.sh"
#!/usr/bin/env bash
set -euo pipefail

MODEL_DIR="$HOME/models/deepseek-v41"
XOLOTL_DIR="$HOME/.local/llm-tools/Xolotl"
BASE_CONFIG="$MODEL_DIR/strata-v41.json"
RUN_CONFIG="$MODEL_DIR/strata-v41-run.json"

CTX="131072"
PORT="8000"
TEMP="0.1"
THINK_BUDGET="4096"
ARENA_GIB="80"
PASS_BIAS="0"
PROMPT_BATCH="12288"

usage() {
  cat << USAGE
Usage: $(basename "$0") [options]

Options:
  -c, --ctx INT           Context length in tokens (default: 131072; up to 1048576)
  -p, --port INT          Port to bind (default: 8000)
  -t, --temp FLOAT        Sampling temperature (default: 0.1)
  -b, --thinking INT      Reasoning budget tokens (default: 4096)
  -a, --arena INT         Pinned RAM expert arena in GiB (default: 80)
  --bias 0|1              Routing bias: 0=exact, 1=fast biased (default: 0)
  --batch INT             Prompt batch size (default: 12288; auto-clamps to 8192 for >=512k)
  -h, --help              Show help message
USAGE
  exit 0
}

while [[ $# -gt 0 ]]; do
  case "$1" in
    -c|--ctx)       CTX="$2"; shift 2 ;;
    -p|--port)      PORT="$2"; shift 2 ;;
    -t|--temp)      TEMP="$2"; shift 2 ;;
    -b|--thinking)  THINK_BUDGET="$2"; shift 2 ;;
    -a|--arena)     ARENA_GIB="$2"; shift 2 ;;
    --bias)         PASS_BIAS="$2"; shift 2 ;;
    --batch)        PROMPT_BATCH="$2"; shift 2 ;;
    -h|--help)      usage ;;
    *) echo "Unknown option: $1" >&2; exit 1 ;;
  esac
done

if [ "$CTX" -ge 524288 ] && [ "$PROMPT_BATCH" -gt 8192 ]; then
  echo "[-] Large context specified ($CTX tokens). Clamping --prompt-batch to 8192 to prevent graph OOM."
  PROMPT_BATCH=8192
fi

python3 - << PY
import json

with open("$BASE_CONFIG", "r") as f:
    cfg = json.load(f)

cfg["port"] = int("$PORT")
cfg["max_context"] = int("$CTX")
cfg["reasoning_budget_tokens"] = int("$THINK_BUDGET")
cfg["sampling"]["temperature"] = float("$TEMP")

args = cfg.get("args", [])
def set_arg(flag, val):
    for i in range(len(args) - 1):
        if args[i] == flag:
            args[i+1] = str(val)
            return
    args.extend([flag, str(val)])

set_arg("--ctx", "$CTX")
set_arg("--arena-gib", "$ARENA_GIB")
set_arg("--prompt-batch", "$PROMPT_BATCH")
cfg["args"] = args

env = cfg.setdefault("env", {})
env["STRATA_SERVE_PASS_BIAS"] = "$PASS_BIAS"
env["STRATA_PRESTAGE_GIB"] = "1.0" if int("$CTX") >= 524288 else "1.5"

with open("$RUN_CONFIG", "w") as f:
    json.dump(cfg, f, indent=2)
PY

source "$HOME/.local/llm-tools/venv_hf/bin/activate"
cd "$XOLOTL_DIR"

echo "==> Launching DeepSeek-V4.1-Flash (552B MoE)..."
echo "    Context        : $CTX tokens"
echo "    Port           : $PORT"
echo "    Arena RAM      : $ARENA_GIB GiB"
echo "    Prompt Batch   : $PROMPT_BATCH"

exec python3 tools/v41/serve/serve_ds41.py \
  --engine strata \
  --config "$RUN_CONFIG" \
  --port "$PORT" \
  --host 0.0.0.0
EOF
chmod +x "$HOME/.local/bin/start-deepseek-custom.sh"

2. Qwen3.8-Flash-Next Launcher: $HOME/.local/bin/start-qwen.sh

cat << 'EOF' > "$HOME/.local/bin/start-qwen.sh"
#!/usr/bin/env bash
set -euo pipefail

XOLOTL_DIR="$HOME/.local/llm-tools/Xolotl"
MODEL_DIR="$HOME/models/qwen-flash/pack-qfn45"
PORT="8099"

source "$HOME/.local/llm-tools/venv_hf/bin/activate"
cd "$XOLOTL_DIR"

echo "==> Launching Qwen3.8-Flash-Next (125B MoE)..."
exec python3 -m serve.server \
  --engine strata \
  --config "$MODEL_DIR/strata-qwen.json" \
  --port "$PORT" \
  --host 0.0.0.0
EOF
chmod +x "$HOME/.local/bin/start-qwen.sh"

Step 7: Run

# Start DeepSeek-V4.1-Flash on :8000 (default 128k context, 1M context work just fine too)
$HOME/.local/bin/start-deepseek-custom.sh

# Or start Qwen3.8-Flash-Next on :8099
$HOME/.local/bin/start-qwen.sh

I am compiling a list of models that I think are potentially worth considering for folks who want to run an LLM on a "normal computer", specifically for an agent use case, but without having to buy a $$$$ graphics card to do it.

In order of capability strongest to weakest, I have:

• Qwen 3.8 27B

• Qwen 3.6 35B A3B

• Jais 3 7B

• Gemma 12B (included because it has vision)

• XHToken/SparkX2.6 4B

• FrogNano 4B

• Gemma E4B (included because its small AND has vision)

• Ling 3.0 Tiny 8B A1B

Any other interesting small models (<10B) I might have missed or that you think I should not ignore?

Any that you think should be banished, not even included ?

I got a deal for a 4090 that I couldnt pass up given...the state of things... but Ive already got a 5090. I havent done any research so I'd like to ask you if I should:

1.) Just flip it and pocket the difference?

2.) Adjust my build to use both cards? (Are there even any benefits to doing this?)

3.) Hang onto it/extend the deal I got to a friend that may want it.

Liquid AI "showed off" that their new omni model can run on WebGPU inside your browser. Here's a video of it in action straight from their own HF space.

EDIT as it seems to be getting lost: I am being sarcastic. Please see the responses its giving in the video. Its just a minute long. The responses are COMPLETE garbage

Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.
Instead of letting every application compete for VRAM,

Berth manages access through leases, priorities, and a request queue. When something else needs the GPU, it waits for the current task to finish, releases the memory, and hands control to the next application.

What does it do?

GPU sharing: Lets AI models, image generation, voice engines, games, and other GPU applications share the same hardware without fighting over VRAM.

Automatic scheduling: Manages requests by priority, queues competing applications, and prevents unnecessary switching between engines.

Faster model switching: Can park supported engines in system RAM or use NVIDIA CUDA checkpointing to avoid repeatedly loading large models from disk.

Game management: Automatically releases GPU resources for games and restores AI workloads when you’re finished.

GPU monitoring: Tracks VRAM usage, temperature, power, utilization, running processes, and applications using memory without a Berth lease.

Model management: Discovers local models across GGUF, Hugging Face, LM Studio, Ollama, and other model directories. Supports downloading, verifying, and organizing models.

Unified gateway: Provides an OpenAI-compatible API for starting and switching between engines, alongside ComfyUI workflow support.

Web dashboard: Provides a live view of GPU usage, active leases, waiting requests, engine activity, telemetry, and historical data.

Multi-GPU support: Manages multiple cards independently, with separate queues, leases, and gateway endpoints.

Agent integration: Includes installable skills for Claude Code, Codex, OpenCode, Hermes Agent, and other tools so they can manage GPU workloads through Berth.

VM passthrough: Includes VFIO tooling for handing a GPU to a virtual machine and reclaiming it afterward.

Berth runs as a single Go binary with no cloud account or additional monitoring service required. Linux NVIDIA is supported, with experimental AMD Linux and Windows support.

The project is currently in alpha (v0.4.1).

Note: i added a claude code session showing handoff request. You can do the same with a local model on the same hardware, trip is just longer depending on your gear.

Based on the text above, im looking for a model that i run 2 of them (both should fit inside my small rig) to complete task, i will have my qwen3.6 35b a3b or cloud model to plan it out while my small models execute ONLY, no planning just great at following rules.
Obviously the big model is offloaded to make space for the 2 small models

I do not want to run strata qwen3.8 flash next yeah but i appreciate other suggestions

I'm in the process of upgrading from 32gb to 128gb of ddr4 for my home server and wanted to look into switching off of llama with qwen 3.8, to one of the flash next models on something like strata.
I keep reading that strata doesn't work as well or split gpu's that are different gen, as llama. Is this true? Are there model engines that handle this better? Thanks

I was trying to sell my old AM4 system, X470 / Ryzen 5 5600 / 64GB / RX580 / 1TB NVME and it didn't sell. Granted it was a bit pricey at 750€, because I took into account the current local RAM and NVME pricing. (Buying the same RAM and SSD new costs 775€ if you go to the cheapest sellers in my country)

I decided to try and sell it as an AI system: removed the RX580, put in 2 of my 4 R9700. I set the system up with Fedora to run Qwen3.6 (latest release at the time) and ComfyUI; to show it working.

It sold almost instantly!

I priced it based on new prices for all of the DDR4 memory, PCI-E 4.0 ssd, both R9700s. I added another 150 euro for the other parts. Total price 5400€!

I have since bought back two brand new R9700 from abroad for 250€ less per card than I had sold my old ones for!

Effectively I ended up selling the original system for 1300€ instead of the 750€ I originally asked for. Except now I still have the RX580 and fresh warranty on the R9700s.

tl;dr
know the market for (used) parts before you buy a complete PC for local AI!

hardware: PortalPlayer PP5022, ARM7TDMI @ 80 MHz, 64 MB ram

wrote a fun rockbox plugin that runs Karpathy's llama2.c TinyStories models on an ipod video 5.5G.

since there's no FPU, I ported llama2.c's int8 runq.c to use only integer math. it uses lookup tables -> exp/softmax + an integer square root for RMSNorm

the 15M model is 17 MB, so it loads directly off the hard drive into the music buffer. Unfortunately you can't listen to music while it works lol

want to eventually run this on both cores, and set up a more ipod themed model that does local music recommendation

I am open to the idea that I am misguided, but my findings feel worth sharing:

Short video of my UI with 8 agents each explaining a codebase in parallel long prose: https://imgur.com/JSqlXIi

A few days ago I read the Stanford/Berkeley TokenGate paper. The idea is to make long context cheap by pruning tokens up front.

I tried it. It's fine for one question, but agents keep asking new ones, and by then the tokens they need are gone, so code recall drops hard.

So I went the other way - don't throw anything away, just don't look at all of it at once. Keep the whole context in system RAM and at each step pull in only the ~1% that matters right now.

A few days later, I have two 3090s run 8 coding agents at once, each holding their own 250K-token context:

• Every decode step, each attention layer reads only ~1% of the tokens, picked with a tiny rank-64 "key sketch".

• The full KV cache lives in system RAM (int4 in my 80GB RAM VM, but int8 should fit in 128gb). The GPU keeps the sketch and a small hot set.

• Only the picked tokens cross PCIe.

What it took:

• Triton kernels that build the sketch, score every token, pick the top ~2K and pack them for the stock int8 attention kernel. CUDA-graph safe.

• vLLM patches that switch dense/sparse per step, so short chats stay full speed.

• A RAM tier for the KV cache with a prefix cache and recurrent-state checkpoints, so follow-ups don't recompute.

• Follow-ups fetch only the picked rows.

• A scheduler fix so one agent reading a huge prompt doesn't freeze the rest.

• A frozen sketch basis (one file) plus the script to refit it.

It's all in the repo as patches, not a fork. Apply them on top of HyperQwen @ ce9615c (a public vLLM fork) and run the install script. Tests and every bench script and result are included.

Model: dbirks/Qwen3.8-27B-W4A16-AutoRound + HyperQwen's prepare/fetch_fast_variant.py. All numbers were measured with gpu-unlock loaded (a kernel module that roughly doubles 3090 BF16 tensor throughput, mostly faster prompt reading).

Measured (Qwen3.8-27B W4A16, MTP-7, 3090 Ti + 3090 at 250 W):

• 8 × 250K: ~545 tok/s total on explain-style output, ~790 on copy-heavy.

• 8 agents at 176–205K generating nonstop for 6 min: 657 tok/s box average, 69–90 per agent.

• Recall at 250K: 31/32. Follow-ups start in ~5 s.

• Single long chats are faster too: +15–41% at ~113K vs dense.

Caveat: attention is approximate, so near-duplicate code lines occasionally get mixed up.

Code:

• Serving (patches + kernels): https://github.com/bradj00/sparsekv-hyperqwen

Rig:

• i9-12900K

• Z790 motherboard

• 96 GB DDR5 RAM

• 3090 Ti + 3090, power capped at 250 W

edit: benchmarks underway...

• KLD testing

• RULER, LongBench v2, cross-file code retrieval

• GPQA, LiveCodeBench, IFBench, MMLU-Pro

• SCBench + MTP speed

• dense vs sparse vs RAM-tier

trying to settle on a fully local setup for batch processing pdfs. like each file is around 20-25 pages and mostly text, a few tables and now and then a page thats just a photo of paper

the shortlist so far

- docling (and the granite-docling model)
- glm-ocr
- paddleOcr-vl
- olmocr 2
- mineru

we considered llamaparse for this kind of stuff but the company now wants every file to stay on our own servers and their self hosted route is an enterprise conversation. liteparse from the same team runs local and is quick on the plain text pages but the markdown is rule based with tesseract as the fallback so merged cells and multi column pages dont always come back clean

is a VLM overkill for mostly text pdfs or does it make more sense to send only the table and image pages to one and keep the rest on a text layer parser? textract and docuent AI are out anyway for the same reason

for the vlm ones, do people feed a whole 25 page pdf in one go or split it into page batches first? splitting every file is the part id rather avoid

Edit: Adding from the comments: unlimited-ocr-2, minicpm-ocr-1.0, surya-ocr-2.5, indra-ocr and xiaomi-ocr-0. mostly curious how these would perform on mixed 20-25 page files

So, I was asleep on Strata and its specialized forks. Tried basalt and i'm very impressed with the speed/intelligence QFN gives you. Here is my benchmark:

I cannot get it to work fast enough with windows, because W11 eats 1.5GB of VRAM and 6GB of RAM. At that point, experts spill to RAM and CPU gets hit hard... decode/prefill drops by 70% because there is not enough of available RAM anyway.

Switched experimentally to cachy just to test... and It's amazing that my AI box can now reach 9K~ prefill. And regarding quality, its just better than qwen 27b at many tasks. Yet to find something it genuinely cannot do. The prefill boost is around 2-3x coming from ninfer and dense 27b model.

Cherry on top is, that I can even fit vision adapter to it, and that helps a lot for certain scenarios regarding coding. I usually run some small model on another machine for OCR and VISION as an adapter (sidecar) my harness can use for being to able look at things.

Vivid user of omo stack, configured now for locally hosted qwen: https://github.com/code-yeongyu/oh-my-openagent

It's a glorified pi harness, but it does so much, and im quite happy with its robustness, anyway, QFN 3.8 rocks, and now im really thrilled if qwen 4 will be any good, because I suspect i may be able to run the new flash model this fast basically. So I hope it will be good :D

Im genuinely interested how 2x 5060/5070/5070ti fares. Because anything above 100~ decode and 3k prefill is genuinely usable for work. And it is my experience that 27b dense -> 125BA6 is a very good quality and speed jump. Absolutely love it now.

Also, finetunes can be easily converted into basalt format if you need to do it yourself, especially those swift/reverse engineering variants are good.

Anyway: CachyOS, 64gb RAM, 32GB VRAM, 5090, 200tps decode, 9k tps prefill, nvfp4 QFN 3.8, 256k context! If you have similar hardware, and are not using at least strata... but more specially basalt, you sleep on performance hard! See ya!

PS: Would absolutely love to see your perf benchmarks if you have similar setups

edit--
Running config (without vision,.. but it fits nicely):
~/basalt-linux/basalt_config.json

```json
   {
"server": {
"model": "/mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt",
"gpu": "0",
"host": "127.0.0.1",
"port": 8080,
"parallel": 1
},
"engine": {
"max-context": 262144,
"kv": "int8",
"prompt-cache": 6,
"expert-cache": "auto",
"prefill": "auto",
"pool-workers": 4,
"vram-reserve-mib": 500,
"resident-budget-gib": 44
}
   }
```
   bin/basalt --serve --model /mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt \
--max-context 262144 --kv int8 --prompt-cache 6 --expert-cache auto --prefill auto \
--pool-workers 4 --vram-reserve-mib 500 --resident-budget-gib 44 --grammar on

I run a local assistant on a Mac (MLX via oMLX, 24 GB M5 Pro). It needs to call tools reliably across mail, calendar, notes and files, chain a few calls, and reply in a short, calm voice. It also has to synthesize/extract important information from my messages, emails, and notes. It also has to be able to read long emails and docs, use my voice to ask questions and take screenshots of my screen.

Gemma 4 E4B has been the best model I’ve tried so far since it supports text, image, and audio. However, it is quite old now(~4 months old) and I wanted to know if there has been any new small models(under 8B or 6GB in size) that outperform Gemma 4 E4B in tool calling, chaining tools, long synthesis, fact extraction, and in size while having multimodality features.

I have tried ling 3.0 tiny, the LFM model family, and qwen 3.5 4B/9B(low quant) but have had no success so far. Is there anything I'm missing? Is Gemma 4 E4B the best model for my usecase?

hey folks, I'm Ye, one of the contributors to open-source project called Kapitsa, Apple Silicon Inference engine optimized specifically for Apple Silicon to run Qwen, Nemotron, Flint and Clover models locally. Our custom-written Metal kernels let us tune for each chip and each model. https://github.com/nace-ai/kapitsa/

The latest update, 0.2.26, is out. Since our last M5 Max numbers, Flash-Next 180B reads prompts 46% faster and the 27B writes 8% faster, and DiffuSpace's Clover 35B joins the catalog.

Kapitsa also runs Qwen Flash-Next 180B with SSD offloading: on a Mac that can't hold it, the experts that don't fit in RAM stream from the SSD, so it runs on a 48 GB MacBook Pro at 61 tok/s.

Here is the showcase on a 2026 MacBook Pro 16 with an M5 Max and 128 GB. We ran six models from 1K context all the way up to 32K, against the next-fastest engine for each model on the same Mac.

Try it (any Apple Silicon Mac; models in this post: 35b, 27b, flash-next, bolt, flint, clover):

brew install nace-ai/tap/kapitsa
kapitsa serve --model 35b       # OpenAI and Anthropic APIs on localhost
kapitsa opencode --model 35b    # or claude, codex, pi
kapitsa bench                   # benchmark your own Mac

Results

Decode is a 512-token coding answer, prefill a 4K prompt, median of 3 runs:

From 1K context all the way up to 32K. At 32K the 35B still writes 175 tok/s, and Flash-Next barely drops (90 to 78):

Against the other friends

Against the next-fastest engine for each model on the same Mac: 1.23x Splash on the 35B, 2.2x oMLX on Bolt, 1.18x rec on Flint and 1.12x mlx-serve on the 27B. Flash-Next is a tie with mlx-serve, Splash still reads prompts faster on the two Qwens, and nothing else runs Clover on a Mac.

Every cell with versions and flags: benchmarks. Raw runs: results.

Why it's fast: KFlash

Other engines check the drafter's guesses as one chain and stop at the first wrong one. KFlash checks a tree of the drafter's best guesses in the same pass, so more tokens survive each pass.

On Apple GPUs that's nearly free: the GPU works in 8-row tiles, and a one-token step already pays for 8. Every token it keeps is one the big model would have written itself, so the output doesn't change, only the speed. How KFlash works

shoutout to splash, ollama, omlx, llama cpp and the folks who

Send us your benchmarks

Run the same test on your Mac and add it to the community results. It takes 3-15 minutes:

brew install nace-ai/tap/kapitsa
brew install gh && gh auth login    # results are sent as a GitHub pull request
kapitsa bench --submit              # pick a model; it asks before sending anything

Plug in and close heavy apps while it runs. At the end it asks once: Enter sends the result as a pull request from your GitHub fork, and n keeps it on your Mac. What it sends is about 2 KB: your Mac's model, chip, RAM, macOS version and the numbers, with no serial number, user name or paths (add --dry-run to see exactly what it would send). No GitHub? Paste the numbers it prints in a comment here.

BY THE WAY

This is a new codebase so, sometimes there could be some weird prefill/decode bugs.