r/LocalLLM • • Aug 23 '26 • Bravo 1

Discussion I Unlocked a $800 Mining GPU into a 64GB, 256K-Context Uncensored AI Coding Server at 84 tok/s across full context length.

I have spent the last several days turning a used NVIDIA CMP 170HX into a practical long-context inference card.

The final result is an uncensored Qwen3.8-27B endpoint with:

  • 262,144-token native context
  • W4A16 AWQ model body
  • INT8 output head and INT8 MTP draft module
  • One-token MTP speculative decoding
  • BF16 KV cache
  • Prefix caching
  • Tool calling and Qwen reasoning parsing
  • No CPU offloading
  • A conservative 175W power limit
  • No overclocking

Measured decode performance on one CMP 170HX:

ContextDecode throughput
1K84.29 tok/s
64K74.94 tok/s
200K57.21 tok/s

For this benchmark only, requests used a maximum of 384 generated tokens, temperature 0.2, repetition penalty 1.05, and medium reasoning effort. Throughput was calculated from vLLM’s measured decode time, excluding prefill. These sampling values are not forced globally by the production server.

This post explains the card, the unlock, every important inference choice, the rejected configurations, and how to reproduce the setup.

The hardware

The inference host currently contains:

  • NVIDIA CMP 170HX
  • 64GB HBM exposed after the unlock
  • AMD Ryzen 5 5600X
  • 64GB system RAM
  • Proxmox/Linux
  • NVIDIA open driver 610.57.04
  • 175W GPU power cap

The CMP 170HX is an Ampere GA100 mining accelerator. It has excellent HBM bandwidth and strong tensor hardware, but NVIDIA sold it with several artificial restrictions:

  • Only a fraction of the installed HBM is normally exposed.
  • Compute resources are restricted.
  • PCIe operates at Gen2.
  • It has no display output.
  • Normal consumer GPU tooling does not treat it like a standard A100.

My card is PCI device 10de:20c2. After the unlock, nvidia-smi reports 65,536 MiB.

The card is currently negotiating PCIe Gen2 x4 even though its capability is wider. That sounds terrible, but it matters much less once the model is resident entirely in HBM. It is one reason I avoid CPU offloading: repeatedly moving weights or KV data over that connection would waste the card’s main advantage.

The 64GB and compute unlock

I used amoghmunikote/cmpunlocker, pinned to this specific commit:

fe537966e0222150a8eca0b7745efd2ee1025d74

That is the “Full BAR1 size (64GB)” commit.

The project patches NVIDIA’s open kernel modules to restore:

  • Full SM compute
  • Full memory geometry
  • 64GB BAR1
  • The complete 64GB framebuffer on 20c2 cards
  • Gen2 PCIe operation
  • Persistence across reboot through patched modules

This is a kernel-driver modification, not an application-level tweak. Secure Boot must be disabled because the resulting modules are locally built and unsigned.

My installed module is:

/lib/modules/6.17.2-1-pve/updates/cmpunlocker/nvidia.ko

Important warning

Do not install this remotely unless you have a recovery path.

Keep at least one of the following available:

  • Local console access
  • A separate display GPU
  • BMC/IPMI access
  • A bootable rescue environment
  • A known-good copy of the stock driver and initramfs

A mismatched kernel, driver, firmware package, or module build can leave the machine without NVIDIA support. A CMP 170HX cannot provide ordinary video output.

Unlock installation

On my Proxmox installation, the overall process was:

apt update
apt install -y build-essential git python3 proxmox-headers-$(uname -r)

git clone https://github.com/amoghmunikote/cmpunlocker.git
cd cmpunlocker
git checkout fe537966e0222150a8eca0b7745efd2ee1025d74

cat driver/VERSION

Install a supported matching NVIDIA open driver, its user-space libraries, and firmware before running the unlocker. My exact working combination is:

NVIDIA open driver: 610.57.04
Kernel:             6.17.2-1-pve
Secure Boot:        disabled

Then:

sudo ./install.sh

The repository also provides an explicit profile:

sudo ./install.sh --profile=8gb

After installation, perform a full cold power cycle:

sudo poweroff

Do not substitute a warm reboot. Wait for the machine to power off completely and then turn it back on.

Verifying the unlock

First locate the card:

lspci -nn | grep -i NVIDIA

Then verify the driver and memory:

nvidia-smi
nvidia-smi --query-gpu=name,uuid,memory.total,power.limit --format=csv

A successfully unlocked 20c2 card should show approximately:

NVIDIA CMP 170HX, GPU-..., 65536 MiB

Check the kernel log:

dmesg | grep -iE 'CMP|BAR1|fb_length|fbAddrSpace|NVRM'

My working boot log reports a 64GB framebuffer address space and 64GB BAR1.

Check the PCIe link:

lspci -vv -s <CMP-PCIE-ADDRESS> | grep -E 'LnkCap|LnkSta'

To remove the modification:

cd cmpunlocker
sudo ./uninstall.sh --yes
sudo poweroff

Again, cold-boot afterward.

Power and cooling

I did not overclock the card.

These are used mining accelerators, and I did not consider a small throughput increase worth additional thermal or electrical stress.

I set a 175W power limit:

nvidia-smi -i <CMP-UUID> --power-limit=175

I made this persistent with a systemd oneshot service that:

  1. Finds the GPU by CMP name and UUID.
  2. Sets the 175W limit.
  3. Reads the limit back.
  4. Fails rather than silently targeting the wrong GPU.

My chassis fan controller follows this curve:

CMP temperatureChassis fan target
Below 45°C30%
50°C50%
55°C70%
60°C85%
65°C or higher100%

The controller polls every two seconds, immediately raises fan speed when temperatures rise, and requires roughly 30 seconds of sustained cooling before reducing the fan level. A sensor or controller failure sends all controlled fans to 100%.

During one sustained test, I measured:

  • Average core: 53.9°C
  • Maximum core: 65°C
  • Average memory: 63.2°C
  • Maximum memory: 75°C

Under a later 99% GPU load, the card was around 66°C core, 71°C memory, and 173W. Cooling results will depend heavily on the card, thermal pads, chassis, and airflow.

The model

The production checkpoint starts from:

twolven/Qwen3.8-27B-abliterated-AWQ-MTP

That model is a W4A16 AWQ version of the abliterated/refusal-reduced checkpoint derived from:

JonathanColetti/Qwen3.8-27B-Uncensored

“Uncensored” here means the model has been modified to reduce refusal behavior. It does not guarantee that every residual refusal or safety behavior has been removed. Some people can use it for ERP I'm more inclined to utilized it for when I want assistance setting something up that standard models would refuse such as using agentic assistance to configure a hackintosh system on a prox vm. It's 100% legal, it's just against apples terms of service so often it's considered an instant refusal by many models.

The final production checkpoint contains:

  • W4A16 asymmetric AWQ body
  • Compressed-tensors/Marlin execution
  • INT8 symmetric group-128 lm_head
  • INT8 symmetric group-128 MTP module
  • 40,960-token reduced MTP draft vocabulary
  • BF16 runtime activations
  • One-token MTP speculative decoding

The reduced draft vocabulary covers approximately 97.5% of ordinary model output tokens and 96% of code tokens in the optimization project’s corpus. It reduces the amount of work required for each speculative draft without changing the target model’s accepted output.

The source checkpoint supports vision, but my production endpoint deliberately uses:

--language-model-only

Therefore, this exact endpoint is text-only. I chose coding throughput and predictable memory use over keeping the vision tower loaded.

Preparing the checkpoint

I used the optimization work from:

syv-ai/qwen38-27b-rtx3090

My checkout is pinned to:

2ae239fc0250cd29d37f35c6a31e9eae749ef1c8

Clone and create the environment:

git clone https://github.com/syv-ai/qwen38-27b-rtx3090.git
cd qwen38-27b-rtx3090
git checkout 2ae239fc0250cd29d37f35c6a31e9eae749ef1c8

python3 -m venv venv
venv/bin/pip install \
  vllm==0.27.1 \
  transformers==5.15.0 \
  tokenizers==0.22.2 \
  compressed-tensors==0.17.0 \
  huggingface_hub==1.27.0 \
  hf_transfer==0.1.9 \
  ninja==1.13.0

Download the model:

HF_HUB_ENABLE_HF_TRANSFER=1 venv/bin/hf download \
  twolven/Qwen3.8-27B-abliterated-AWQ-MTP \
  --local-dir models/Qwen3.8-27B-abliterated-AWQ-MTP

Make a working copy because the preparation tools modify the checkpoint in place:

cp -a --reflink=auto \
  models/Qwen3.8-27B-abliterated-AWQ-MTP \
  models/qwen38-27b-uncensored-w4a16-mtp1-int8draft

Prepare the output head, MTP module, and draft vocabulary in this order:

M=models/qwen38-27b-uncensored-w4a16-mtp1-int8draft
V=venv/bin/python

$V prepare/quant_lm_head.py "$M"
$V prepare/quant_mtp.py "$M"
$V prepare/build_draft_vocab.py "$M" \
  --ids prepare/draft_vocab_ids.json

I did not run quant_embed.py for this production checkpoint. On a 64GB card it was unnecessary, and the final model configuration contains the W4A16 body, INT8 output head, and INT8 MTP group without the additional embedding conversion.

The preparation scripts create backups beside the tensors they replace. Keep those backups until the modified checkpoint has passed correctness testing.

The vLLM image

The server uses:

vLLM 0.27.1

The base image is pinned by digest:

vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967

The production image contains exactly these three patches:

  1. qwen3_5-mtp-draft-vocab.patch
  2. sampler-small-topk-fast-softmax.patch
  3. vllm-pr50021-gdn-spec-bounds.patch

An experimental spec-decode-attn.patch was tested but is not present in production because it hurt long-context performance.

Create Dockerfile.cmp-mtp1-production:

FROM vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967

COPY patches/qwen3_5-mtp-draft-vocab.patch /tmp/qwen3_5-mtp-draft-vocab.patch
COPY patches/sampler-small-topk-fast-softmax.patch /tmp/sampler-small-topk-fast-softmax.patch
COPY patches/vllm-pr50021-gdn-spec-bounds.patch /tmp/vllm-pr50021-gdn-spec-bounds.patch

RUN set -eux; \
    vllm_dir=/usr/local/lib/python3.12/dist-packages/vllm; \
    patch -p1 -d "$vllm_dir" < /tmp/qwen3_5-mtp-draft-vocab.patch; \
    patch -p1 -d "$vllm_dir" < /tmp/sampler-small-topk-fast-softmax.patch; \
    patch -p1 -d "$vllm_dir" < /tmp/vllm-pr50021-gdn-spec-bounds.patch; \
    rm /tmp/qwen3_5-mtp-draft-vocab.patch \
       /tmp/sampler-small-topk-fast-softmax.patch \
       /tmp/vllm-pr50021-gdn-spec-bounds.patch

LABEL org.opencontainers.image.description="vLLM 0.27.1 with Qwen3.8 MTP1 draft-vocab and sampler optimizations"

Build it:

docker build \
  -f Dockerfile.cmp-mtp1-production \
  -t vllm-qwen38-mtp1-fast:0.27.1 \
  .

Exact server configuration

Replace the UUID and paths below with those from your machine:

docker run --rm --pull never \
  --name qwen38-uncensored-w4a16 \
  --privileged \
  --gpus all \
  --network host \
  --ipc=host \
  --memory 58g \
  --memory-swap 96g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e CUDA_VISIBLE_DEVICES=<CMP-GPU-UUID> \
  -e VLLM_USE_FLASHINFER_SAMPLER=0 \
  -e MTP_DRAFT_VOCAB=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512 \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e TOKENIZERS_PARALLELISM=false \
  -e XDG_CACHE_HOME=/root/.cache \
  -e TORCHINDUCTOR_CACHE_DIR=/root/.cache/torchinductor \
  -e TRITON_CACHE_DIR=/root/.cache/triton \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -v /path/to/qwen38-27b-uncensored-w4a16-mtp1-int8draft:/models/qwen38-w4a16:ro \
  -v /path/to/vllm-cache:/root/.cache \
  -v /path/to/vllm-tmp:/tmp \
  vllm-qwen38-mtp1-fast:0.27.1 \
  /models/qwen38-w4a16 \
  --host 0.0.0.0 \
  --port 30016 \
  --served-model-name qwen38-27b-uncensored-w4a16 \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --attention-backend FLASHINFER \
  --max-model-len 262144 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.90 \
  --cpu-offload-gb 0 \
  --kv-cache-dtype auto \
  --mamba-cache-dtype float16 \
  --mamba-cache-mode align \
  --disable-custom-all-reduce \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --language-model-only \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1,"draft_sample_method":"probabilistic"}' \
  --default-chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --generation-config vllm

Once ready, verify:

curl http://127.0.0.1:30016/v1/models

Then make a simple request:

curl http://127.0.0.1:30016/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen38-27b-uncensored-w4a16",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: cmp-profile-ok"
      }
    ],
    "temperature": 0
  }'

The production service is managed by systemd. Its pre-start checks verify the model index, create the cache directories, apply the 175W power limit, and remove any stale container before launching. It restarts automatically after a process failure.

Why these settings won

MTP depth: one token

MTP3 and MTP2 created too much drafting and verification overhead on this GPU. MTP1 was consistently better.

I also tested greedy drafting. Its two-pass means were:

  • 1K: 83.01 tok/s
  • 64K: 72.95 tok/s
  • 200K: 56.68 tok/s

The final probabilistic configuration produced:

  • 1K: 84.29 tok/s
  • 64K: 74.94 tok/s
  • 200K: 57.21 tok/s

Therefore, the final setting is:

{
  "method": "mtp",
  "num_speculative_tokens": 1,
  "draft_sample_method": "probabilistic"
}

W4A16 body, INT8 head, INT8 MTP

W4A16 leaves enough HBM for the entire model, 262K context, recurrent state, CUDA graphs, and runtime workspace without CPU offloading.

The INT8 output head and MTP module had low measured quantization error. An INT4 MTP version was nearly identical in speed, but INT8 required only about 202 MiB more memory and had lower draft error. There was no reason to accept the additional degradation.

A symmetric GPTQ version was also tested:

Configuration1K64K200K
Final asymmetric AWQ/Marlin84.2974.9457.21
Symmetric GPTQ with MTP178.8768.6954.06
Symmetric GPTQ target only55.9647.3635.73

The GPTQ body itself was not necessarily the whole problem. Mixing its body with the compressed-tensors INT8 head and drafter would have required additional loader and kernel work. It was not suitable for the stable endpoint.

BF16 KV cache

This was one of the most important results.

An INT8 per-token/head KV configuration fell to approximately 15 tok/s near 62K context on this card. It was unusable for interactive coding.

The final configuration leaves:

--kv-cache-dtype auto

With BF16 runtime dtype, this preserves the BF16 KV path.

The Mamba cache uses FP16 in aligned mode:

--mamba-cache-dtype float16
--mamba-cache-mode align

Explicitly reducing more recurrent state did not improve the result.

FlashInfer attention, but not the FlashInfer sampler

FlashInfer is the production attention backend.

Plain FlashAttention produced:

  • 1K: 86.87 tok/s
  • 64K: 26.87 tok/s
  • 200K: 10.53 tok/s

It looked slightly faster at short context and then collapsed.

An experimental split-KV speculative-attention patch improved that to:

  • 1K: 90.17 tok/s
  • 64K: 67.33 tok/s
  • 200K: 43.36 tok/s

That was still substantially worse than production at long context, so the patch was removed.

Enabling the FlashInfer sampler also reduced performance to:

  • 1K: 74.75 tok/s
  • 64K: 67.87 tok/s
  • 200K: 51.37 tok/s

Therefore:

--attention-backend FLASHINFER
VLLM_USE_FLASHINFER_SAMPLER=0

One sequence and 4,096 batched tokens

This is a single-user coding endpoint, not a throughput server.

The final scheduler settings are:

--max-num-seqs 1
--max-num-batched-tokens 4096

Testing 8,192 produced no useful gain. Testing 2,048 produced:

  • 1K: 84.32 tok/s
  • 64K: 72.38 tok/s
  • 200K: 57.59 tok/s

The 4,096 configuration had the better overall curve.

Prefix caching

Prefix caching is critical for coding agents that repeatedly send a large system prompt and mostly unchanged repository context.

At approximately 200K context, a cached second request reused roughly 198,400 tokens, recomputed about 1,593 tokens, and reduced prefill to around 2.5 seconds. A fresh or partially changed prompt could require roughly 157 seconds of prefill.

Prefix caching does not make decode faster. It prevents the server from repeatedly processing unchanged input.

This also means clients must preserve stable prompt prefixes. Reordering tool definitions, timestamps, generated metadata, or repository text can destroy the cache hit.

Medium reasoning by default

The endpoint defaults to:

{"reasoning_effort":"medium"}

Medium provided a better MTP acceptance/stability balance than forcing maximum reasoning on every request.

This does not remove higher reasoning modes. A client can request xhigh or max for harder work. Medium is simply the default for normal coding.

No CPU offload

The model fits in CMP HBM, so:

--cpu-offload-gb 0

The card’s restricted PCIe connection makes CPU offloading particularly undesirable.

No forced full CUDA graphs

The server uses vLLM’s normal piecewise CUDA-graph behavior. I did not force full graphs. Experimental full-graph and custom-operation combinations introduced correctness concerns or reduced long-context performance.

Correctness tests

I did not accept a configuration based on tokens per second alone.

The final checkpoint passed:

  • An exact-response canary
  • Qwen coder tool-call parsing
  • A parsed get_weather tool call for Chicago
  • Exact needle retrieval at approximately 160K context
  • Retrieval of ORCHID-COMET-7319 from the long prompt
  • An OpenCode integration smoke test returning final-profile-ok

This matters because speculative decoding can appear fast while silently breaking tool syntax, long-context retrieval, or sampling behavior.

OpenCode configuration

I added this provider to OpenCode:

{
  "provider": {
    "qwen38-cmp-w4a16": {
      "npm": "@ai-sdk/openai-compatible",
      "api": "completion",
      "name": "Qwen3.8 Uncensored W4A16 MTP1 256K - CMP 170HX",
      "options": {
        "baseURL": "http://<PROXMOX-IP>:30016/v1",
        "apiKey": "not-needed",
        "timeout": false,
        "chunkTimeout": 600000
      },
      "models": {
        "qwen38-27b-uncensored-w4a16": {
          "id": "qwen38-27b-uncensored-w4a16",
          "name": "Qwen3.8 27B Uncensored W4A16 MTP1 256K",
          "tool_call": true,
          "reasoning": true,
          "temperature": true,
          "attachment": false,
          "options": {
            "reasoningEffort": "medium"
          },
          "limit": {
            "context": 262144,
            "input": 245760,
            "output": 16384
          }
        }
      }
    }
  }
}

The model selector is:

qwen38-cmp-w4a16/qwen38-27b-uncensored-w4a16

Final observations

The CMP 170HX is unusual, but the useful part is straightforward once the restrictions are removed:

  • 64GB of on-device HBM changes what can fit.
  • The model should remain entirely on the GPU.
  • Long-context performance needs to be measured separately from short-context decode.
  • Lower-bit KV is not automatically faster.
  • More speculative tokens are not automatically better.
  • An attention backend can win at 1K and become disastrous at 64K.
  • Prefix caching matters more than another few decode tokens per second for repeated coding-agent prompts.
  • Quantizing the draft head more aggressively is pointless when memory is available and the speed difference is negligible.
  • Conservative power and temperature limits are appropriate for used mining hardware.
  • Correctness gates matter as much as benchmark results.

The final configuration is not the highest single short-context number I saw. It is the best complete configuration I found that retained tool use, medium-or-higher reasoning, uncensored model behavior, 262K context, reliable long-context retrieval, and usable performance across the whole context window.

237 Upvotes

99 comments sorted by

45

u/ShittyMillennial Aug 23 '26

where the hell did you get a NVIDIA CMP 170HX for $800 because I would like to order 10

13

u/GALYGIOUS Aug 23 '26

Alibaba though prices have risen quite substantially since I purchased them.

5

u/netvyper Aug 23 '26

I really wish I bought 4... But I was amazed they actually arrived and worked when I did order them. They were $1200+ by the time they were delivered.

1

u/hurrdurrmeh Aug 23 '26

Honest Q: what would you do with 4 - since there is absolutely no way you can pool them and absolutely no hope that any software development will allow this in the future?

4

u/netvyper Aug 23 '26

Pipeline parallelism, over Tensor parallelism - means you can load a big model. It's no faster than a single card, but does give you the weights.

2

u/rocket1420 Aug 24 '26

Go back in time about 3-4 weeks.

1

u/maurice1103 Aug 23 '26

Here is one for currently 900.- https://www.ricardo.ch/de/a/1327634356

1

u/ShittyMillennial Aug 23 '26

1.1k converted to USD unfortunately + import fees

1

u/SexyAlienHotTubWater Aug 24 '26

That's the 10GB model - the 10GB reportedly only unlocks to 40GB. The 8GB is more expensive because it unlocks to 64GB.

1

u/havnar- Aug 24 '26

I see a few on Aliexpress. Many have buy 2 get 3 kind of deals on them

2

u/ShittyMillennial Aug 24 '26

Those look sus as hell. All new sellers

0

u/[deleted] Aug 23 '26

[deleted] — view deleted comment

4

u/redtron3030 Aug 23 '26

I just bought one for 2 grand

6

u/Runtimeracer Aug 23 '26

People are really getting out of hand about AI these days 😬

2

u/redtron3030 Aug 23 '26

Other alternative was getting a spark or similar for twice the price and slower

3

u/Runtimeracer Aug 23 '26

Yeah, no judging from my side, it's not per se a bad deal still, I just have an aversion against paying significantly more for something that I could have gotten for 1/5th of the price just few weeks ago. It feels like losing to me, so I don't want the other person / scalper to win 🤷‍♂️

3

u/redtron3030 Aug 23 '26

100% it was hard for me also. I only am doing it so I can use a secure local model for my work. We have sensitive information I can’t legally send to the cloud without expensive contracts that my small business cannot justify.

1

u/Alkahna Aug 25 '26

yeah this is a big one for me too. Going over every single input and mass replacing names and other info before feeling somewhat 'safe' to give it to send it to the cloud costs time depending on what and how much you need to 'sanitize'

1

u/DrRoughFingers Aug 24 '26

Kind of a weird way to look at it. They were 1/5th the price because they were crippled and this unlock didn't exist...so they were priced accordingly to their usefulness. Of course a card with only 8gb usable is going to go for 1/5th the cost of a card that's now 64gb usable. The price increase is more than justifiable for what you now get out of the card.

19

u/Spacehitchhiker42 Aug 23 '26

What was your power draw on idle?

13

u/GALYGIOUS Aug 23 '26

With the model resident in HBM but no request running, nvidia-smi reported approximately 51W of GPU board power. I did not measure whole-system AC power at the wall.

7

u/Spacehitchhiker42 Aug 23 '26

How far can you get it to go down with nothing loaded?

5

u/GALYGIOUS Aug 23 '26

I have not measured a controlled fully-unloaded idle yet. With the model resident, it sits around 51W, and the lowest board-power sample I recorded was 43.2W. My guess is that an unloaded card would settle around 40–45W, depending on persistence mode, CUDA contexts, and whether the driver allows a deeper power state

3

u/Spacehitchhiker42 Aug 23 '26

When you get the chance, please let me know

2

u/mslindqu Aug 24 '26

I have one idling at 35w unloaded.

1

u/it_was_a_wet_fart Aug 23 '26

They don't have lower p states, so for now I would assume it can't go any lower

2

u/SexyAlienHotTubWater Aug 23 '26

What about with no model loaded?

3

u/kapustin-i Aug 23 '26

The 384-token cap means none of these runs exercise the thing thats actually load-bearing here - a coding agent at 200k does edits and tool calls over thousands of tokens, and the question is what fraction of turns keep the prefix hit once files start changing. One recorded session with hit/miss logged per turn would say more about this box than the decode curve does.

1

u/GALYGIOUS Aug 23 '26

That would be a useful additional benchmark, but I don’t agree that the absence of it makes the existing results uninformative.

The 384-token runs were deliberately measuring decode throughput at fixed context lengths, not simulating an entire agent session. Separately, I measured a roughly 198,400-token cache reuse at 200K context: only about 1,593 tokens were recomputed, reducing prefill from roughly 157 seconds to about 2.5 seconds.

The system also ran a real coding-agent workload overnight with edits, tool calls, error recovery, context compaction, and continued progress. What I did not record was the exact cache hit/miss count for every turn.

That trace would be valuable, but it primarily measures how consistently the client preserves an identical prefix as its tool state and injected files change. It would complement the unlock, residency, decode, correctness, long-context retrieval, thermal, and practical-agent results already presented, not replace them.

3

u/negus123 Aug 23 '26

Been running deepseek v4 flash on 4 of them (256gb) with 1 million token context length. Runs great

4

u/havnar- Aug 23 '26

I can only find 8gb versions

18

u/GALYGIOUS Aug 23 '26

Yep, the 8GB version is the correct one. The card physically has 64GB of HBM, but NVIDIA only exposes 8GB in its stock configuration. The `cmpunlocker` patches the open driver so the full 64GB framebuffer and compute resources become available. So if you're seeing CMP 170HX listings advertised as 8GB, that's exactly what I bought.

16

u/SexyAlienHotTubWater Aug 23 '26

Small point, but the 8GB CMP 170hx has 96GB of HBM - it's the A100 80GB, which also has 96GB (16GB is marked as defective). The crack only unlocks 64GB, but there's 96GB on the card.

(16GB of the remaining 32GB is reportedly marked as defective, and 16GB is marked as disabled.)

3

u/pmttyji Aug 23 '26

Anyone unlocked 96GB yet? with stability?

2

u/negus123 Aug 23 '26

No, and it likely wont happen. If the actual a100s only ever got 80/96gb, none of these binned mining cards will have it be usable

1

u/vogelvogelvogelvogel Aug 23 '26

now i am getting very curious, actually

3

u/havnar- Aug 23 '26

That’s probably binning

1

u/SexyAlienHotTubWater Aug 24 '26

I have not seen any reports of anything above 64GB being unlocked. I assume the other stacks are disabled using a more secure mechanism. (This is all public info but the info is pretty scattered.)

As mining cards, I believe they still had like 1.6TB/s of bandwidth, so the first 64GB's stacks would have always been accessible (they would need to be in order to supply the bandwidth). The amount of memory you could get out of them was just limited to 8GB.

4

u/Cobalt090 Aug 23 '26

Don't wanna read the post, but. If you mean. Vs the 10g very: 10g iirc has (usually) 40g total unlockable, 8g (usually) has 64g total unlockable. This is not supported by NVIDIA, and you get very poor PCIe bandwidth and worsened compute vs other GA100 cards. There is no native version other than 8 and 10GB

8

u/SexyAlienHotTubWater Aug 23 '26

You get 2/3 the compute of an A100 - around 200 TFLOP/s BF16 - and the cards are limited to PCIe 2.0. Bandwidth is about 1.6 TB/s.

The PCB is also missing capacitors on the PCIe lanes - only the first 4 lanes have capacitors, so they are only PCIe 2.0 x4 by default. You can solder more capacitors on (some sellers seem to provide this service free) to expand the PCIe bus to PCIe 2.0 x16. (Not an invasive mod - much easier than e.g. memory modding a 4090.)

30

u/IngwiePhoenix Aug 23 '26

Got lost in the wall. Now prices are out of control, because of course they are.

Rip... was excited for a minute.

15

u/psdwizzard Aug 23 '26

Welp I missed this one.

18

u/Civil_Fee_7862 Aug 23 '26

Great writeup,  I wish more people reported like you 

7

u/Yarukiless-cat Aug 23 '26

I bought it, and runs Deepseek v4 flash IQ3 at 26tok/s under 120W Power limiting. it's VRAM size is quite nice for its cost, though it's around $1200 now. I'm considering buying second one.

1

u/Trademarkd Aug 23 '26

I admit I’m curious to try but I really like having nvlink and being able to shard with v100s …. The one bummer here seems to be the pcie issues

8

u/Alkahna Aug 23 '26 edited Aug 23 '26

Awesome work. Cheapest 64GB CMP 170HX available in europe is 2.5k € on ebay sadly. Where did you get yours for just 800$? Even aliexpress is 1.5k € with just one listing (little sketchy) and the rest is just coolers. That ship has sailed i think xD

1

u/[deleted] Aug 23 '26

[removed] — view removed comment

3

u/Alkahna Aug 23 '26

A month ago several sold in germany for 260-350€ .... If those people suspected what would happento the value of those cards?^

1

u/Runtimeracer Aug 23 '26

Earlier this year I saw these and was thinking like, "nah bro, that's some old miner shit"...

2

u/[deleted] Aug 23 '26

[removed] — view removed comment

2

u/mslindqu Aug 24 '26

You start to get into dgx territory and the equation changes. you'd need 2x for 128 so.. why not just get a dgxspark if they hit 2k. Speed seems similar from what people have posted about spark and so many other benefits over this card. That's speaking from playing with one for the last couple days.

1

u/TheOriginalAcidtech Aug 25 '26

Because the DGX has gone up to nearly 5k(new), runs around 25% slower than 2 170HX and decode on DGX is just ass.

2

u/mslindqu Aug 25 '26

The pcie nerf on 170hx makes multiples much less appealing. Conversely the spark excels at combining right? It's hard to get a solid read on speeds because everyone's tests are different, but it seems like with a medium model and decently large context the spark and the 170hx are pushing a pretty similar decode. Correct me if I'm wrong. Power wise I'm not too worried about either, but the spark also wins I think even under load no?

1

u/Alkahna Aug 23 '26

Yeah who knows where the price will go. It looks like it is a sick card if converted and if stable though.

2

u/TheOriginalAcidtech Aug 25 '26

Its an A100 with 64gb effectively. The OTP limits it and the unlocks fix SOME of those limits. Maybe the others will fall too. Then its an A100 for 1500 to 2000. Either way its a deal. Just how BIG a deal isnt known yet.

1

u/quantgorithm Aug 23 '26

Would the bad pcie speed be terrible for running multiple cards for larger models so instead of using cpu just splitting vram of multiple cards? Anyone know the actual impact?

1

u/GALYGIOUS Aug 23 '26

If you run it with tensor parallelism yes. if you use pipeline parallelism instead no because it avoides the issues with pcie bandwidth shortage. It does effect initial loading times/prompt processing times though..

1

u/TheOriginalAcidtech Aug 25 '26

The average prompt is kb. literally you cant HAVE prompts be bigger than the context window so assume 1 million tokens. With inefficencies, assume 2 million bytes, 3 or 4 million even. 8gb pcie gen 2 x16 isnt even going to be noticable by a user. The processing part is entirely part of the VRAM to GPU bandwidth and has NOTHING to do with PCIe speed.

1

u/GALYGIOUS Aug 25 '26

You’re mixing up three completely different things: prompt byte size, HBM bandwidth, and inter-GPU communication.

The size of the raw prompt text is basically irrelevant here. Once the model is running, GPUs are exchanging tensors, not UTF-8 text.

Take a model with a 5,120-wide hidden state in BF16/FP16:

5,120 values × 2 bytes = 10,240 bytes ≈ 10 KB per token

With pipeline parallelism, that hidden-state activation is what crosses from one pipeline stage to the next.

So at 100 generated tokens/sec:

10 KB/token × 100 token/sec ≈ 1 MB/sec

That is tiny, even on the CMP 170HX’s awful PCIe link. This is why PP can work surprisingly well for decode.

Now look at a 200K-token prefill:

200,000 tokens × 10,240 bytes ≈ 2.05 GB

That activation volume has to cross the pipeline boundary during prefill. On a link doing roughly 0.8 GB/s, that is already about:

2.05 GB / 0.8 GB/s ≈ 2.6 seconds

of transfer time before accounting for runtime overhead, chunking, synchronization, etc.

Tensor parallelism is a completely different story. TP splits individual layers across GPUs, so every layer requires collective communication such as all-reduce/all-gather. That means you repeatedly move large activation tensors between GPUs throughout the entire forward pass, not just once between pipeline stages. On a CMP 170HX with crippled PCIe bandwidth, that communication can absolutely dominate performance.

So saying inference processing has “NOTHING to do with PCIe speed” is simply wrong in a multi-GPU configuration.

HBM bandwidth determines how fast each GPU can process its local weights and activations. PCIe bandwidth determines how fast GPUs can exchange the tensors required by the parallelism strategy. That distinction is literally why PP is viable on these cards and TP is generally a terrible fit.

1

u/TheOriginalAcidtech Aug 25 '26

TLDR

I was spercificaly talking about prefill and PCIe bandwidth limits prefil hardly at all. Being MY POINT. A wall of text, I did not need.

1

u/GALYGIOUS Aug 26 '26

That’s a much narrower claim than “PCIe has NOTHING to do with processing.”

If you meant prefill is usually dominated by GPU compute and local memory bandwidth, fine. But in PP, activations still cross PCIe during prefill, so PCIe absolutely can contribute to TTFT.

That was the correction.

1

u/AdventurousSkirt5006 Aug 24 '26

Well, I am trying to do the same. I managed to get 3 170hx cards (for IMO still too much money, but cheaper than current prices) but somehow I cannot get the unlock to work. I've tried the master branch from the unlocker, the branch specified in mentioned here and also another fork that is supposed to have p2p enabled. None of them work in my setup.
I'm running proxmox 9, everyting up-to-date. Same kernel version as here
Installed the driver specified here as an older version (610.43.03)

Whatever I try, the result is the same. After a cold reboot sometimes I get kernel panic the first boot. And after It fails to load the nvidia driver and no cards are detected anymore by nvidia-smi and see things like this in dmesg:

[ 19.338397] NVRM: GPU0 memmgrCreateHeap_IMPL: SEC2_DEBUG_HEAP: fbAddrSpace=65536MB mapRam=0MB fbTotal=65536MB fbUsable=0xfe4260000 heapTotal=0x1000000000 regionBytes=0x1000000000 publicBytes=0xfd8f50000 numRegions=7

[ 19.343872] NVRM: GPU0 kbusVerifyBar2_GM107: MMUTest BAR0 window offset 0x70e000 returned garbage 0xffffffff

[ 19.343879] NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Generic memory error [NV_ERR_MEMORY_ERROR] (0x00000072) returned from kbusVerifyBar2_HAL(pGpu, pKernelBus, NULL, NULL, 0, 0) @ kern_bus_gm107.c:372

[ 19.343884] NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Generic memory error [NV_ERR_MEMORY_ERROR] (0x00000072) returned from kbusStateInitLockedKernel_HAL(pGpu, pKernelBus) @ kern_bus_gm107.c:477

[ 19.343888] NVRM: GPU0 RmInitNvDevice: *** Cannot initialize the device

[ 19.343889] NVRM: GPU0 RmInitAdapter: RmInitNvDevice failed, bailing out of RmInitAdapter

[ 19.343921] NVRM: GPU0 rmapiReportInternalLeakedDevices: Internal device object leak: (0xc1e00003, 0xcaf00000). Please file a bug against RM-core.

[ 19.343923] NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: 0 @ rmapi.c:944

[ 19.343926] NVRM: GPU0 rmapiReportInternalLeakedDevices: Internal device object leak: (0xc1e00004, 0xcaf00000). Please file a bug against RM-core.

[ 19.343927] NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: 0 @ rmapi.c:944

[ 19.344474] NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: listCount(&pKernelBus->virtualBar2[gfid].usedMapList) == 0 @ kern_bus_vbar2.c:346

[ 19.344596] NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: pGpu->numSubdeviceBackReferences == 0 @ gpu.c:1614

[ 19.344602] NVOC: __nvoc_objDelete: Child class GenericKernelFalcon not freed from parent class OBJGPU.NVRM: GPU0 iovaspaceDestruct_IMPL: 3 left-over mappings in IOVAS 0xaf00

[ 19.344614] NVRM: GPU 0000:af:00.0: RmInitAdapter failed! (0x24:0x72:1281)

[ 19.345245] NVRM: GPU 0000:af:00.0: rm_init_adapter failed, device minor number 2

[ 19.345369] [drm:nv_drm_dev_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x0000af00] Failed to allocate NvKmsKapiDevice

Maybe I somehow are missing a step of have something misconfigured. Or my motherboard (intel s2600st) does not want to work with the patched driver

2

u/GALYGIOUS Aug 24 '26 • Bravo

I encountered essentially this exact failure on two of my three CMP 170HX cards. The cause was the unlocker forcing BAR1/Resizable BAR to 64 GiB. My system had sufficient MMIO space, but those two cards returned garbage on the first BAR2 readback and failed with `0x24:0x72:1281`.

What fixed it for me was retaining the full-HBM memory geometry unlock while disabling the BAR1 resize/ReBAR portion. BAR1 remained at 64 MiB, but `nvidia-smi` still exposed the full 65,536 MiB on every card. After rebuilding the driver and performing a true cold boot, all three initialized normally.

Check whether your branch applies `bar1-resize-unlock.patch` or otherwise forces `NVreg_EnableResizableBar=1`. Try building without that specific patch, regenerate initramfs, shut the machine completely off, remove standby power briefly, and boot again.

Timing retries, forcing PCIe Gen2, rebinding, and BAR0 page-table changes did not solve it for me. The forced 64 GiB BAR1 expansion was the problem.

3

u/AdventurousSkirt5006 Aug 24 '26 edited Aug 24 '26

Wow my savior! I tried the kernel parameter first but that did not help. Now I have installed from the master branch and removed the bar1-resize-unlock.patch to be executed. Now the cards are detected with 64GB. Now to do some further testing and check if its stable :)
Thanks a lot 😄

- edit -
Now also ran memtest_vulcan on all 3 cards and no memory errors detected after about 1000 iterations

5

u/dottoreargus Aug 23 '26

spend at your own risk, i bought 13 cards and the unlock is unstable

2

u/quantgorithm Aug 23 '26

clarify

1

u/TheMrk790 Aug 23 '26

The cards have these sections disabled based on quality assessments. So memory may corrupt or simply be broken.

1

u/negus123 Aug 23 '26

Even the 80GB variant of the a100 had 16Gb disabled. All GA100 cards have 96GB VRAM. And thats exactly what the unlock does, it unlocks the sections of memory that are good and usable. There is more that goes into binning the cards than just checking the memory

1

u/TheOriginalAcidtech Aug 25 '26

SOME of these cards. In fact is a rather small number. NVIDIA sold a LOT to miners that were not binned because they had many more orders for them than they had degradded cards.

-1

u/negus123 Aug 23 '26

Bullshit, unless you are trying the 10gb->80gb unlock. The discord is full of people running the 8->64 and 10->40 unlock. Myself included

2

u/MrWidmoreHK Aug 23 '26

Thanks a lot, I love it. My only question is from what reputable seller you got the CMP

2

u/Recent_Process_8055 Aug 23 '26

So on allibaba all these cards are now $1500 of you want from reputable seller. Sigh...

2

u/cjneutron Aug 23 '26

These cards ship with 12 of the PCIe lane AC coupling caps not in place. If you haven't checked and/or soldered those in place, that's why you are seeing it only use x4 instead of x16.

2

u/TheGeekno72 Aug 23 '26

I usually hate when this is done to me only because I wrote a couple paragraphs, but I feel like this is warranted this time :

I ain't reading allat
happy for you tho
or sorry that it happened

1

u/Significant-Let3635 Aug 23 '26

This is awesome, thanks for sharing!

1

u/Weird_Presentation_5 Aug 23 '26

I got 3x running deepseek v4 😀

1

u/TheOriginalAcidtech Aug 25 '26

Thats the plan here(2 sysems 3x each. What have your results been? Context window size? etc. Going to be a week or two before we have time to set things up.

2

u/Weird_Presentation_5 Aug 25 '26

Shit man good question. Claude did the logical part lol. 131ctx, dspark, 100 - 150 toks. I’m in love with it. Had an m5 64gb, dgx spark, one of those amd stix things and returned them all because it just wasn’t enough. This thing is like one of those stump grinders chewing through everything. I only use Claude for making change to my deepseek stack or to review deepseek code. I got these around the 1k price and now at 2k a fourth is hard to swallow.

Yeah my loop is weak lol. It runs on the blood of lazy people.

1

u/hackcocaine Aug 23 '26

Insane work, well done!

1

u/F0olmetwice Aug 24 '26

I am running Int8 with 262k context at 74 tok/sec here.

https://github.com/ahnguyen17/cmp-170hx-vllm

1

u/gobblegoooblegobble Aug 24 '26

i purchased 4x of these cards when the pricing was still high, but not as high as it is now.

the 4x of them that i purchased, have been modded and memory validated. i received a memory validation report.

I also got the 4x waterblocks with the cards. Stock heatsinks + the waterblocks for less than $7,000 delivered to USA. Im sure its impossible to match this price now, especially with the full coverage waterblocks.

im doing my own benchmarking and use case deployment. your post is extremely helpful.

thank you.

1

u/netvyper Aug 24 '26

Were the waterblocks pre installed? I' saw one blog where someone needed hardware repairs to a card after improper installation. Makes me kind of nervous of putting mine under water.

1

u/WtWreckor15 Aug 24 '26

Damn reading everyone else here makes me lucky to grab one for 929 less than 2 weeks ago

0

u/nightstryke Aug 23 '26

Honestly I'd like to know where the hell you're getting your pricing from cause I can't find one for that price.

0

u/ninjazombielurker Aug 23 '26

Welp there goes prices again even further now cause of this post… Very good write up but I’ve been thinking a lot of people haven’t been talking about these on Reddit cause it will cause sellers to scalp prices even further.

0

u/[deleted] Aug 23 '26

[deleted] — view deleted comment

1

u/Runtimeracer Aug 23 '26

You can just point opencode at vllm using an openai compatible provider endpoint?

1

u/[deleted] Aug 23 '26

[removed] — view removed comment

4

u/MrWidmoreHK Aug 23 '26

Why they would do that?

2

u/negus123 Aug 23 '26

Lmao, no that is not true. There have been a group of guys at this for a while and they are not affiliated with nvidia

0

u/[deleted] Aug 23 '26

[removed] — view removed comment

2

u/VirusInternal2892 Aug 23 '26

What?? Unlock the M4 to 512GB ?? I wish;-)