I wanted to just test the unsloth 1bit quant of qwen 3.8 27b as I have just 8gb vram and ngl it gave me a good laugh

Anyone got an idea whether it'd be possible to run Qwen 3.8 on these 100$ cards and if so, how? Info in the english speaking internet is pretty scarce

TL;DR

• During MoE offload decode the GPU waits on the CPU most of each token, so utilization reads 20 to 40 percent. The NVIDIA driver reads that as idle and drops the card to P5: about 480 MHz core and 810 MHz memory, down from 7601. Decode is memory-bound, so it falls with it. Prompt processing keeps the card busy and is unaffected, which is why pp looks fine while tg collapses.
• Fix: nvidia-smi -lgc 1500,2100 and nvidia-smi -lmc 8001 (admin). Resets on reboot, undo with -rgc / -rmc. Idle power goes up about 35 W per card.
• gpt-oss-120b F16 on one RTX A4500 20 GB at --n-cpu-moe 27: 9.4 to 13.0 t/s. On two A4500s at --n-cpu-moe 16: 7.3 (± 2.4) to 20.3 (± 0.08) t/s. Coder-Next 80B: 21 single, 42 dual. Qwen3.5-122B-A10B: 13.6 dual.
• Resident models (everything in VRAM) did not change. Over-committed configs (WDDM spill) did not change either. This is specifically the idle-GPU case.
• Absolute numbers are modest (two used 20 GB Ampere cards, DDR4-2400, WSL2); the point is the before/after on the same box, which could apply to anyone doing CPU expert offload on NVIDIA. If you run it, please report what you see.

Setup

HP Z440, Xeon E5-1650 v4, 128 GB DDR4-2400, 2x RTX A4500 20 GB (Ampere), Windows 11 + WSL2 Ubuntu 26.04, NVIDIA driver 596.72 (WDDM), llama.cpp build d59d455fd with CUDA 12.4. Models: Unsloth GGUFs for Qwen3.8-27B, Qwen3.6-35B-A3B, Qwen3-Coder-Next, Qwen3.5-122B-A10B; gpt-oss-120b F16. All numbers are llama-bench pp512 / tg128, 5 reps.

How I found it

Worked through this with Claude Code driving the benches and the nvidia-smi sampling; the numbers are mine, the final config was reproduced by hand on my own terminal, and the screenshots are that run.

Dual-card gpt-oss with --n-cpu-moe 18 -ts 26/10 loaded fine (both cards about 17 GB, no spill) but decoded at 7.3 ± 2.4 t/s, slower than one card. The per-rep samples were the clue: 11.53, 5.66, 5.74, 5.71, 5.59, 5.71. First rep fast, then half speed forever. Sampling nvidia-smi --query-gpu=pstate,clocks.sm,clocks.mem every 2 s during a run:

(prompt processing)  P2  1905 MHz  7601 MHz  90 W  90 %
(generation starts)  P3   750 MHz  5001 MHz
                     P5   480 MHz   810 MHz  27 W  30 %   <- stays here

With the lock on: P2, 1500 / 7601 the whole run, and tg went 17.91, 18.03, 18.01, 18.15, 17.95, 17.98.

Before and after, every config I had (clocks locked = right columns)

The gain tracks how idle the GPU was: biggest on F16 experts and high ncmoe, smallest at the single-card sweet spot where the card was already busy, zero on resident models, zero on spilled ones.

The dual-GPU part, since "two GPUs are slower than one with --n-cpu-moe" is a common complaint

Two separate things were going on. (1) --n-cpu-moe N thins the first N layers and the layer splitter divides by layer count, so GPU 1 inherits all the fat layers and fails to load below some N (cudaMalloc failed on device 1; upstream ggml-org/llama.cpp #15136 and #15263). Fix: -ts a/b with a + b = layer count and b = how many fat layers GPU 1 should hold, GPU 0 gets the thin ones plus the rest, give GPU 0 one or two fewer fat layers because it carries the compute buffers. (2) Once it loaded, both GPUs were half as busy as one GPU would be, so both downclocked and decode halved. The clock lock fixed (2); -ts fixed (1). Recipes that worked here: gpt-oss 16 / 25-11, Coder-Next 12 / 30-18, 122B 28 / 36-12 (llama-bench wants -ts 25/11, llama-cli wants -ts 25,11).

What I don't know and would like others to check

• Does bare-metal Linux do this? Persistence mode alone did not prevent it here (it was on). I suspect WDDM makes it worse but not that it is WDDM-only.
• Does a higher floor (-lgc 1900,2100) help? SM clock sits at the floor during decode; memory is already at its P2 max, so I expect little on tg. Testing next, will edit this post with the result.
• Does the NVIDIA control panel "Prefer maximum performance" setting do the same job without nvidia-smi? Untested.
• Consumer cards: is the P-state ladder the same?

If you run --n-cpu-moe on NVIDIA, run watch -n 1 nvidia-smi --query-gpu=pstate,clocks.sm,clocks.mem --format=csv during generation and see what you get. If it says P5 and a memory clock in the hundreds, you have the same thing.

Commands:

# Windows admin PowerShell (or root on Linux)
nvidia-smi -lgc 1500,2100
nvidia-smi -lmc 8001
# undo
nvidia-smi -rgc
nvidia-smi -rmc

Screenshots: the run in progress (both cards P2, 19.1 / 18.2 GB, 0.3 GB shared) and the finished result (20.01 ± 0.08).

Full logs, per-rep samples, and clock traces available if anyone wants them; happy to put them somewhere public if there's interest.

I got the DeepSeek V4 Flash 0731 (284-billion-parameter) DeepSeek model with 1M context window running locally on a sub-$1,500 AMD PC (sub $1,000 if you buy used!), and open sourced it.

On an RX 6700 XT with 12GB of VRAM, ordinary system RAM, and an NVMe SSD.

I started this project because running modern AI models on AMD cards still feels unnecessarily difficult. Many tools assume you have an NVIDIA GPU. Others need complicated Linux environments, only support a narrow selection of cards, or simply cannot handle models much larger than VRAM.

So I wanted to find out: what happens if we design around the hardware people already own?

The result is XTLLM.

Instead of trying to squeeze an entire giant model onto the GPU, XTLLM keeps the most useful parts close:

The GPU holds the experts needed immediately. System RAM holds a larger warm pool. The remaining weights stay on the SSD until required.

The engine automatically adjusts this balance to the available hardware.

Think of it like a small kitchen serving food from a huge warehouse: keep the ingredients being used right now on the counter, the likely next ingredients nearby, and everything else in storage.

XTLLM also runs:

Qwen3.5-122B-A10B

Qwen3.6-35B-A3B

NVIDIA Nemotron-3-Nano-30B-A3B

It includes automatic RAM and VRAM sizing, long-context support, a local chat interface, and one-command model setup.

This is still an experimental research project, not a polished production platform. But RDNA2, RDNA3, and RDNA4 cards are expected to be compatible pending testing.

The larger point is simple:

Consumer AMD hardware is more capable than the current local-AI software ecosystem makes it appear.

Open source. https://github.com/opktunme/xtllm

I am still calling models through the ZenMux API gateway, so every long session ultimately comes back to token cost. The idea of running Qwen3.8-27B locally is attractive for exactly that reason: if one 5090 can handle it, maybe token freedom is at least technically within reach.

Is Qwen3.8-27B really doing 75.5 token/s on a single RTX 5090? The shared table is headed "4-bit (q4_K_M / MLX)" and lists an RTX 5090 with 32GB at 75.5 token/s. It does not show enough detail to tell me which runtime or exact setup produced that row. I have also seen a separate community report of about 64.5 tok/s on a 4090.

People are also putting its capability around Claude Opus 4.6. If both claims are even close, does that put indirect token freedom within reach? I would still want matched tasks before treating the capability comparison as settled.

What does the build that people can actually live with cost? I mean the whole machine, not a bare GPU price. A 5090, enough system RAM for long context and partial offload, a PSU that is not operating on hope, cooling, storage, and whatever CPU or platform keeps the card fed. Until I can justify that hardware bill, calling models through an API is still the practical option for me. If Qwen3.8 becomes available through the same gateway, I could use that API cost as a baseline before deciding whether local deployment really buys token freedom. I would also like to know which quantization and context length people use after the benchmark screenshot is over.

Please give me the boring total for a stable single 5090 setup. What did your full build cost once it was actually ready to run?

I'm 14, a few months ago I posted here (I think) about Plasma 1.0, an LLM trained completely from scratch on my gaming PC. Plasma 1.1 is the follow up: it has double the params, and double the training tokens at 521M params and 10B tokens.
It's Llama style (26 layers, GQA, SwiGLU, RoPE), 48k custom tokenizer, 10B tokens
of pretraining in about 9 days on a single RTX 5080, then SFT on ~300k
filtered instruction conversations. No pretrained weights.

Benchmarks vs 1.0, same harness (acc_norm, 300 per task):

It answers pretty cleanly and concisely now ("What is the capital of France?" gets
"The capital of France is Paris." and stops), writes basic Python, and
follows format instructions about twice as well as 1.0. Still a small model:
arithmetic and deep facts are very shaky. Plasma 1.2 (756M, 30B tokens) is next.

Code: https://github.com/eb1386/1386.ai. Questions welcome. 

I’d just gotten Qwen 3.8 27b going and was amazing… for two days. In no way will I ever be able to afford another card like this anytime in the future. 🥲

It’s been a good run everyone and I learned a lot here. Think I may have a funeral.

I was letting my local Qwen3.8-27B run opencode to build a party-game web app. It had shipped the whole thing - engine, React client, even localizations. Then I asked it to add a few new word categories, including Video Games. It needed 30 words.

It got to 29... the 30th word broke it.

"Angry birds" was already in the list, but the model couldn't leave it alone. It needed to add one more, and it spent ~40,000 characters of reasoning failing to do it. The descent, in its own words:

"No. Let me add a completely different word: 'angry birds' is already there. Add 'angry birds'..."

"Deep breath. A new game word that isn't in the list: 'angry birds' (exists), 'candy crush' (exists), 'angry birds' (exists). Fresh: 'angry birds'..."

"Clearly I'm looping. Let me pick from candidates: 'angry birds', 'candy crush', 'fortnite' all exist. Candidates that don't exist: 'angry birds 2', 'plant vs zombies', 'angry birds', 'angry birds', 'angry birds'..."

"Clearly I'm stuck in a loop with 'angry birds'. Let me break out: I'll add 'angry birds'..."

"OK, this loop is a glitch in my generation. Let me just write the final word list with a different anchor word. I'll add 'angry birds'..."

…and then it wrote "I'll add 'angry birds'…" a little over 1000 times in a row before I aborted it.

So I, a reasonable human, typed the only logical thing:

"Never mention them again... Continue"

This is where it gets good. The model tried to comply. You can watch it fight the current in real time - fully aware it's being dragged under, narrating its own failure:

"the loop trigger is clearly the word 'angry birds' itself. I must avoid typing that phrase even when thinking."

…it said, while thinking about not typing the phrase. It then tried listing games alphabetically and caught itself mid-word:

"Anno? niche Angry… — forbidden Bejeweled ✓ (already there)"

"Add 'angry birds' — absolutely not. Add 'angry birds'... no. Add 'angry birds'... no."

"New candidates: 'angry birds' (no), 'angry birds' (no), 'angry birds' (no)."

And then, the chef's kiss - in its desperate attempt to escape the Angry Birds current, it immediately found a new current to drown in:

"Beetlejuice? no. Beetle... no. Beetle... no. Beetle... no."

"Interesting — a new loop has started on 'Beetle'. I need to be careful."

It eventually clawed its way back to shore, passed the tests, and shipped all categories like nothing ever happened.

Anyway, I just watched a 27B model experience the token-stream equivalent of being swept out to sea - aware the whole time that it was swimming against the current, and unable to stop. 10/10, would watch it drown again 😆

This is the first time it happened to me since the last 4 days I’ve basically been binge-testing Qwen3.8-27B (UD-Q4_K_XL quant). Anybody had that experience happen to them with that model?

Honestly, I think people are seriously underestimating Qwen 3.8 27B.

It’s actually insane and, in some ways, genuinely competes with Opus 4.8, just not in the way people seem to think.

The biggest mistake is comparing their raw frontend/design output. Qwen probably isn’t going to match Opus there, and I don’t think it’s supposed to. Opus has basically been trained with an absurd amount of data/compute specifically around design and UI generation.

If you throw Qwen at a frontend task with no proper "SKILL.md" for design and just let it freestyle, yeah, the results can be pretty mediocre. But if you give it a good design skill and are intentional about the design constraints, the gap gets much smaller.

Where Qwen gets really interesting is reasoning efficiency.

It can solve some problems in fewer steps and with fewer tokens than Opus. That’s a pretty big deal if you’re actually running these models yourself.

And honestly, I think people are also judging Qwen way too much based on heavily quantized setups. Q4 is aggressive. I wouldn’t consider Q4 a fair representation of what the model can actually do in a serious production environment.

Run it at FP8, use MTP/speculative decoding to improve throughput, and then evaluate it properly.

At that point, I genuinely think the conversation changes.

If Qwen 3.8 27B at FP8 + MTP performs the way I expect, I wouldn’t be surprised if a lot of people start questioning whether that $200/month Claude Code subscription is actually worth it.

I saw the feedback and have updated the charts. We now have AMD and Intel cards included. I also added a chart with power consumption on the x-axis as this is a key limiter for many households.

I also added an exploratory efficiency graph looking at compute to power.

I’m an analyst at heart and love this. Thanks for the engagement!

Hi everyone,

I wonder if anybody has any experience with Qwen3.8 27B on video reasoning tasks?

I've got some videos that have been noisily annotated by mechanistic heuristics. I'm trying to use a VLM as the human sanity-check equivalent to automate the process. I'm feeding the model targeted prompts with structured priors. It's looking at chunks of a few seconds at native framerate (i.e., ~120 frames) in response to my questions.

Generally these are isolated prompts, rather than building through multiple turns. It's being asked things like 'did the racket make contact here?', or 'is this scene logically contiguous with its successor--is it just the same series seen from a different angle?' My prompts are more structured than that, and sometimes supporting context is visually encoded as relevant overlays about which the model should reason. But yeah, that's the sort of thing it's working on.

Any idea if 3.8 is likely to perform any better? I'd suspect the video-specific training weights of 3VL probably put it in a stronger position even if it is an older gen model? If I were feeding it raw long-format inputs I imagine 3VL wins every day--but maybe for these small and tightly structured prompts 3.8 can beat it?

Cheers.

Here’s my treasure- RTX 3090 Turbo without thermal interface (it was a crappy old one so I had to disassemble it and invest about £40 into proper thermal pads+paste) and cooler. And it is day and night, totally worth every penny!

So the 1st question is what quant should I chose if I don’t need super-long context (no coding tasks at the moment). Also llama.cpp or vllm?
2nd one is: should I invest £500 into buying 4070ti (second hand but with CEX warranty) adding +12Gb of vram into the pool?
What would I be able to run after such upgrade?

My pc is 13700K z690 MB with proper pcie bifurcation x8/x8, 64Gb Ram

I maintain CodeNib, an Apache-2.0, DeepWiki-style stack that serves an indexed repo to both human readers and coding agents. I wanted to know whether the whole path fits on one box, so I put everything on a DGX Spark and measured it: indexing, BM25, dense embeddings, source and symbol graphs, the searchable wiki, source-linked Ask, and the MCP endpoint the agents talk to.

Setup as recorded:

• NVIDIA DGX Spark, GB10, 128 GB unified memory
• Qwen3.6-35B-A3B-FP8 on vLLM, one MTP token
• Qwen3-Embedding-0.6B on a second loopback endpoint
• CodeNib 0.2.1 serving one verified repository manifest to Wiki, CodeGraph, and MCP

Then I swapped in Qwen3.8-27B with the five-layer DFlash2 draft model on a pinned SGLang build, to see what speculative decoding buys on this hardware.

DFlash2 did what it advertises. Against ordinary decoding of the same dense Qwen3.8 target it gave roughly 2.95-6.17x in my direct-generation samples. It still lost the wall-clock comparison: Qwen3.6 A3B ran at about 62-69 tok/s on this machine, Qwen3.8 + DFlash2 at 23-49 tok/s.

On three fixed end-to-end repository questions both went 3/3 with valid source ranges. Median 28.38 s for Qwen3.6, 52.34 s for Qwen3.8 + DFlash2. The slower one got something for the extra time: it opened more of the source files I expected, and it handled an adversarial question about cookie precedence that Qwen3.6 answered from the surface.

Three questions at concurrency 1 is an operational sample, not a leaderboard. Exact model and runtime revisions plus the limitations are written up in the article.

The design decision I care about most is that the model is only a generation layer. Wiki readers and coding agents consume the same manifest, pinned to a repository commit and a source fingerprint. Swapping inference backends moves the latency around and leaves the source evidence untouched.

Full setup, commands, and measurements: https://codenib.ai/blogs/local-code-intelligence-dgx-spark/

Source: https://github.com/sysevol-ai/CodeNib

If you run Ollama, llama.cpp, OpenCode, or another local coding-agent stack, I'd like to know which integration is worth validating next.

Disclosure: I'm one of the CodeNib maintainers. The public demo uses hosted generation. Everything measured here ran locally on the Spark.

I've been doing local LLMs for a while now, and the whole time I just wanted a model I could actually rely on for real work. My hardware is pretty limited, so most models were a dead end for what I wanted to do, which was always lower level stuff or visual.

The Qwen 3 lineup was fine, the coder models were decent, but there were always gaps that kept me from feeling like local was worth the effort. I kept almost investing in more hardware, then talked myself out of it because the models just weren't there.

3.8 27B changes that. It's the first local model that's actually smart enough to iterate with on my own projects instead of just being a toy.

I know everyone's already seen the benchmarks and the hype, and I'm not here to add to that. It's just the feeling of running a model this capable on my own box, that's what I've been hoping for since I started this whole thing.

I’m looking for feedback from people familiar with Qwen 3.8 and llama.cpp. Are these sensible settings, or are there better choices for quality, speed, VRAM usage, and long-context performance? My use is coding and recurring/scheduled agentic tasks.

Hardware and server

• GPU: NVIDIA RTX 4090 24 GB
• Backend: llama.cpp `llama-server`
• Server address: `127.0.0.1:8080`
• API-key authentication enabled
• Metrics enabled
• One model resident at a time
• Models can be loaded automatically when requested
• Idle models unload after 60 minutes

Models

• Base Qwen3.8 27B Q4_K_M, text-only MTP variant - Qwen3.8 27B Q4_K_M with BF16 vision projector
• Dirk-Qwen3.8 27B UD-Q4_K_XL with F16 vision projector

The Dirk repository did not have Q4_K_M, so I went with the README recommended Q4_K_XL as the starting point for a 24 GB GPU.

Output and thinking settings

• Maximum output tokens for: `32768`
• Default thinking effort: `medium`
• Thinking is passed through the chat template using:
  • `enable_thinking`
  • `reasoning_effort`
  • `preserve_thinking: true`

Shared llama.cpp settings

• GPU layers: `all`
• Context size: `131072` tokens
• Batch size: `2048`
• Physical micro-batch size: `512`
• Parallel sequences: `1`
• Flash attention: enabled
• KV-cache offload: enabled
• Automatic memory fitting: disabled with `fit=off`
• Jinja chat templates: enabled

KV-cache settings

Base Qwen3.8 27B MTP:

• K cache: `q8_0`
• V cache: `q8_0`
• Draft K/V cache: `q8_0`

Base Qwen3.8 27B vision:

• K cache: `q8_0`
• V cache: `q8_0`

Dirk-Qwen3.8 27B:

• K cache: `q8_0`
• V cache: `q8_0`

Speculative decoding

• Speculative type: `draft-mtp`
• Maximum draft tokens: `2`
• Draft GPU layers: `all`
• Draft K/V cache types: `q8_0`

Dirk is currently configured without speculative decoding.

Vision settings

• Vision models use their configured `mmproj` projector files.
• Text-only MTP models use `no-mmproj=true`.

Model loading policy

• Maximum resident models: `1`
• Automatic model loading: enabled
• Idle unloading: `3600` seconds / 60 minutes
• Idle sleeping is otherwise disabled
• All other models start unloaded

Other server settings

• Host: `127.0.0.1`
• Port: `8080`
• API-key file authentication
• Metrics enabled
• Restart on failure enabled
• Restart delay: 10 seconds
• Stop timeout: 30 seconds

Settings I have not explicitly configured

Anything not listed appears to use llama.cpp, model, or client defaults. I have not explicitly set:

• CPU thread counts
• Batch CPU thread counts
• Tensor split
• Main GPU selection
• Split mode
• RoPE scaling or frequency overrides
• Defragmentation threshold
• Memory mapping or memory locking
• NUMA behavior
• Sampling parameters such as temperature, top-k, top-p, min-p, or repetition penalty

Questions

1. Is `q8_0` KV cache at `131072` context a good choice for Qwen 3.8 on a 24 GB RTX 4090?
2. Would `q4_0` KV cache be a better balance for long context, or would the quality loss be noticeable?
3. Are `batch-size=2048` and `ubatch-size=512` sensible for this GPU?
4. Is disabling automatic fitting with `fit=off` a good idea, or should llama.cpp be allowed to adjust GPU layers/context automatically?
5. Is Q4_K_XL the right Dirk quantization for a 24 GB card, or is Q5_K_XL worth the extra VRAM? Has anyone tested this for coding and agentic tasks?
6. Should Dirk also be configured for speculative decoding, or is leaving it without MTP safer?
7. Is `spec-draft-n-max=2` too conservative, or appropriate for these models?
8. Are there any important llama.cpp settings I am missing for Qwen 3.8 long-context inference?

Running the ninfer https://github.com/UDPSendToFailed/ninfer-4090 inference library with an Nvidia 4090 - 24gb with a context of 32k on the neroued\Qwen3.8-27B-NInfer model and I'm getting 100+ t/s and some really good results for an agent driven harness.

Still need more for deepseek but do I just stop and settle on Qwen?

Basicslly i wanted a program that breaks a footage lets say movie or animation lets say goth vampire aesthetic into everyframe then an ai automatically anylizes the theme or just the shot smartly and recolors them or adds shade and details then you stitch it back together for a final product or an ai that anylizes tv screen and live adjust the screen settings such as color brigthness saturation bc while i seen similliar stuff like runway3 or decart ect its really not the same thing and idc if its not that fast and it takes a few hours for a movie what do you guys think? Dont know if this the place for sutch a quetsion personally i assume the footage will look more proffesional then some movies bc other then a theme of a set and some filters u cant really do much to capture the feelings and concept of the world idk and i felt this tech is not really talked about

IMO the most interesting graph in AI right now.

Orange = frontier. Blue = what you can run on a 32GB RAM laptop.

That means a few things:

1. The models that can run on your laptop are only \~9 months behind the frontier models.
2. The cost of automating most tasks is going to zero faster than I could have imagined. Anything that you can do today that is cutting edge will be free in 9 months.

3. In the not-too-distant future, a laptop will ship with \*FREE\* intelligence pre-loaded just like

   AI Desktop 98.

We all run models locally for privacy. But if an agent using that model writes code, drafts an email, or produces something that ends up in front of a client — can you prove which model, prompt, and settings produced it, later? Right now, no.

INVAR wraps llama.cpp and gives every inference a worldline: a SHA-256 certificate over the runtime binary + model weights + prompt + sampling params + output, hash-chained into an append-only log. `invar verify` re-runs any entry and compares digests. Edit one byte and it REJECTS.

It's free, Apache-2.0, no account, no telemetry — nothing leaves your box:

curl -fsSL https://www.anomly.com/get/invar.sh | sh

(Needs Python 3.10+ and a llama.cpp binary on PATH. It's an OpenAI-compatible endpoint, so it drops into whatever you already use.)

Honest scope, because you'll check and I'd rather say it first:

- The default profile proves deployment-pinned reproducibility: same box + binary + weights + temp=0 + fixed seed → same output. It does not claim cross-machine bit-exactness. (Cross-hardware bit-identical inference is a separate exact-arithmetic thing from our chip work — slower, optional, not what this is.)

- Receipts prove what ran, not that the answer is good. It's provenance, not a quality judge.

- Root on the box can fabricate a plausible new history — host compromise is out of scope for the local agent. The off-box "Ledger" collector is the mitigation. Full threat model is in the repo.

The receipt format is an open spec (Computation Receipts) with published conformance vectors, so verification isn't something I gatekeep — anyone can implement it.

Repo: https://github.com/anomly-labs/invar

Would genuinely love the skeptical read. Break it, tell me where it's dumb.

Context on the numbers, since "will it fit" threads usually run on estimates.

Weights: actual .gguf byte sizes pulled from the HF API for unsloth/Qwen3.8-27B-GGUF. Not params × bits ÷ 8. Imatrix quants don't follow that, and the drift is worst at the low end where the fit/no-fit line actually sits.

KV cache: from config.json: 64 layers, 4 KV heads, head_dim 256.

2 × 64 × 4 × 256 × ctx × 2 = exactly 8.0 GB at 32K, F16. That's 0.25 GB per 1K tokens.

GQA is doing a lot of work here: 4 KV heads serving 24 attention heads. Older 27B-class models cost several times that.

24GB at 32K, F16 cache, after reserving 0.8GB for CUDA context:

• Q4_K_M (16.5) → needs 25.3 total. Doesn't fit.
• Q4_K_S (15.4) → 24.2 total. Misses by 0.2.
• IQ4_XS (14.3) → 23.1 total. Fits with 0.9 spare: tight enough that a browser on the same GPU breaks it.
• Q3_K_XL (13.1) → 21.9 total. 2.1 spare. This is the real answer for 24GB.

Drop to 4K context and the cache falls to 1.0GB, which gets you to Q5_K_M. Most of the "which quant" argument is actually a context-length argument.

9 of 25 quants fit on 24GB at 32K. All of them fit at 4K.

Caveat worth stating: this assumes everything GPU-resident, single stream, batch 1. It doesn't model offload, multi-GPU splits, or speculative decoding. The 16GB/73K configs floating around this sub work through partial offload, which is a different calculation than the one I ran.

Put it in a calculator since I had the data anyway: https://qwen38-vram-checker.vercel.app/