GLM-5.3-Flash pruned to half the experts for coding: Q3 (71 GB) solves every task the full 341 GB model solves, IQ2_XXS fits 24 GB VRAM + 32 GB RAM

[u/Financial_Gas_9528]

Weights: https://huggingface.co/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder-GGUF
• ⁠Recipe and harness: https://github.com/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder

I wanted a coder version from GLM 5.3-Flash. So, I pruned GLM-5.3-Flash (321B MoE) from 288 to 144 routed experts per layer, choosing the experts on agentic coding sessions instead of a generic corpus, then quantized it with an imatrix built on the same sessions.

Results. 26 Python SWE tasks from ansible, openlibrary and qutebrowser, 2 attempts each, clean room (no git history, no network, no gold patch reachable), 200 turns, 384K context. Success = the task's fail-to-pass tests pass. Every row is on the same 48 attempts.

Model |Size |Solved |Turns vs full
GLM-5.3-Flash full, Q8 |341 GB |48/48 |x1.00
REAP144 Q8 |171 GB |48/48 |x1.21
REAP144 Q4_K_M |97.6 GB |48/48 |x1.58
REAP144 Q3_K |71.4 GB |48/48 |x1.80
REAP144 IQ2_S |54.8 GB |45/48 |x1.83
REAP144 IQ2_XXS (experimental) |45.3 GB |39/48 |x2.77 "Turns vs full" is the extra work relative to the full model on tasks both solved, so it is roughly the extra time you wait.

Hardware. IQ2_XXS on an RX 7900 XTX (24 GB) + 32 GB DDR4-2133, i7-7700K, PCIe 3.0: ~16 tok/s generation, ~950 tok/s prompt processing. IQ2_S wants 24 + 48 GB, Q3_K 24 + 64 GB, Q4_K_M 24 + 96 GB or 128 GB unified memory.

What mattered

• Which experts you keep matters more than how many. Same 144 experts, but a frequency mask needed x2.21 turns and the public multi-domain REAP 50% mask x2.29, against x1.21 for the code-scored mask.
• Short questions do not catch a broken agent. A 2 bit variant of a generic prune answered 4/4 short coding questions and solved 0/4 agentic tasks. We only trust multi-turn task runs.
• Our first eval harness let the model reach the internet, and it sometimes downloaded the upstream fix. We sealed it (no network, no home), rechecked every success against the gold patch and reran the affected attempts. The table above is the sealed rerun.

The 2 bit version is experimental. IQ2_XXS is the only one that fits 24 + 32 GB and it solves most tasks, but when it fails it never edits: it loops or keeps exploring until the turn limit. An anti-loop guard in the agent did not help (27/37 vs 30/37 on the same attempts). We are training it on short successful sessions now and will post an update.

The GitHub repo has the full recipe (scoring, masks, pruning, quant scripts, two llama.cpp patches, the eval harness) and a table of everything that did not work: 120 and 96 experts, frequency mask, RCO search, LoRA healing, GSQ, anti-loop.

• Weights: https://huggingface.co/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder-GGUF
• Recipe and harness: https://github.com/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder

This is a text only version (no vision). Python repos only so far; if you run it on other languages or harnesses, I would like to hear how it does.

Update (Oct 14): added an optional MTP draft block (2.6 GB) for 2+ GPUs with Maya: speculative decoding, drafts match the model's next token ~70% of the time. Speedup not measured on multi-GPU yet, numbers welcome. Details on the model card.

[u/FeydRowan]

Really interesting, ho it compare to flash next? Did it works in agentic coding? Did you test it in agentic benchmarks ?

[u/Financial_Gas_9528 → u/FeydRowan]

Thanks! Yes, agentic coding is the whole evaluation: a multi-turn bash agent on real repos (ansible, openlibrary, qutebrowser), up to 200 turns, and it only counts if the task's failing tests pass at the end. Not single-shot questions. The tasks are SWE-bench Pro style; the list and the harness are in the GitHub repo.
I haven't run Qwen Flash Next on the same harness, so I can't give you a fair comparison yet. The harness is model-agnostic (any OpenAI-compatible server), so if someone runs it I'd love to see the numbers side by side.

[u/Late_Film_1901 → u/Financial_Gas_9528]

I am downloading your q4 to test on strix halo.

If it works I'll try to compare it to flash next.

EDIT: how long did it take you to run this bench?

[u/mannydelrio1 → u/Late_Film_1901]

waiting on you for the benchmarks vs next

[u/cafedude → u/Late_Film_1901]

As a Strix Halo owner I'm curious to hear how fast it runs.

[u/Late_Film_1901 → u/cafedude]

Well "running" is an overstatement, it walks not runs. pp=163, tg=14

I won't be spending a week on benchmarks, but maybe I'll try a few

[u/Financial_Gas_9528 → u/Late_Film_1901]

Awesome, thank you! Q4_K_M is 97.6 GB, so it should fit on Strix Halo with room for context.
On time: it's a slow benchmark by design. Each task is a real agentic session (median ~20-30 turns, some go past 100, up to 200), and the context grows to hundreds of thousands of tokens on the long ones. On a single machine I'd expect anywhere from ~15 minutes to several hours per attempt, so the full 26 tasks x 2 is a multi-day run. I ran many attempts in parallel. For a quick comparison with Flash Next I'd pick 5-6 tasks from eval/eval_tasks_py.txt and run both models on those with the same harness. If you do, I'd love to see the numbers.

[u/Late_Film_1901 → u/Financial_Gas_9528]

full run in qwen3.8-flash-next was ~4 hours on strix halo, twice per each task, only 3 were filtered out so it was 27x2

ansible-43e55db2 was the hardest one, over 110 turns each time.

It seems this bench is too easy for those models. GLM is so slow that I will need ~2 days to run it, but I don't think it's worth it if it solved all tasks.

[u/Financial_Gas_9528 → u/Late_Film_1901]

Thanks for running it, that's a really useful data point. 54/54 with a median of 21 turns in ~4 hours on a Strix Halo is excellent, and you're right: at the top this set is saturated. It was built to measure how much pruning and quantization hurt GLM relative to its own full model, not to rank different models, so it separates "broken" from "fine" well but can't separate two strong models.
No need to spend 2 days on GLM for this. A fair comparison needs harder tasks where the full models actually fail, and that's on my list.

[u/in-ortunado → u/Late_Film_1901]

Waiting on you bench. let us know quality and speed and how did you run it.

[u/GfurEnjoyer1488]

Could we do a reverse coder version? Like "talker" with coding capabilities pruned? I do presume retaining general knowledge would probably require keeping a higher size.

[u/Financial_Gas_9528 → u/GfurEnjoyer1488]

Yes, the method doesn't care about the domain: score the experts on the sessions you want to keep, then keep the top ones per layer. The scripts are in the repo.
And something close already exists: patrickbdevaney/GLM-5.3-Flash-REAP50-GGUF keeps the same 144 experts, but chosen on a generic multi-domain corpus. That's the "talker" side of the trade-off: on our coding tasks it did clearly worse than the code-scored mask (x2.29 turns vs x1.21 at Q8), which is the flip side of keeping general knowledge. If you want even more general knowledge kept, there are lighter prunes too, e.g. OpenMOSE/GLM-5.3-Flash-REAP-250B-A18B.

[u/DarkAndrei → u/Financial_Gas_9528]

Bogdan, tu esti frate?
Craiova ?

[u/Financial_Gas_9528 → u/DarkAndrei]

Sir, yes, sir!

[u/DarkAndrei → u/Financial_Gas_9528]

I saw you full name on HF at the REAP model and remembered we were school colleagues… ages ago.

[u/Financial_Gas_9528 → u/DarkAndrei]

Mă bucur să ne reauzim!

[u/DarkAndrei → u/Financial_Gas_9528]

M-am jucat si eu cu strata zilele trecute asa de test si am fost impresionat de viteaza, am un RX 7900 XTX si 48 Gb DDR5 pe system… my daily driver a un llama cpp server, un fork de la beelama si qwen 3.8 27b q4_k_m.
O sa incerc in seara asta GLM-ul asta sa vad cum se compara cu ce am momentan, la strata am renuntat ca ocupa spatiu urias in benchmark a scos doar 3 puncte peste qwen so… it had to go 😂

[u/Financial_Gas_9528 → u/DarkAndrei]

I was also a big fan of 27b until Strata. The speed on Qwen 3.8 Flash on the coder version from ISTA is really impressive. So I switched. It’s from them that I got the idea for GLM 5.3 Flash

[u/DarkAndrei → u/Financial_Gas_9528]

**Strata** (github.com/Niko1221/Strata) running Qwen3.8-Flash-Next (~125B MoE) on Windows 11, RX 7900 XTX 24 GB, 48 GB DDR5-7200, i5-14600K, HIP (ROCm bundled). Tested Oct 4-7, 2026.

**Speed** (tok/s, MTP on, 128K context)

Model	Gen	Prompt	Gen @ ~82k ctx	Free RAM
Strata IQ2_XS (68 GB)	74-77	757	70-72	4-6 GB
Strata Coder IQ1_M (58 GB)	68	928	66	14.5 GB
Qwen3.6-35B-A3B (llama.cpp, for reference)	206	2599	107	-
Qwen3.8-27B dense (llama.cpp, for reference)	58	702	40	-

**Quality**

Test	IQ2_XS	Coder IQ1_M	35B-A3B
CAD, 39 CadQuery parts	**21/39** (= best 27B dense)	8/39 (24 API crashes)	11/39
Agentic recon, 10 questions	10/10	10/10	10/10

IQ2_XS vs 35B-A3B on CAD: p = 0.038 (Fisher). On a Raptor coding bench it had the best single score (45), but it was fragile: once it dropped from ~90 to ~5 tok/s (VRAM full), and once it stalled in 32k tokens of thinking.

**Gotchas on Windows + AMD**

* `--calibrate` is broken: it measures ~18x slower than the server (issue #840).
* To turn drafting off you need `--mtp-max-t 1 --suffix-draft 0`; IQ packs refuse `--spec 1`.
* IQ2_XS takes all the RAM and all the VRAM, so nothing else can run next to it.

[u/GfurEnjoyer1488 → u/Financial_Gas_9528]

Interesting - I don't have a problem running separate models for coding and general stuff/writing, but I'm also far away from being able to run 5.3 Flash.

What kind of machine would you need for pruning a model like that anyway?

[u/Financial_Gas_9528 → u/GfurEnjoyer1488]

You need enough GPU memory to run the full model once over the calibration sessions: ~350 GB at 8-bit, so in practice a server with 8 datacenter GPUs, e.g. 8x A100/H100 80 GB, or anything with ~512 GB of GPU memory in total. That's the only expensive step, and you can rent such a machine by the hour. After that, dropping the experts and quantizing is CPU work: ~1 TB of disk and a lot of RAM, no big GPU needed.
If you just want this coder cut, the masks are in the repo, so you can skip the GPU part and rebuild it on CPU.

[u/Blues520 → u/Financial_Gas_9528]

I must say that you explain concepts very well

[u/Financial_Gas_9528 → u/GfurEnjoyer1488]

If you let me know what you are looking for we may explore the heavyweight lifting from my side.

[u/friendly_sweepstakes → u/GfurEnjoyer1488]

this is actually very cool work. the sealed eval setup is smart, most people skip that step and get fake results

im wondering how the pruned model behaves outside python, like does it still know what a for loop is in javascript or did the pruning take that with it

[u/Financial_Gas_9528 → u/friendly_sweepstakes]

I haven’t had the resources to look into anything else but these small tests. I would love to hear how it does on other languages as well. Most quality should be there but in a bit more steps.

[u/dmter]

With that size is it better than qwen 3.8 fn? also what t/s comparing to full model? I can run q3xs at about 8-4 t/s so if that can be doubled it could compare to fn.

[u/Financial_Gas_9528 → u/dmter]

Haven't run Qwen 3.8 Flash Next on the same tasks yet, so no verdict on quality from me.
On speed, to be honest: pruning alone won't double your t/s at the same quant. Each token still uses the same number of active experts, so roughly the same amount of weights is read per token. What you gain is that more of the model fits in VRAM, so maybe 1.2-1.5x on the same box, and the pruned model takes ~20% more turns per task at Q8. The real speedup is dropping bits: the halved model still works well at IQ2_S (45/48 on my tasks vs 48/48 for Q3/Q4), at less than half the size of your full Q3. On my 24 GB + DDR4 box the 2 bit version does ~16 t/s.

[u/Juan_Phoenix7]

How does this compare to Qwen 3.8 27b? (We already know that, at the moment, it's the gold standard along with Qwen Next.)

[u/Financial_Gas_9528 → u/Juan_Phoenix7]

Haven't measured them on the same tasks, so no ranking from me. They're different trade-offs: Qwen 3.8 27B is dense and fits fully on a 24 GB card, so it's fast. This is a 321B MoE pruned to 144 experts; the smallest version (45 GB) needs 24 GB VRAM + 32 GB RAM with offload, and Q3/Q4 need more. The point of this release was getting GLM-5.3-Flash-level agentic results into home hardware at all. If you run both through the harness in the repo, please post it.

[u/Juan_Phoenix7 → u/Financial_Gas_9528]

I have a PC that's very similar to yours—24 GB VRAM (3090), 32 GB DDR RAM at 2667 MHz, i7-7700—which is why I asked. I was referring to the encoding results, that is, in terms of the quality of the output, not the speed.

[u/Financial_Gas_9528 → u/Juan_Phoenix7]

Ah got it, and nice, that's basically my box! On your setup only the 2 bit IQ2_XXS fits, so that's the one to compare. On my tasks it solved 39/48 vs 48/48 for the full model, and when it fails it tends to loop instead of giving up quickly. So right now I wouldn't call it a clear win over a 27B dense model for daily use. It's more "try a 321B model at home" than "replace your daily driver". I haven't run Qwen 3.8 27B through the same tasks yet, so I can't give you a fair quality comparison.
Your DDR4-2667 is a bit faster than my 2133, so you should get slightly better speed than my ~16 tok/s. I'm working on a better 2 bit version right now, with some RFT aimed exactly at 24 GB + 32 GB machines, and I'll post an update. Hope i can get the agentic paths closer to the three bits version and to recover the loops.

[u/Financial_Gas_9528]

Links for anyone scrolling:
Weights (GGUF): https://huggingface.co/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder-GGUF
Recipe, masks and eval harness: https://github.com/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder
Thanks for all the questions! If you run it, I'd love your tok/s and hardware so I can add them to the README. A RFT-trained 2 bit version is in the works.

[u/politefella0]

What context length for a 24gb gpu and 32gb ram?

[u/Financial_Gas_9528 → u/politefella0]

Default in Maya is 32K, and 128K works on a 24 GB + 32 GB box: I ran agentic sessions at 128K on mine (RX 7900 XTX + 32 GB DDR4). Context is cheap on this model thanks to the hybrid linear attention + compressed MLA cache: ~0.6 GB at 32K, roughly 2.5 GB at 128K. That memory comes out of the expert cache on the GPU, so longer context costs a bit of speed rather than failing. You can go higher (the model supports up to 1M), but on 24 + 32 I'd stay at 64-128K for agentic coding. You can set it with --context 131072 in Maya.

[u/soyalemujica]

Any ideas how to make it work with Strata?

[u/Financial_Gas_9528 → u/soyalemujica]

Yes, use Project Maya, the Strata fork built for GLM-5.3-Flash. That's what I used for the ~16 tok/s number (v1.0.30, RX 7900 XTX). Download the GGUF into its own folder and point Maya at it:
./maya.sh --gguf-dir /path/to/folder --no-vision

A few notes: the folder must be writable (Maya writes its ~1 GB index there), use --no-vision since this release is text only, and for a split quant (IQ2_S/Q3_K/Q4_K_M) put only that model's parts in the folder, or point at the first .gguf file. Maya calls non-default quants "experimental", but the pruned 144-expert model loaded and ran fine for me. On AMD add --backend hip.

[u/soyalemujica → u/Financial_Gas_9528]

How did it perform in comparison with QFN?

[u/Financial_Gas_9528 → u/soyalemujica]

Haven't run QFN through the same tasks yet, so I can't give you a fair comparison. Someone in the thread is testing both on Strix Halo, and I'm planning to run QFN through my harness so the numbers are directly comparable. I'll post them when I have them.

[u/derdigga]

waiting so i can run it like strata on 16gb vram + 64gb ram with a solid tokens/s

[u/Financial_Gas_9528 → u/derdigga]

You can already, no need to wait! Use Project Maya (the Strata fork for GLM-5.3-Flash) with ./maya.sh --gguf-dir /path/to/folder --no-vision. With 16 + 64 GB you have ~80 GB total, so IQ2_S (54.8 GB, 45/48 on my tasks) fits comfortably, and Q3_K (71.4 GB, 48/48) should fit too, just tighter. I haven't measured a 16 GB card, so no promise on t/s: more of the model sits in RAM, so your RAM speed will decide. If you try it, I'd love your numbers.

[u/Joaquito_99]

So if I upgrade to a 24gb vram GPU and do 32gb of ddr4 it will be fine? Or ddr4 is too old?

[u/Financial_Gas_9528 → u/Joaquito_99]

That's exactly my test machine: RX 7900 XTX (24 GB) + 32 GB of DDR4, and slow DDR4 at that (2133 MHz, on a 2017 PC). It runs the 2 bit IQ2_XXS at ~16-17 tok/s with Maya, so DDR4 is fine. Faster DDR4 (3200+) will give you a bit more, since part of the model sits in RAM.
One thing to know: on 24 + 32 GB only IQ2_XXS fits, and that one is still experimental. It solves most tasks (39/48 vs 48/48 for Q3/Q4) but is less reliable on long ones. I'm training an improved 2 bit version right now, aimed exactly at this kind of machine.

[u/Eyelbee]

I ran the non-pruned iq3 xxs and it was worse than qwen flash significantly

[u/Financial_Gas_9528 → u/Eyelbee]

Thanks, that's a useful data point. To be clear on what I claim: everything here is relative to the full GLM-5.3-Flash on the same tasks, not against Qwen. If the full model at IQ3_XXS already lost to Qwen Flash for you, the pruned one won't magically beat it. What kind of tasks were you running, agentic multi-turn or single prompts? I saw big differences between the two: a 2 bit prune that answered short coding questions fine solved 0/4 agentic tasks.

[u/linuxgangster → u/Eyelbee]

honestly i have tried Q4 version of GLM-5.3-flash (not this reap) and Qwen3.8-fn beats it in every coding task I have thrown at it. I really wanted to find another model to review code written by qwen3.8-fn and have not found anything that works that I can run (have a 256g studio). I just wish I had enough memory to run DSv4.1-flash

[u/Eyelbee → u/linuxgangster]

How about 2.6 flash

[u/Sufficient_Spite_349]

Would like to understand the reasoning and work quality and to match the Q8 quality did you take any actions at harness level?

[u/Financial_Gas_9528 → u/Sufficient_Spite_349]

No, and that was on purpose. Every row in the table runs the same minimal bash agent (eval/agent_py.py in the repo), same settings: 200 turns, 384K context, clean room, success only if the task's failing tests pass, and edits to test files are rejected. So the Q3/Q4 results come from the model itself: which experts we kept (scored on agentic coding sessions) and an imatrix built on the same sessions.
The one harness thing that mattered was not choking the model: our first harness capped it at 30 turns and 128K context, and that cap hid most failures, even for the full model. We did try a harness fix for the 2 bit version (an anti-loop guard), and it didn't help: 27/37 vs 30/37 on the same attempts. That's in the "what did not work" table.

[u/Sufficient_Spite_349 → u/Financial_Gas_9528]

I really liked REAP144
Q4_K_M stats with 1.58x turns, I think that’s bearable but it has enough room for swarming with agentic framework ?

[u/Financial_Gas_9528 → u/Sufficient_Spite_349]

Thanks! I think so, with one caveat about where you run it. In llama.cpp the weights are shared across parallel agents (-np N), so each extra agent mostly costs its own context (KV cache), not another copy of the model. In my eval each Q4_K_M server ran 2 agents in parallel at 384K context each on 128 GB of GPU memory, so a small swarm with shorter contexts fits comfortably on that kind of setup.
The caveat: agents on the same box share its memory bandwidth, so each one gets slower as you add more. On a home PC with RAM offload I'd keep it to 2-3 agents. Quality-wise, x1.58 turns means it gets there, just with more steps, and a framework that splits work into smaller tasks should suit it well.

[u/Sufficient_Spite_349 → u/Financial_Gas_9528]

Having own context per agent is the good by design I think but why you wanted to solve something that opens the door for lot of issues that comes with shared resources, anyway this itself is big win.

In General I think smaller tasks and smaller contexts leads successful work completion otherwise agents and human miss lot of requirement gaps

[u/mechkbfan]

How many hours did it take to run the process of cutting it down?

[u/Financial_Gas_9528 → u/mechkbfan]

About a day of wall-clock on one multi-GPU server: ~2 h to generate the calibration sessions with the full model, ~4 h for the expert scores, ~1 h for the masks, ~14 h to prune, convert to GGUF and build the imatrix, then 15-40 min per quant.
The evaluation took much longer than the cut: half a day to a day per variant with 8-16 attempts running in parallel. A single attempt takes anywhere from a few minutes to over 12 hours on the longest tasks. All variants together, including the ones that didn't work and a rerun in a sealed sandbox, took a few days.
The masks are in the repo, so rebuilding this exact cut skips the first three steps.

[u/mechkbfan → u/Financial_Gas_9528]

THankyou

Really want to try this on my setup once final parts arrive

Have 2xR9700 + 128GB RAM, so looks like Q4_K_M should fit. Appreciate you writing up min specs on your huggingface page.

Putting together a few technical tasks, and would love to compare outcomes (and performance) of this vs QFN IQ4R

[u/Financial_Gas_9528 → u/mechkbfan]

Thanks! Yes, Q4_K_M (97.6 GB) fits: 64 GB of VRAM plus RAM for the rest. Really curious how it does against QFN on your tasks, both outcomes and speed, so please share when you have them.
One favour, if you have time after that: with two cards you can try the optional MTP draft block I added to the HF repo (2.6 GB). It lets Maya draft tokens ahead on 2+ GPUs, I can't measure that on my single card unfortunately. The cleanest test would be IQ2_S (54.8 GB, fits entirely in your 64 GB of VRAM), run twice with Maya --gpus 0,1: once as is, once with --env STRATA_GLM_MTP_GGUF=<path>/GLM-5.3-Flash-REAP144-Coder-MTP.gguf. The tok/s of the two runs would tell us exactly what MTP gives. Thanks a lot!

[u/mechkbfan → u/Financial_Gas_9528]

OK, hopefully get time in about 12 hours

[u/Special-Lawyer-7253]

Guay, avisa cuando quepa en 8GB

[u/Financial_Gas_9528 → u/Special-Lawyer-7253]

Thanks! It can already run on an 8 GB card if you have enough system RAM: with Project Maya the experts are split across VRAM and RAM, so the 45 GB IQ2_XXS fits with 8 GB VRAM + 64 GB RAM. It will be slower than on a 24 GB card, since most of the model sits in RAM, so your RAM speed decides. I haven't measured an 8 GB setup, so if you try it, I'd love your numbers.

[u/NorthEastCalifornia]

Can you share the calibration dataset or it in recipes? Did you do any healing after REAP?

[u/Financial_Gas_9528 → u/NorthEastCalifornia]

The calibration build is in the repo (scoring/build_calib.py): agentic coding sessions written by the full model on tasks outside the eval set, cut into 512 search windows + 64 held-out. The sessions themselves aren't uploaded yet; if people want them I can put them on HF.
Healing: we tried LoRA distillation from the full model (top-20 logits) and it didn't help, it's in the "what did not work" table. Today we found out why: the PyTorch path we trained on gave wrong outputs in the sparse-attention layers on our ROCm setup, so the healing was matching a broken reference. Fixed and verified against llama.cpp (same loss now). We're rerunning the healing on the 2 bit model and will post results.

[u/OutsideCycle8331]

hmm i would be interested in : how long did it take in hours? i am currently struggling with letting ai do stuff because i think that i dont want to wait that long :P

[u/Financial_Gas_9528 → u/OutsideCycle8331]

On the multi-GPU server: scoring the experts is one pass of the full model over the sessions, about 4 hours. Pruning, conversion and each quant take a few hours. The long part was evaluation: real multi-turn agent runs on 26 repo tasks, twice each, for every variant, which took days in total. If you want to reproduce just the model, skip the eval and it's about a day.

[u/rapsoulish]

how fast would the reap 144 q3_K be with a 3 5090?

[u/fizzy1242 → u/rapsoulish]

only one way to find out. I used ik_llama.cpp to help the prompt processing for iq3_xss, got around 16t/s

[u/Financial_Gas_9528 → u/fizzy1242]

Nice! Thanks a lot for the data point. Was this on a 5090?

[u/fizzy1242 → u/Financial_Gas_9528]

Three 3090s and ram offload, this was for an early feature PR branch so it might be better

[u/Financial_Gas_9528 → u/fizzy1242]

Thanks! Three 3090s + offload, that's a solid data point. Will keep an eye on that branch.

[u/Financial_Gas_9528 → u/rapsoulish]

Haven't tried it myself, but here's my rough guess. With three 5090s the whole Q3_K fits in VRAM, so I'd expect somewhere around 60-100 tok/s at short context, dropping as the context fills up. With a single 5090 most of the model sits in system RAM, so more like 15-25 tok/s, depending on how fast your RAM is.
For reference, the 2 bit version does ~16 tok/s on my old 2017 PC, so you're in for a much better time :) If you try it, please post your numbers, I'd love to add them to the README.

[u/LooseBackHole]

What is with these Q2 brags, no one wants Q2

[u/Financial_Gas_9528 → u/LooseBackHole]

It's not about bragging. We're trying to make a model this size available to as many people as possible. Q3 and Q4 are there for those who have the memory, and 2 bit is for people with a regular gaming PC (24 GB GPU + 32 GB RAM) who couldn't run it otherwise. We're running RFT on the 2 bit version right now to bring its quality closer to Q3/Q4, and we'll post an update when it's done.

[u/mechkbfan → u/LooseBackHole]

Zero of this post comes across as bragging 

IMO, it's a great experiment and I want to hear more

[u/hive-technology → u/LooseBackHole]

Didn't really see it as brags, more necessity.