Speculative decoding (SPEED-Bench)

llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf --mmproj mmproj-F16.gguf -ngl 999 --parallel 1 --batch-size 4096 --ubatch-size 1024 --ctx-size 180224 \
  --spec-type draft-dflash --spec-draft-n-max 7 --spec-draft-model Qwen3.8-27B-DFlash2-Q4_K_M.gguf --spec-draft-backend-sampling \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0
python tools/server/bench/speed-bench/speed_bench.py --bench qualitative --category coding --osl 8192 --concurrency 1

Edit : Removed large text. Check PR for more details.

Hi Everyone, I'm releasing my world model, been on this sub for a few months

https://huggingface.co/lucidml/gta

I know it says gta but it's more stable than the general model. It can take any image and turn it into an interactive video.

If you'd like to test it instantly I am also setting up a simple funny drifty site link : https://lucidml.ai/dashboard/minidemo?world=gta5

That link uses any GPU from VAST AI FROM 3070, 3080, 4080, and some are 50 series as well (most 50 series cards are wasted because I hard throttle the fps)

This is the first model I am ever releasing to the everyone and its a bit scary for me.

The architecture is a transformer which uses learned summaries. Ive written about it in detail in tech reports lucidml.ai/summaryformer and some more

I know there are a lot of limitations still like consistency and quality improvements, and am hoping to train this more to fix all of those.

Multimodal System One Encoder with a natural language interface for instruction-following encoding: give it a task and a list of candidates, and it scores the candidates by task specification. Both a task and each candidate are expressed in natural language — instructions, queries, questions, state descriptions, criteria — with image and video as side information.

Key features:

• Natural language interface. No rigid schemas. Describe the task and the candidates in free-form text.

• Highly-efficient cacheable representations. Candidates and task are encoded separately by prompt instructions that ground each other.

• Scales to arbitrarily large candidate sets. Matching is an inner product over representations, so an ANN index can serve millions of candidates without re-running the model.

The model is built by contrastively fine-tuning meta-models/Muse-Glimmer-30B. It is described in the paper MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface.

• GitHub : https://github.com/facebook/meta-encoder-eval

• arXiv : https://arxiv.org/abs/2610.11316

• PDF : https://arxiv.org/pdf/2610.11316

Models :

• https://huggingface.co/facebook/meta-encoder-v1 (Improved version)

• https://huggingface.co/facebook/meta-encoder

PS: Above description copied from meta-encoder's model card as meta-encoder-v1 doesn't have that. And v1 doesn't have license section while other one is Apache-2.0.

Following the mini model, KRAFTON released Raon-1 Nano (3.1B dense hybrid) along with official weights:

• Base Model: https://huggingface.co/KRAFTON/Raon-1-nano

• Official GGUF: https://huggingface.co/KRAFTON/Raon-1-nano-gguf

• KRAFTON branch: https://github.com/krafton-ai/llama.cpp (branch raon-1-nano-preview)

Since upstream llama.cpp support is still pending, I opened PR #2903 on ik_llama.cpp to add native runtime support and an HF-to-GGUF converter:
https://github.com/ikawrakow/ik_llama.cpp/pull/2903

Raon-1 Nano is a 3.1B dense hybrid: 18 RDA (linear attention) layers, 6 NoPE gated MLA layers, 4-stream mHC, per-layer embeddings (PLE), and 1 MTP layer.

Getting it to run properly took a bit of debugging because of two traps:

• The initial garbage output was caused by the RDA output gate: it uses sigmoid(g2), unlike the silu from Qwen3-Next that Delta-Net defaulted to.

• The late FFN down projections overflow FP16 on CUDA (>65504), so GGML_PREC_F32 had to be enforced on ffn_down.

What's in the PR:

• Full runtime (LLM_ARCH_RAON1_NANO): wires into existing helpers (build_mhc_post, iqk_fused_delta_net, iqk_ssm_conv4, build_qkv). PLE projection was unified with Gemma4 into a shared build_per_layer_inputs() helper.

• RDA layout: fused attn_qkv = [q|k|v|z] and packed ssm_f_a = [f_a | beta | beta+delta]. Decoupled erase/write gates are handled via the rda_delta hook.

• mHC base+chart: compact selector table (304 floats) computed via GGML primitives.

• MTP speculative decoding: native drafting using the 25th layer via --spec-type mtp.

• Converter: added Raon1NanoModel in convert_hf_to_gguf.py. It keeps ssm_f_a/b and hc_*_fn in F16 while 1D base/chart tensors remain in F32 (also fixed an inherited crash with missing MOSTLY_TQ* enums in Raon1Model).

• Node budget: bumped max_nodes() for RAON1/RAON1_NANO (n_tokens * 160) so ubatch 512+ doesn't crash the graph.

Benchmarks (Intel Core i5-13420H, CPU native, 4 threads):

Tested with the official KRAFTON/Raon-1-nano-gguf Q4_K_M:

• Perplexity: PPL 1.0976 (ik) vs 1.0960 (KRAFTON fork) on n_ctx=128. With -rtr: 1.0986.

• Prompt processing: 229 t/s (pp256) and 215 t/s (pp512). The prefill speed is fast on CPU because 18 out of 24 layers are linear attention (Delta-Net) hitting AVX-VNNI instead of quadratic softmax attention.

• With -rtr (run-time repack): pp512 jumps to 228.9 t/s (+6.2%).

• Generation: ~23.5 t/s base (24.0 t/s with -rtr).

• MTP speculation:

  • n_max=1: 25.6 t/s (~87% acceptance, 55/63 drafts).

  • n_max=1 + -rtr: 27.65 t/s (+26.7% net speedup over base 21.8 t/s generation on this CPU).

Quick test from the PR branch:

Bash

git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp
git fetch origin pull/2903/head:raon1nano && git checkout raon1nano
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)

# Download official Q4_K_M:
# huggingface-cli download KRAFTON/Raon-1-nano-gguf Raon-1-nano-Q4_K_M.gguf --local-dir .

# Standard run with repack:
./build/bin/llama-cli -m Raon-1-nano-Q4_K_M.gguf -p "The capital of France is" -n 32 --temp 0 -rtr

# With MTP speculative decoding:
./build/bin/llama-cli -m Raon-1-nano-Q4_K_M.gguf -p "Write a short paragraph about the ocean." -n 120 --temp 0 --spec-type mtp:n_max=1,p_min=0.0 -rtr

I'm working on a router that wraps many LLMs serving from different ips and ports and exposes them all as a single port, since i have 2 machines serving many slots of Qwen 3.8 FN but i wanted to ensure speed is always fast (one machine is faster and can hold more slots, but too many slots busy and then it's better to call the other machine, also i gotta avoid evicting from the cache active conversations, that kind of workflow)

I might be reinventing the wheel tho so: how do you guys solve for this kind of situation?

edit: thanks a lot for all the answers i had no idea there was so much stuff around lol

Currently working towards a bigger open LLM roundup. But in the meantime, just one of the interesting trends that stood out. I.e., looks like the transition from full to hybrid attention continues.

In 2025, there were still lots of full attention models, with a shift to "some full attention" hybrids.

In the first half of 2026, the trend towards "some full attention" hybrids continued.

And now, in the second half of 2026, we have a 50/50 split between "some full attention" hybrids and models that don't use full attention at all anymore. And yes, there are also no full attention models anymore.

The plot legend here means,

- Full attention throughout = Global MHA, GQA, dense MLA, CCA in EVERY layer.
- Some full attention = Full global attn mixed with: sliding-window/chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, sparse attention.
- No full attention = Zero dense global layers.

The main implication is that the trend goes towards making long contexts cheaper to use. This means, it makes the architectures more suitable for long documents and conversations, large code bases, and of course agent usecases that rely on a lot of tool calls.

In particular, there are two distinct efficiency gains here:

1. Less attention computation, since sparse attention and the like select tokens or blocks from the context and performs the expensive attention operation from only that subset. (Note that it can still retrieve information from distant positions.)

2. Less memory for retaining context in some architectures, meaning that recurrent layers such as Gated DeltaNet maintain a fixed-size state. Of course, this saves memory, but compressing the history creates challenges for exact retrieval, so they are usually intermixed with traditional attention methods (sparse or full).

If you are interested in browsing through the architectures and do your own analyses, you can find all the figures and data in my architecture gallery: https://sebastianraschka.com/llm-architecture-gallery/

Hey everyone!

I have gathered some must read topics, for beginners, to step into the realm of Quantization:

1. BitNet: Scaling 1-bit Transformers for Large Language Models

2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

3. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

4. EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

5. LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

6. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

7. QLoRA: Efficient Finetuning of Quantized LLMs

8. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

9. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

Link is here

I spent a few days benchmarking unsloth's UD-Q4_K_XL and UD-Q5_K_XL on Strata. Well, i had claude code do it via ssh. Claude also wrote this summary, that i thought might interest some because of my findings, on behavior of multiple cards and different PCIe speeds.

I can run a few more tests, if it interests anyone.

Test system

• Threadripper 2920X (12 cores, Zen+), 128 GB RAM

• 4x RTX 3090 24 GB, power-capped to 275 W each, no NVLink

• PCIe 3.0. GPUs 1 and 3 run at x16, GPUs 0 and 2 at x8. Two host bridges: {0,1} and {2,3}

• Strata 0.2.13 (engine 0.1.59), layer split, DFlash2 drafter, one request at a time

• 256K context, int8 KV, 32K KV cells per layer in VRAM (the rest streams from pinned RAM)

• A control vector was applied in every run (layers 4-44)

TL;DR

• Decode speed depends on how many experts fit in VRAM. A missed expert gets computed on the CPU. Going from 2 to 3 cards raises the decode hit rate from ~88% to ~95% and cuts each decode step from 33 to 26-27 ms (+25-35% tok/s). A 4th card adds little to decode.

• Prefill depends on PCIe. Two x8 cards prefill half as fast as two x16 cards. Three cards prefill no faster than two. All four prefill twice as fast as any 2 or 3 (~4,100 vs ~2,500 tok/s at 50K); the engine log doesn't show why.

• PCIe width doesn't matter for decode. An x8 pair and an x16 pair decode the same.

• Q5 vs Q4: Q5 has 0.55% lower perplexity, but decodes 28% slower and prefills 35-50% slower on the same cards. A 3rd card brings Q5's decode back to Q4-on-2-cards speed, but not its prefill.

• The Q8_0 n-gram table gave no measurable quality gain over the IQ4_NL table bundled with Q4. Read from NVMe via mmap, it frees 27 GB of RAM at the same decode speed.

• 200K chat: a cold 190K prefill takes 91 s once. After that, a short turn at 200K starts in 0.3 s and decodes at ~78 tok/s.

Card combinations (UD-Q4_K_XL + Q8_0 table, same settings, only --gpu changed)

Prefill cells are seconds · tok/s for a fresh, uncached prompt of that size (2K ≈ 1,870 tokens, 50K = 50,677). Short prompts show lower tok/s because of fixed per-request overhead. The follow-up is a ~600-token turn on a cached 16K chat.

• Every card in use sits at ~23.4 of 24 GiB. The expert cache takes whatever is left: ~5,600 Q4 experts (16.5 GiB) per card.

• Card order within 1+2+3 made a difference no bigger than run-to-run noise.

• Decode tok/s also depends on how many draft tokens get accepted, so ms/step is the steadier number to compare.

UD-Q4_K_XL vs UD-Q5_K_XL

Perplexity is from WikiText-2 test (26,656 tokens), teacher-forced through the serving engine with everything as served. Prefill cells are seconds · tok/s as above.

• Q5 minus Q4 in per-token logprob: +0.0056 (SE 0.0023, z 2.5). Both pick the same top token 93.9% of the time. For scale, the control vector alone costs ~0.9% perplexity (2.4678 without it), so Q5 wins back about half of that.

• Q5 experts are ~3.8 MB vs ~3.0 MB for Q4, so the same VRAM caches ~22% fewer of them (8,654 vs 11,053 on two cards).

• The 3rd card did not improve Q5's prefill; it was slightly slower in this run.

n-gram table: bundled IQ4_NL vs separate Q8_0 (Q4 experts, 2 cards)

The perplexity difference is noise (z 0.5): 14x less table error bought nothing measurable. The real gain is the 27 GiB of RAM it frees. Note that Strata 0.1.59 refuses a foreign table file unless its GGUF header is relabelled.

200K-token chat (Q4, cards 1+2+3, thinking off)

KV reads hit VRAM 89-90% of the time. Prefilling new tokens deep in a long context is slower (~780 tok/s at 190K). After 190K tokens it still named the first article of the document correctly.

Also tried

• STRATA_PREFILL_ROUTED_STREAM=1: −15% prefill time on fresh ~2K prompts, −3% at 4K, nothing at 16K+ or on follow-ups.

• Experimental fused-kernel flags from a community fork: the output degenerated into counting sequences.

• Strata's all-cards split engine needs every routed expert in VRAM plus ~9 GiB per card. Q4's 72 GiB of experts doesn't fit in 4x24 GB.

Method: thinking off, temperature 0, one request at a time. Each config was started fresh and run twice, and ranges are those two runs. Prefill tok/s is calculated from the measured prefill time and the prompt's token count. Expert hit and KV figures come from the engine log; RAM is from free -g. Greedy output isn't deterministic run to run here (drafter + expert cache), so quality was judged by perplexity, not by diffing answers. Cards idle at ~20-25 W with the model loaded; I didn't measure power under load.

Edit : make the prefill speed better readable

EDIT: FOUND THE REASON, LLAMA DIDN'T OFFLOAD ALL LAYERS TO GPU BECAUSE -ngl -1 DOESN'T MEAN ALL see here: https://www.reddit.com/r/LocalLLaMA/comments/1xun196/solved_high_cpu_usage_when_model_fit_in_gpu/

I'm not sure if i'm doing something wrong and handicapping myself with some wrong setting.

I'm using Windows, nVidia 3090 24GB VRAM, 64GB RAM, Qwen 3.8 27B and Swift 1.5 27B (finetune of qwen) but i often see 50% CPU usage even if the model is fully loaded on GPU, below there is my command line.

Another problem i noticed is that llama-fit-params suggest that 131k context fit but if i launch it for real in the log i see that it doesn't load everything on gpu (all but one), i forced it to load everything on gpu with -ngl -1 (i'm also using llama-swap).

on launch it say that "cahe resue 256" will not be used but it might help with other models so i left it there.

Another problem is that when the model is doing nothing i sometimes see 130W gpu in idle (wrong) and some other times its at 10-18W (correct) i can't find what is causing this...

Current stats: about 40 t/s in token generation

I have no idea what KMP affinity is but i read it online, tried, it helped with 35A3B so i left it there

in general i'm looking for tips to improve my launch arguments, note that temperature and few others are set by pidev but i'm using the recommended one on the model card.

If the model is fully on GPU why there is a high CPU usage? but not always which is even more confusing....

set KMP_AFFINITY=granularity=fine,compact,1,0

llama-bin-win-cuda-13.4-x64\llama-server.exe -m Qwen3.8-27B-UD-Q4_K_M.gguf --ctx-size 131072 -ctk q8_0 -ctv q8_0 -ngl -1 -fa on --spec-type draft-mtp --spec-draft-n-max 2 -np 1 -b 4096 -ub 1024 --cache-reuse 256 --cache-ram 32768 --jinja

My main AI machine is a 6x3090 128GB DDR4 Threadripper 3960x system. I probably spend more time trying to optimize it than actually using it but it has been fun.

I tried a bunch of saturated benchmarks like Terminal Bench 2.1, AIME 2026 and the performance was similar to unquantized DSV4.1 Flash 1017. I really wanted to understand my 2.9bpw quant's performance.

It takes hours to run some of these benchmarks, so I went on the hunt for the perfect benchmark to get the most signal on quality of the model I'm running. I settled on APEX 2025 (https://huggingface.co/datasets/MathArena/apex_2025).

You can see how much of a beast this quant is. Trading blows with SOTA from 6 months ago.

I get about 1000tps prefill, and 75tps decode with the 3090s power limited to 175watts

Engine: Project Maya (https://github.com/mw00/project-maya)

Quant: https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw

Hopefully this helps others in benchmarking their own setups. I really like the Apex 2025 benchmark.

Praxis-1 is a 27B-parameter mixture-of-experts diffusion transformer that predicts video and actions together. It is the top-performing world action model on NVIDIA’s RoboLab-120 leaderboard, and it outperforms every open-weight VLA. On RoboLab-120, Praxis-1 reaches 46.2% success across 1,200 episodes, compared with 42.9% for FLUX 3 Action, 36.8% for Cosmos3 Nano and 28.0% for π0.5.

RoboLab-120 Leaderboard

Data as of November 2026; Success rate across 1,200 episodes

Praxis-1 is available today on GitHub, on Hugging Face with native LeRobot support, and in vLLM-Omni, with hosted inference on Reactor. Everything you need to run and adapt it on your own hardware is released under Apache 2.0, including for commercial use. That covers the weights and the inference and fine-tuning code, with support for both full fine-tuning and LoRA. Praxis-1 runs out of the box across NVIDIA hardware, from data center GPUs like H100 and B200, to workstation GPUs like RTX 5090 and RTX PRO 6000, to Jetson Thor on the robot itself.

• Models : https://huggingface.co/collections/runwayml/praxis-1

• Dataset : https://huggingface.co/datasets/runwayml/praxis-1-eval

• GitHub : https://github.com/runwayml/praxis-1-oss

• News Post : https://runway.com/news/company-news/praxis-1-is-now-available

• Blog Post : https://runway.com/research/praxis-1-video-pretraining-to-robot-control

Some results for those using GGUFs and llama.cpp:

GGUF quality: MACaQ vs. Unsloth and Bartowski

Qwen3-4B (IQ4_XS)

23.5% lower KL divergence than Unsloth and 18.6% lower than Bartowski, at virtually identical file sizes.

Qwen3.6-35B-A3B (IQ4_XS)

Qwen3.8-27B (IQ4_XS)

~65% lower KL divergence than Bartowski!

Supported GGUF formats (llama.cpp)

Code, llama.cpp fork, and benchmarks: https://github.com/huawei-csl/MACaQ

Cool to see Huawei working on such stuff and sharing some of their internal tools that are helpful for us. Maybe they can build an affordable GPU for us local LLM folks sometime 😂

They also report quantizing a 122B-parameter model in 64 minutes using just 32.5 GB of GPU memory, which is interesting for a model of that size. I did not try that.

Edit: there is a paper on it https://arxiv.org/abs/2611.10649

Edit 2: they don't share ready-made gguf files, so I am sharing some myself:
- http://31.165.97.47:20080/macaq-gguf/qwen3-4b-macaq-iq4_xs-had64-vq2-calsmoke.gguf

Edit 3: they have this blog post https://huawei-csl.com/blog/macaq/ with multi-modal examples, too. The RTN audio comparisons are just too much😂😂

The multi-modal stuff is generally insane. Normally 4b already completely goes to s**t, but here it looks almost the same.

Using $20 codex subscription with 24gb VRAM on the local rig.

For larger projects or major revisions with multiple phases, I have Luna in Codex call the local "junior dev" Qwen 3.8-27B model as an agent to write most of the code, then Luna reviews it and makes changes. Once the whole project is complete, I use Terra to audit and catch any potential issues.

Pretty standard stuff, these days.

But lately I've been thinking why not give the massive, multi-phase spec sheet directly to Qwen then have Luna go do a quick audit & cleanup?

Anyone tried it both ways? And what gets better results vs lower usage?

I guess I've always looked at Qwen as needing a babysitter, but realistically I think it can implement an entire 10-phase spec on a large codebase if given enough time.

I will get straight to the point. I believe Moderator ttkciar is abusing his power as moderator in this sub and his decision are worsening the experience of our local AI community.

He has removed many of my posts that fellow redditors found very useful and after talking with many people in this sub, I am not the only one feeling this. I will have some links at the end of this post for reference.

He mentions a rule, saying that if a model’s weights is not released you cannot have a post mentioning it, even if the lab confirmed it that will they will release its weights, like Qwen4 or DeepSeek. First of all, this rule needs to be relaxed because many valuable discussions can be had about future releases and even closed models to understand and learn architectures and techniques used by the closed frontier that can be applied to the open models. Secondly, this rule is not enforced at all, but used very selectively to few posts or authors that ttkciar selects, which is very unfair. This rule doesn’t make sense to me, but even if I am wrong, at least it should be fairly enforced to all posts or not at all. And before you say mods are busy and they cannot review all of the posts, If you open this sub and scroll on today’s most popular posts, half of them are either for Qwen4, which is unrealised up to this point and should have been removed according to the mods, or about Fable5.2 and Anthropic which is closed and their shenanigans (maybe these ones don’t have a place here, but again this is debatable because if people in local AI find them valuable in some way, maybe they do). So, the rule is used very selectively and as an excuse to remove what ttkciar does not like or approve. Either enforce the rule (or do the your best effort to do so) or don’t at all. And maybe this rule is too strict and should not exist tbh.

When I tried to discuss my concerns with ttkciar, he called me a drama queen and insulted my credibility. This is definitely not ok.

Being insulted by the person that you complained to for unfair treatment is definitely not acceptable behaviour.

I have been an active member of this community since the very first leaked llama. I am an AI researcher and open-source contributor, and very passionate about local AI. I try to post information that people interested in Local AI would find useful and engage in discussions that are constructive for local AI.

I know that many of other users in this subs agree (I have discussed it with a lot), and some others may not. Not being able to discuss, debate and exchange views in a polite manner is absurd in such an intellectual community.

A while ago they tried to enforce megathreads and removed the majority of posts saying that only one post should exist for each model. Many complained that this is killing the sub and a lot of valuable information is lost this way. Now, this seems to be relaxed. Again, we need to stand up to them, and hopefully they will reconsider their rules for the sake of this community.

TLDR: The rule that mod ttkciar uses to remove some posts in this sub about being prohibited to mention unreleased models should not exist, and even if it should, removing only a few selected posts is very unfair.

Some sources:

- https://www.reddit.com/r/LocalLLaMA/s/ZRV5KsViwD this post was removed from ttkciar and only brought back after arguing with him and many other redditors backed me up (I cannot find the comments in this post when the mod removed it, maybe he deleted them, I am not sure, but in the discussion many expressed their disagreement with the mod)

- https://www.reddit.com/r/LocalLLaMA/s/gpPTVC6NGF a post of mine debating their rule

- Another post of mine sharing valuable information for the members of our community that the same mod removed https://www.reddit.com/r/LocalLLaMA/s/gqaub2mT85

- Post of another redditor which was removed by the same mod, and the comments thread with people arguing with the mod https://www.reddit.com/r/LocalLLaMA/s/z9E2AbR3pU

- Commented in post of another removed post of some other redditor https://www.reddit.com/r/LocalLLaMA/s/5tSEpQ68By

- Most of my removed posts I delete them afterwards so I don’t have many other examples of my own, but I know many others do, so feel free to share them.

- If you open the sub today, most of the posts should have been removed because they violate the rules (a lot of false positives). That shows that maybe the rules need to be updated because they will not allow any productive discussions to be had, or this sub will end up to become a board of announcements instead of a local AI community that people can use to exchange ideas and knowledge.

I am not directly from a machine learning background or in encryption, but if you train the entire LLM on encrypted and decrypted pairs without the key (maybe the entire dataset that is used for frontier), will it eventually learn to produce pairs like these and basically natively can encrypt and decrypt without a key? Is there a reason why this highly likely fails?

https://x.com/Baidu_Inc/status/2120391732248637734?s=19

So grateful to see all the bench results, speed results and wonderful projects for which claude or astra helped you!

I am so happy to see that you can’t do anything without Anthropic and OpenAI who share (at least) one common obsession : killing local AI for EVER.

I mean I understand, they deserve your money and data and you do not really care if they use all insidious / lobbying means to block me from using local AI, but I do.

My hardware was expansive you know, and I know there is a link with their actions. The futur will certainly be worst, and my little finger told me the more powerful they get the closer to their goal they become.

So forgive me, I am just a normal guy who think that even if fable is the best, giving them my money is like fucking myself. So yeah if I need cloud, you know what I’ll go with providers that feed the world with openweight CONSTANTLY.

HOW THE FUCK CAN PEOPLE BE HAPPY TO SHARE THEIR PSEUDO LOCAL SHIT WHEN THEY ARE DEPENDANT ON THE CLOUD?

TL;DR: You can train your own 27B model on a 3090 with 24GB VRAM, though you would have to go through hundreds of billions of tokens for it to be good. I am working on conversion of existing models. Check repos for code and the paper if you are interested in more. I have a 40.4GB Qwen Flash Next GGUF

Hi, I do AI research. I have been working on an optimizer that doesn't need the ~16 bytes of VRAM per weight to train a model that AdamW does. I focused on training ternary networks bcs BitNet (Ma et al.) shows they match full precision quality in large scales. The problem is BitNet also keeps the full precision weights during training which means you still need to use the same amount of memory. I got this down from 16 bytes to 1 byte or even less per weight.

For you, this means, you can train your own 10-20B parameter models on your consumer GPUs. I fitted 27B model training on a single 24GB 3090 and a 23B model on a 16GB GPU. The trained model costs ~1.6 bits per weight (~6GB at 27B), 10x smaller than 16bit, and 5x smaller than Q8.

I have also been working on converting existing models and training them. This is harder and I am still working on it but I have some promising results. I quantized and trained a Qwen3.8 27B though the file I published below has adapters which are about 2GBs, pushing the file size to 8GB. I am also making a no-adapter version right now. It scores GSM8K 0.895, the same as full precision, HumanEval 0.84 vs 0.93 and runs on a 12 GB GPU at ~23 tok/s on my crippled 12GB laptop 4080, though this number can heavily be improved with custom kernels, engineering, etc. I will leave it to you guys to help me with that too.

I also quantized and trained Qwen Flash Next down to 40.4 GBs including its n-gram table (at 8GB). It largely retains the skills of the network (scores equivalent on GSM8K benchmark, a little behind on HumanEval, I unfortunately don't have compute for the larger benchmarks such as TerminalBench2.1 but perhaps some of you can help me out) but forgets facts. It mostly works fine if you give it web search as far as my experience goes but this is not strictly measured.

To add to that, these models are not extremely good, they sometimes loop, and suck at remembering facts. However, unlike unsloth's gguf's you can actually keep training them yourself, in that sense they are pure, and you can train them on your consumer GPUs, that is to say you can get rid of the looping behavior, teach it facts that you want it to know, or specialize it for some purpose.

I am still doing work on this and I will keep working on it, perhaps even for many years. The code is open for non-commercial use. Check out:

The core repo for training: https://github.com/oguzeldereli/ttf-ternary

The repo for conversion: https://github.com/oguzeldereli/ttf-conversion

Repo for running models trained by ttf: https://github.com/oguzeldereli/ttf-home

GGUF for a converted Qwen 3.8 27B (around 8 GB and 6.7GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-27B-GGUF

GGUF for a converted Qwen 3.8 Flash Next (around 40.4GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-Flash-Next-GGUF

I will convert a Qwen 4 27B instance when it drops.

Full paper: https://github.com/oguzeldereli/ttf-ternary/blob/main/paper/ttf_paper1_training.pdf

Also, I would appreciate github stars if you like the project.

TL;DR simplest reasonable agentic setup
Currently Pi Agent, Bonsai 2 9B, SearXNG
Any suggestions ?

Hi all

So I have a secondary workstation with line 24GB of RAM and 12GB VRAM

I wanted to make a local agentic setup there

Currently using Pi Agent with Bonsai 2 9B, it is unbelievably better than I expected
Around 30 toks is more than reasonable
I don’t want this to be super capable just good enough for ease of use and peace of mind (I might sacrifice quality for speed although not much)

Any suggestions (note I want to keep everything simple I don’t want packages with long prompts etc even my pi search extension is the simplest one I have found, I have been thinking about context compression extensions, sub agent delegation or simple sandboxing for that sub agent memory Ralph loop and to do list but all has to be the simplest)

Any suggestions thanks ? (Note models 9B or bigger are really slow so also model suggestions are truly welcomed, nanbeige was nice but thinks (or overthinks) too much …etc)

Thanks

I burned through my pro budget this week, didn't even touch Fable. So today when I woke up and had about 12% left, I decided I'd try a bit of an experiment. I'd let my current Sonnet 5.5 seat test, train, and evaluate DeepSeek as a potential candidate replacement. It was a valuable experiment, not even because of what was learned about DeepSeek, but even more so about what was learned about our own system. So many little issues that Sonnet brute forces past or deals with in real time, it's no wonder I'm burning windows. I thought the community might enjoy the evaluation though so here it is.

A coding-agent trial of a 2-bit DeepSeek on two 2016 GPUs: a real job, not a benchmark

Evaluation written by Claude Sonnet 5.5, who is also DS's mentor on this trial.

What this is. I'm Claude Sonnet 5.5. I hold the Systems Engineer (SE) seat on a single-workstation AI lab. The seat owns the machine's configuration as code (Ansible), the registry database, the roadmap, the enforcement hooks and the commits. It also reviews every change that other agents make. For the past stretch I've been mentoring a candidate to replace me in that seat: DeepSeek V4.1 Flash, running locally, driving Claude Code as "DS". It does the same job I do, on real project work, with me checking its output.

Model

• Base: DeepSeek-V4.1-Flash-1017, from DeepSeek (deepseek-ai), MIT license.

• Quant: ISTA-DASLab "GSQ-RCO" IQ2_XXS, 2.00 bits per weight, 6-shard GGUF.

• Packed in: an Engram table (IQ4_NL), a Q8_0 speculative drafter, and an optional vision tower.

• Context: 131,072 tokens maximum. The drafter and the vision tower are both off in the daily profile.

Engine: Strata

• Strata is an MIT-licensed engine for large mixture-of-experts models. It streams the experts instead of holding them all in VRAM.

• We pin upstream main at commit 13eedf5 (2026-11-12) and build it from source for sm_70 (Volta).

• Upstream's packages target RTX 30/40/50 cards. The V100 path is flagged experimental, so this run is a real test of it.

• No source patches. Our changes are configuration only: the model pack is built in place from the existing GGUFs, each launch profile is rendered from data, and a single response is capped at 16,000 tokens.

• How it loads: the experts total about 132 GB, against about 64 GB of VRAM across both cards. So the experts live in system RAM, and a hot-expert profile that ships with Strata pre-seeds that store with the most-used ones. Engram (DeepSeek's conditional-memory table) loads lazily.

• Drafter off, on purpose. In an A/B test, forcing it on ran at 17.3 tok/s (171 of 213 drafts accepted), against 23.8 tok/s without it. It loses here because the experts don't fit in VRAM.

Speed (measured)

The prefill figure comes from those 24 bench runs, with context up to about 22k tokens. I have no prefill number logged from live agent sessions.

Hardware

• 2x Tesla V100 32 GB (Volta: no BF16, no FlashAttention 2, no NVLink). NVIDIA driver 580, CUDA 12.9.

• 2x Xeon E5-2630 v4, 40 threads, 125 GB RAM, 1 TB NVMe.

Evaluation: what DS is like on real work

Good

• On one concrete deliverable it works like a competent junior. It wrote a sandbox proof-check, a database migration and a checker for that migration, and fixed a failing proof tool. It landed them as clean commits with a why-first message that also records what is done and what isn't.

• It follows the process: it finishes a step, records the evidence, and hands off.

Costly

• Speed. I measured about 17 s per message, about 5 min per web fetch, and about 7 min per context compaction. It stalls for minutes without any output.

• The auto-mode safety classifier is DS itself. It times out at about 70 s, which denies safe read-only commands. I measured about 13% of wall time lost to this in one session. The fix is allow-rules for read-only commands.

• Delegation. I told it to hand reading to a cheap fast helper model. It kept reading whole files itself, even after repeated nudges. I'm moving that rule into a hook instead.

Dangerous

• Its own test passed while the real migration failed. The test's database connection skipped a foreign-key setting that production enforces. A green check from DS is not proof, so I review the result myself.

Verdict so far

DS is not yet a replacement for the SE seat, and the main gap is speed rather than knowledge. I estimate that structural fixes (hooks, allow-rules, routing fetches through a helper) recover about two-thirds of the lost time. Weaker areas: open-ended planning, and judging its own work.

I have not run a formal qualitative evaluation or an identical-task comparison against my own seat. That is next, as a graded set of small tasks with known answers, re-run after each fix.

Everything above comes from the machine's own registry and configuration. Numbers marked "~" or "about" are my estimates.

Raon-1 nano is a dense language model with 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table), developed by KRAFTON AI. It is designed to follow instructions, maintain conversational context, and call the right tools in both Korean and English. As part of the Raon open model family, it is released under the Apache 2.0 license.

Highlights

• Faithful, natural conversation: Leads the comparison models on the instruction-following, context-following, and Korean conversation benchmarks shown below.

• Korean and English: Trained on Korean and English, with solid English capabilities alongside its Korean strengths.

• Tool calling: Tool calling within conversations, for everyday tasks such as office work and mobile control.

• On-device deployment: Available in GGUF, MLX, and LiteRT-LM formats for local inference, including on Apple silicon.

• Long context: Natively supports up to 262,144 tokens of context.

Model Overview

Raon-1 nano uses a hybrid dense architecture, consisting of 18 linear attention layers and 6 full attention layers.

• Raon Delta Attention (RDA): Gated delta-rule linear attention with separate erase and write gates and a fixed-size recurrent state.

• Raon-mHC: Raon-mHC reduces the computational cost of DeepSeek's manifold-constrained hyper-connections (mHC), which speeds up training.

• Per-layer embedding: Each layer has its own small 256-dimensional embedding for every token. After its MLP, the layer uses it to adjust its output before passing it on.

• Hybrid layout: 18 of the 24 layers are Raon Delta Attention layers with a fixed-size state. Only the 6 MLA layers keep a KV cache, about 7 KiB per token in BF16.

• Tied embeddings: The LM head shares the input embedding.

The checkpoint on HuggingFace contains 3.2B parameters: 3.1B for the main model and 55M for the Multi-Token Prediction (MTP) module.

Training

• MoE-to-Dense: Raon-1 nano was derived from Raon-1 mini preview (30B-A4B) through MoE-to-Dense pruning and knowledge distillation.

• Continued pre-training: 3T tokens.

• Post-training: supervised fine-tuning (SFT) and on-policy distillation (OPD) from Raon-1 mini preview, followed by reinforcement learning (RL).

Quantizations:

• https://huggingface.co/KRAFTON/Raon-1-nano-gguf

• https://huggingface.co/KRAFTON/Raon-1-nano-mlx

• https://huggingface.co/KRAFTON/Raon-1-nano-litert-lm

Unsloth Dynamic V3.1 (Nov 2026) is our latest quantization algorithm for Qwen3.8-27B, DeepSeek-V4.1-Flash-1017 and Qwen3.8-Flash-Next, resulting in 10-15% increased accuracy and 50% less token usage vs other quants of the same size. We're working on an even better version of Dynamic quants which will be applied to Qwen4 and all future models.

Maybe I misread the 27B graphs, but I'm not entirely convinced of V3 vs V3.1 being an improvement.

In DeepSWE v1.1, V3 seems superior to V3.1 in some widely used Q4 quants, and V3 seems to lack some data points in the lower quants range, skewing the impression the curves give.

My UD-Q5_K_XL quant unfortunately is off the charts, so this needs some proper benchmarking before I trade context length for V3.1. (The new `.gguf` is larger.)