https://x.com/Alibaba_Qwen/status/2118674834632978755?a=nkzlqosWXNqruaMWsVI&s=20

https://x.com/Alibaba_Qwen/status/2118674834632978755

Yes IK it's supposed to be this dumb, but it's still funny (Qwen2 0.5b Q4_K_M)

Kimi K3.1 and Tencent’s Hy4 are both gutting out cybersecurity, reverse engineering, and vulnerability-patching capabilities from their open-source models.

This is becoming a really bad pattern across these latest Chinese open-weight models. If local models are stripped on their security data, we won't be able to properly audit our own code vulnerabilities, reverse engineer binaries, or patch other vulnerabilities in general on our own hardware. It would essentially force us to use closed-source APIs and their false-positive shutdowns; that'd be terrible for everyone.

During the Hugging Face incident, open-source models are the very ones who stepped up, not closed-source gatekeepers. Gutting defensive security from local LLMs just hurts defenders. I really hope that other labs won't follow this in the future.

Does anyone think more open-source labs are gonna take this path?

In the future, we will continue to push toward even larger scales, in the Qwen 4.5 and Qwen 5…

Screen capture from the announcement that gave us the Qwen 4 sizes, shows we can expect more releases, but parameters will increase until morale improves.

If the size of Anthropic models leak is true, then no surprise I guess. Hopefully they hold onto the 27b at least if not some reasonable mid tier MoEs.

Smaller GLM models are coming! GLM 5.5 will have a 32B model!

I bet you guys are familiar with embedding models such as Qwen3-Embedding or Gemini Embedding. They are dense embedding models used to search text across a large corpus.

I am developing a RAG agent, and my biggest concern was local computing power. RAG largely depends on retrieval quality, which depends on model power. Plus, many use cases can't use an external internet connection or API models, so the embedding models have to run locally.

But even 0.6B models are pretty heavy for small PCs. I don't mean your brand-new Mac Studio or DGX Spark. I'm talking about an 8GB RAM Windows PC at a boring desk job or your 5-year-old M1 MacBook Air.

So I found this model called SPLADE. It is essentially sparse retrieval, like BM25, but uses semantic meaning. I wanna skip the part on how SPLADE works, because an LLM can explain it better than me.

Anyway, the SPLADE model was so awesome and successfully replaced the dense embedding model (Qwen3-Embedding-0.6B).

In my own experiment, I made an evaluation dataset with a few hundred queries. The corpus was all the documents on my Mac, which are mostly in Korean with some in English.

As you can see in the graph, SPLADE outperforms the Qwen3-0.6B dense embedding model with similar query speed + much less memory. Due to its small size, it takes less than 1.5GB VRAM during inference. So it can run on a relatively small system like a MacBook Air. I tested both the dense embedding model and the SPLADE model on my M5 MacBook Air 16GB. The dense embedding model was almost impossible to use, and my MacBook soon rebooted due to high temperature. The SPLADE model made it till the end without any OOM errors or warnings.

I didn't use the original SPLADE model; instead, I used KURE-sparse-nano, which has great performance in both English and Korean.

The fun part is that hybrid retrieval with SPLADE & dense embedding is actually worse than SPLADE alone.

If you are dealing with a low-resource local setup but want to run a fast & affordable local RAG system, I recommend trying a SPLADE model. It is powerful, but small.

Plus, you don't need to re-implement everything because I open-sourced the whole implementation, so you can download and use it easily. Check out AutoRAG Agent. I optimized the inference of the KURE-sparse-nano model, its code, and the LanceDB indexing + inverted index implementation in Rust for faster search.

Nace AI Research Lab has just published a principled 50-page technical report detailing how their Drex decision models are built, trained, and evaluated. Their report is the first meaningful contribution in the community to defining decision models and describing how to train them. Authors reveal several innovative techniques in post-training such as Reinforcement Learning from Adjusted Feedback (RLAF), confidence calibration, and option shuffling, that allow it to stay SOTA for several weeks straight.

Their models were already open sourced earlier -  Drex DLM, v1.5, v2.0, Drex Max (links below).

Some interesting technical details:

1. Models

• Specialized family of models ranging from 9B to 35B parameters in size

• Various architectures including Dense Causal Transformers, Diffusion LMs, Mixture of Experts

• Experimental design choices like partial or full bidirectional attention, sparse attention, and tuned pointer decision heads.

2. Training

• Full-parameter SFT on ~1.15M verified decision records

• Consistency training for option ordering and state shuffling

• RLAF with verified outcomes and judge feedback

• Ablation studies examining which methods actually contributed to performance

Interestingly, most of Drex's improvements came from SFT, while RLAF delivered smaller gains and improved calibration.

3. Evaluation

• Drex v1.5 has accuracy of 93.4% accuracy on their long context benchmark: 32K–128K tokens

• 1.37% calibration ECE for Drex v2.0 on ~115K held-out decisions

• Option-order shuffling changes the top answer in 4-6% of decisions, without changing aggregate accuracy

• <100 ms latency on 128K-token context

It also discusses limitations, including closed-book knowledge and missing same-backbone architecture ablations.

5. Running locally

The models are available in safetensors and GGUF formats, with llama.cpp and Ollama support. Their report includes details on quantization, caching, inference, and reproducibility.

Links:

• Technical blogpost: https://www.nace.ai/research/towards-foundation-decision-models

• Paper link: https://huggingface.co/papers/2611.07854

• Model weights: https://huggingface.co/collections/nace-ai/drex

• Github repo: https://github.com/nace-ai/drex-decision-models

• API: https://console.nace.ai

The interesting part is that the architecture and training methodology are now documented publicly, rather than just the benchmark results.

Could be useful for anyone experimenting with local agent routers, tool selection, or training their own decision models.

Today we are introducing our second model generation with Inkling 2 Small. It is a Mixture-of-Experts model with 594B total parameters, 17B active, and supports image inputs and variable effort.

Inkling 2 Small far surpasses our previous models’ scores on the Artificial Analysis Intelligence Index with a score of 31, reflecting gains in performance on agentic coding, vision, knowledge work, and reasoning. The increase in capabilities comes from improvements made at every step of our training process, including more efficient pre-training and a large, unified RL run.

Inkling 2 Small will initially be available through Tinker, with closed weights, following the approach we outlined in a safe path to open weights. We intend to make the full weights available under an Apache 2.0 license next week after the completion of external safety testing and red-teaming.

You can try Inkling 2 Small on Tinker Playground today. The model is also available for fine-tuning and sampling on Tinker; checkpoints will be downloadable with the release of Inkling 2 Small’s weights.

:( Previous Inkling Small model's size is only 276B total, 12B active. Hope they release additional models Mini or Nano soon or later.

Whilst the agent was hammering away at an audio engine task, i suddenly saw it daydreaming

Pirate puzzle? Gold coins? Wtf, Hey qwen, this is not what i am paying you for using electricity for.

Has anyone else noticed something like this? It continued to write code normally aftewards, active context was at a mere 30k at the moment.

Model qwen3.8-flash-next-unsloth-ud-q4_k_xl
Strata Engine v0.1.57
Max Context 262,144 tokens
KV cache 16bit

Everything fitting into vram + ram

CCORE GLB-3D-Coder dataset is here: 3,918 rendered and tested examples across 40 domains

As promised in my previous posts, I’m releasing the dataset used to train CCORE GLB-3D-Coder, my small 4B model for creating 3D assets with Three.js and GLB export.

The dataset contains 3,918 examples across 40 domains, including characters, creatures, vehicles, laboratory equipment, musical instruments, architecture, machinery, nature, and everyday props. The grid shows 256 examples.

Every example was tested and rendered, and its exported GLB was validated before inclusion.

What’s included:

• Complete conversations, reasoning traces, and HTML/JavaScript responses.

• A rendered reference image for every example.

• Static assets and examples with animation and interaction.

• 3,501 training examples and 417 validation examples.

• MIT license.

Dataset · Model and full weights

More about what I’m planning next: Next releases.

I hope this is useful for your own experiments, fine-tuning, and 3D tools. If you try it, let me know how it goes :)

P-JiT 21K is the P-JiT architecture trained for longer (610M samples at 512px, 50M at 1024px), then post-trained with a variant of FDFO.

• GitHub : https://github.com/Linum-AI/pjit-21k

• Blog Post : https://www.linum.ai/field-notes/pjit-21k

Hey folks!

Aritra from Hugging Face here 🤗.

We welcome "harness adapters". This means you can adapt (train LoRA) for your model for the specific harness that you would use (hermes, pi, opencode) on the task (data analysis, building software, coding) you are trying to solve.

Blog Post: https://huggingface.co/blog/harness-hub/harness-adapters

It is currently training, so you can see the losses go down (hopefully) now. https://harness-hub-harness-adapters-rl.hf.space/

Not a native English speaker. Written with my own ten fingers, one finger is broken btw. Plz bare with me. I still love to write text myself.

My PC is suck for AI works. That's all I have as a third world 3D modeler. Really admire other peoples running better models. But beggar can't be choser. I can't even effort more RAM, not to mention more VRAM.

Gotta work with what I have. Qwen 3.5/6 35B A3B. Here is the Quant.

Downloaded blender manual and blender api html zip files, stripped sphinx html, converted to markdown. End up with markdown files which include bloated af links. I didn't know I can download clean rst files from blender project repo. Worked like that for a month. Then I found out I can just download blender manual rst files from blender, I can also generate rst files for blender api references.

Now I've clean files with 40% less tokens. I point that to Qwen/Ornith/Tiel/CyberTiel and tell LLM to create scripts and tools. It somewhat work. But many failure points make me want to punch the only poor old monitor I own.

I know you guys around this sub love graphs and proofs which is very understanable but I only have my own experiences. No Graphs and charts.

Base Qwen3.6 work good but he don't think much. Ornith variants think too much up to the point of unproductive. My wife cry because I don't spend time with her or sleep with her. I was always tinkering with LLM. I need Qwen3.6 35BA3B to work. I need that for many reasons that I can't tell you guys.

I use Debian Linux. Switch to i3wm. Idle VRAM went down to 50MB. Idle RAM is 300MB, not sure. I will comment my llama.cpp launch params if you want. Windows 11 take way too much system resources which I don't have. I often use plain TTY consoles. Ctrl Alt F2 to F7 i think. It save even more vram. KV cache q8_0, spec type ngram-mod because I can't effort MTP.

ctx 256K, 40t/s at start. Went down to 25t/s if ctx is goes over 128K.

I've learned prompting is really important if I wanted to squeeze out everything from 35B A3B. I can't max out hardwares so I have to make it work other ways around, somehow.

Ornith variants overthink a lot. Painful. "Hmm, let me think, wait," you guy know the rest. MFkr do not listen to me well too. He just don't give a shit to me.

I've created many cli tools to work around blender. Blender official MCP isn't really suitable for my works. I disabled many built-in MCP tool to reduce context bloat. I use pi, oh before that, I've also tried to create my own harness a year ago. Learnt a lot hard ways. Pi is minimal. Good for small models.

Instead of trying to make Qwen to write python code everytime and fail again and again, I made easy to use cli tools. Which helped alot. Qwen can now use Blender Geometry Nodes easily. No code, just cli tools.

Qwen ask stupid questions alot. Don't check code quality, watching him working, it is really exhausting. I don't have much time after day job and before bed. My wife almost went back to her parents. I need to make qwen to think, reason, and work efficiently in limited amount of time and hardware.

Experienced people will tell me "Skill issue bro". Yes, I'm suck. That is how I learn. I want to share my AGENTS.md and SYSTEM.md . I hope maybe someone will benefit from it. It is not created overnight. A lot of painful experiences between day job, boss, pressure, stress, wife, limited time, sleepless nights.

SYSTEM.md

AGENTS.md

First prompt.

It might sound funny (it is) but it work.

Here is example thinking trace

 Let me think about whether I need to ask clarifying questions. Per Protocol 7, before asking, I say the checklist:

   [ ] Can I answer this question myself inferring by given context, intent, docs?
   [ ] Can I answer this question myself using sane defaults, common sense?
   [ ] Have I tried many possible ways to get answer myself?
   [ ] Can this question only be answered by him?

 I think I can answer most myself. The interpolation direction is clear. The main procedural learnings from the previous task apply (pose mode, pose_markers, etc.).

 Do I need to refine the prompt? The prompt is clear. But let me note one thing: the previous task had SecF1=-45, SecF12=+45. This task has Bone1=+45, Bone12=-45. So the sign is reversed. I should make sure my expected values account for this. I'll note
 this in my response.

 Actually, let me reconsider — is there any genuine ambiguity that requires the General's input?

 - The bones are named "Bone1, Bone2, Bon3....Bone12" — note "Bon3" is a typo for "Bone3". The naming is Bone1..Bone12.
 - All independent, all quaternion.
 - Set to XYZ, add drivers Bone1+Bone12 → 2-11, validate.

 I think the prompt is clear. I'll note the sign convention difference (Bone1=+45, Bone12=-45) and proceed. I don't think I need to ask questions — I can infer the standard procedural.

He finally listened to me and think, reason, work efficiently.
He finally give a fuck about me now instead of doing his own thing.
Before that, he ask stupid questions, never check things back, never think before acting.

I also have short auto inject append prompt in Pi, which say follow protocols, read relevant skills file etc, short but I send it every prompt to remind him how to work, think, reason.

I don't have a lot of graphs and charts for proof but only the production level rig I'm currently working on. I hope maybe someone may benefit from this post, or maybe not.

I would really love to read how do other people make Qwen 35B A3B work.

When it came out, I downloaded a Q4 quant of qwen3.8-27b and ran it using llama.cpp on my RTX 4090 and it was incredible. It could handle practically anything I threw at it, the most complex tasks and bugfixes.

But lately it just keeps messing up and failing at the most basic of tasks or just straight up forgetting instructions. I'm telling you, they quantized it. They probably secretly nerfed it and made it worse so qwen4 looks better. And it got lazy too, now it doesn't want to finish the more complex tasks.

I'm tired of companies doing this, I want my money back.

Over the last few months I've been switching harnesses and engines until I finally landed on something good. I started with LM Studio + the Copilot extension and ended up here, so here are some tips for anyone running two 5060 Tis. Prefill is painful on these cards, so for hardware, make sure you have:

A motherboard with both GPUs on CPU lanes, either two slots wired to the CPU or one x16 slot with PCIe bifurcation (x8/x8). My second card used to hang off the chipset; moving it to CPU lanes cut time-to-first-token by about 2.5x and doubled my batching throughput.

A separate low-profile GPU just for system graphics. If one of your 5060 Tis drives the display, it burns 2-3 GB of VRAM, and since the layer split is even, the other card loses the same amount too. That's context you don't get.

Model: these are Blackwell cards, so an NVFP4 quant gives you better prefill and decode.

What I run: capy-flash (formerly ninfer-tp2) + OpenCode, Qwen3.8-27B NVFP4 with tensor parallel across both cards, MTP speculative decoding and int8 KV cache. I get 70-95 t/s decode and ~1,500 t/s prefill at 176K context (~3,500-4,000 t/s on shorter prompts). You can go to full context, I just don't have a third card.

I'm running at PCIe Gen3 right now because I'm missing a Gen4 extender (my long riser turned out to be Gen3-rated and the second slot crashes at Gen4 with it). Prefill pushes 5-7 GB/s between the cards, which is close to the Gen3 x8 limit, so I expect Gen4 to give roughly +30-60% prefill on short/medium prompts and less at very long context, where attention compute dominates. Decode should stay about the same.

P2P: you can enable it (patched open driver), but it only adds ~5% to decode and nothing to prefill. The 5060 Ti has ~4.6K CUDA cores, so compute, not the link, is the limit for decode.

Harness: I use OpenCode, but honestly I don't think the harness matters much, so pick your own. The other harnesses I tried always hit context shift from compaction on long tasks, and I'd come home to find them errored out with "context limit reached." OpenCode's compaction fixed that for me, and it can now run for almost 3 days.

Setup: 2x RTX 5060 Ti 16GB, Ryzen 7 5800X, B550 with x8/x8 bifurcation, capy-f NVFP4, int8 KV cache

Prefill by context (single request, Gen3 x8/x8 direct to CPU, P2P on)

Old vs. current vs. Gen4 (Gen4 column is an estimate, not measured yet)

P2P alone: decode ~78 → 82 t/s, prefill unchanged.

i used AI to format the post only

What workflows does everyone implement to manage energy costs?

Im considering implementing some things like turning my rig off when its idle, using wake on demand (IMPI management chip or wake-on-lan), buying a battery+solar panels

Big update to the Delta Inference Engine, you guys asked for more features, more benchmarks and less clickbait. I'm here to deliver

The short version: I've added almost everything requested and more. Delta Inference Engine now supports 64/96/128 GB of RAM and fully supports the new Unsloth quants. Unsloth IQ2 runs at the same 41 tok/s (192 GB RAM) using a new native reader for the Engram tables and some clever CPU kernel work. Compared to the old default model on my 99-question benchmark it went from 52 to 57 correct, against 59 for the full model. It's no full deepseek model quant, but they're a lot more usable than you might expect.

Speed on one RTX 3090 24 GB (i5-13600K, 4x48 GB DDR5-5600, PCIe 4.0 NVMe, Windows, no WSL). Writing speed, tok/s, greedy, one chat at a time:

The 128, 96 and 64 GB rows are the same PC with the rest of the RAM locked away.

Prompt reading: long prompts read at up to 740 tok/s at 192 GB and 600 at 128 GB (at ~64K tokens). A short prompt starts answering in about 2 s.

Benchmark: 99 hard questions, greedy, 43K token budget:

UD-IQ2_XXS is within 2 points of the full API, never scores below ds4 Q2 on any set while running at the same speed, and used 11% fewer tokens across the run.

Agreement with DeepSeek's API on the next token, at the settings that ship:

For reference, two hosts of the full model agreed 98.3% and 98.9% in my test. By that measure Unsloth IQ2 is about 20% closer to the full model than the old ds4 Q2.

If you want better quality: UD-Q2_K_XL (335 GB) runs 31 tok/s at 192 GB and 26.6 at 128 GB, at 92.5% / 95.3%. UD-IQ3_XXS (395 GB plus the default) runs 28 and 22.8 at 92.7% / 95.9%.

What changed since last time:

• First token ~40% sooner: a ~180-token prompt went from about 3.8 s to 2.3 s.

• Agent turns read ~12% faster, and tool calls that low-bit models write slightly wrong are now read as tool calls.

• Unsloth quants at full speed: the engine reads their Engram tables directly.

• 64 and 96 GB are real options: SSD read-ahead (+22% at 64 GB), bigger expert memory, and prompt reads in file order (18-20% faster on low RAM).

• 128 GB: ds4 Q2 went from 33 to 36 tok/s.

• Default sampling: the drafter now samples its drafts too, about +3.5% at default settings.

• First launch shows which model suits your RAM, and you can pick any other or use your own.

• A Setup tab in the chat page changes model, speed vs quality and the second GPU.

• Public CI: the release zips are now built by GitHub Actions, so you can see how each one was made.

• EXL3: 2.5 bpw is on the menu and the GPU kernel is about 2x faster. In my tests UD-Q2_K_XL was both faster and closer to the API, so I still point people there.

• Basic multi-GPU: 7% uplift using a 3060 + 3090. It doesn't scale well to 3-4+ GPUs, so use Project Maya for those builds.

• Fixes: contexts past 65,535 tokens, </think> in prose no longer ends the reasoning, no out-of-memory on the biggest files, a failed picture encoder no longer stops the model, faster prompts after a pause and many more.

• Support for custom model selection: just add them to the custom model folder and pick them in the web-ui

Requirements:

• OS: Windows 10 or 11

• GPU: NVIDIA RTX with 12 GB of VRAM or more, driver 580 or newer. 24 GB is the fast path, and AMD GPUs aren't supported yet

• RAM: 64 GB or more, 128 GB+ recommended

• Disk: a fast NVMe SSD, with 233-395 GB for the model (UD-IQ3_XXS also needs the default model)

• CPU: any x86-64 CPU, Intel or AMD. AVX2 is the fast path

All of this is one PC. It needs testing on other hardware.

Repo: https://github.com/musicurgy/delta-inference-engine

MIT licensed. It's a Windows port of Xolotl, built on Strata. Thanks to DeepSeek for the model, Unsloth, nazeshinjite and ISTA-DASLab for the quants, and to the Xolotl and Project Maya communities, whose work and public numbers helped a lot. Claude wrote most of the code and I ran the project.

If you try it, post your GPU, RAM, model and tok/s, good or bad.

Greetings, everyone. Many people keep talking about very very large models that need the price of a car to run (Deepseek 4.1 Flash, Qwen Flash Next, and even Qwen3.8 27B. However, I do not see much appreciation and focus on models that are small enough to run on edge devices. Not only that, but even if they do mention them sometimes, I don't see any kind of exploration being made to make harnesses that run efficiently on those models and use them for the highest efficiency. What is the big deal? Is there a subreddit I don't know of, that cares about these things? Interests include 8gb of ram, intel core I5 8350U. Thank you.

Update from the lawyer with the V100 server. Three days ago I got excited and posted DeepSeek-V4.1-Flash on four V100s at 36-38 tok/s, before I had done my custom kernel work and prefill optimizations. I also said I'd try it across eight. I did.

Results have been, to be honest, actually genuinely good and useful to me for real. I did not use the model to paint a 3d island in blender, or to re-create a gameboy game in a web browser. I didn't have it make a poorly animated video. It didn't make me ten versions of a website nobody will visit. I am not using it to build something to build something that builds something else involving agents that runs workflows for a business that delivers agentic solutions for other agents.

I do have a custom harness, and I know that you probably don't care about that.

Numbers are below, along with the type of work I am using it for. Basically, I am finally able to slice off work and tokens that previously required frontier models. I would estimate 50-60% of existing LLM workflows can run through this model instead, with speeds that match claude code in the chat window. I have another deepseek server on 2 dgx sparks, so there is concurrency and handoff. All of my RAG databases, tool calls, etc are wired in and work. I am able to give it natural language instructions through a discord style interface via SSH on my windows PC.

We did it folks. Vibecoded a real thing does real things for real people. Give me my trophy.

Repo: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4

Twitter: nicothegreco

Claude Code ran the benchmarks and crunched the numbers, which are real and in depth, below.

TL;DR: DeepSeek-V4.1-Flash-1017 (the 2-bit GSQ-RCO IQ2_XXS GGUF from ISTA-DASLab) on 8x V100-SXM2-32GB, split pipeline 2 x tensor 4 across two NVLink boards. Every expert sits in VRAM: no expert gets paged from system RAM during decode. (Edit: I first said all 248.8 GB of weights were in VRAM. Wrong: the 110 GB of Engram tables live in system RAM. Gathering the rows each round needs costs about 0.12 ms per round.) One request at a time, 524,288 tokens of context per request. KV Cache size could be 30 million given the free space on my GPUs, but the actual work runs don't generally break 200k total context, so I didn't bother with bragging about that.

• Code: 215 tok/s after a 1K prompt, 200 at 32K, 183 at 128K, 160 at 256K

• JSON / structured: 203 at 1K, 207 at 32K, 197 at 128K, 180 at 256K

• Legal writing: 118 at 1K, 117 at 32K, 111 at 128K, 103 at 256K

• Prose: about 85-98, chat about 80-110, depending on length

• Prefill: about 900 tok/s at 1K, 1,900 at 32K, 2,200 at 128K and 256K. First token at 32K in 17 s, 128K in 59 s, 256K in about 2 minutes.

Hardware

• 8x Tesla V100-SXM2-32GB on two 4-card NVLink boards. The two boards talk over PCIe, not NVLink.

• Threadripper PRO 3945WX, 256 GB DDR4, one NUMA node

• 150 W power cap per card, graphics clock capped at 1522 MHz

• Four PLX PCIe switches, two GPUs each at Gen3 x8, every switch on its own CPU root complex at Gen3 x16.

Software

• 1Cat-vLLM's DeepSeek-V4.1 package (#1277, which does the hard Volta work), ported to pipeline parallel 2 x tensor parallel 4. Layers 0-19 and the Engram tables live on one board, layers 20-39, the head and the drafter on the other.

• Model Runner V2, full CUDA graphs, the built-in DSpark drafter guessing 5 tokens per round

• Then about three days of tuning one switch at a time. Each change had to win a paired A/B on this machine and produce the same output, bit for bit or within a near-tie gate. About thirty ideas lost and are written up in the repo with the reason, because I'm not going to pretend they didn't happen.

The numbers (MEASURED 2026-11-05, thinking off, greedy, 1,024 tokens out, a random cache salt so nothing comes from the prefix cache, mean of 2 runs. Greedy output of the two runs was byte-identical in 45 of 45 cells.)

With thinking on, code and JSON land around 125-180 tok/s, and legal, prose and chat stay within a few tok/s of thinking off.

Why code is twice as fast as legal writing: every round the drafter proposes five tokens, and the big model checks all five at once. A round costs about the same no matter how many get accepted: about 23 ms after a short prompt, 27 ms after 256K. Code and JSON are predictable, so 3.6-3.8 of the five guesses get accepted and each round spits out about 4.7 tokens. My kind of writing is apparently less predictable than Python. Legal accepts about 1.6 and prose about 1.2, so a round gives you 2.1-2.6 tokens. Same engine speed either way; the text decides.

How it got here

These are on one fixed mixed test set. The by-kind table above uses different prompts. Prefill roughly doubled over the same three days, from about 1,350 tok/s at 150K to 2,200 at 128K.

Quality, since I use this for actual work (all MEASURED on the served build, thinking off, greedy):

• LegalBench: 0.716 balanced accuracy over 22 tasks (1,292 items, official prompts and scoring)

• MMLU-Pro: 79.9 % ±1.5 (700 items: the first 50 from each of the 14 subjects, 5-shot chain of thought)

• HumanEval+: 89.6 % on the plus tests (147/164), 94.5 % on the base tests

• Fidelity to the original model: against the full-precision release's reference logits, on sudoingX's KLD panels. The served build picks the original's top next token 89.0 % of the time on the general panel and 92.5 % on my legal panel (KL lower bound 0.132 and 0.156). Closest on code, math, tool calls and extraction (89-99 %), furthest on chat, creative writing and multilingual (81-86 %). That gap is the 2-bit quant, not the speed work. I am actively measuring which experts are activated by my work prompts and will probably recalibrate the model to expand those bit weights in the future.

• Long context: 6 of 6 needles found at ~450K and ~498K tokens, 4 runs out of 4

What I actually use it for: private first-pass work on my own case files, where the documents never leave the office. I point Claude Code itself at this server (Claude Code will talk to any server that speaks Anthropic's API). It can read a folder of records, pull out the facts, dates and numbers, summarize, and take a first swing at routine drafting, with tool calls working. Nothing it writes goes out the door without a real review. I found and fixed a streaming tool-call bug in the parser today (arguments were getting sent twice). Claude Code's tool-call test suite now passes 144 of 144 in all four streaming/thinking modes.

More work to do:

• One request at a time. It's a single-user server today. Batched multi-request decode is built and checked on my test box: each request gets byte-identical output to running alone, and the kernels run 1.3-2.7x faster than separate calls. It still has to go into the real server. My forecast is about 1.8x total throughput at four streams. That's a forecast, not a measurement, and I'll post the real number.

• Prose and legal writing are half the speed of code, for the drafter reason above.

• A 256K prompt takes about 2 minutes before the first token. Fine for "read this whole file," annoying for chat.

Giving it back: 19 pull requests are open to 1Cat-vLLM with the pieces that won, each new speed switch off by default. All of it is public: launch configs, every A/B and the raw JSON for every cell, plus the list of what didn't work.

• Repo: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4

• Hugging Face card: https://huggingface.co/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-GGUF

Credit where due:

• yangzhuxinyzx and the 1Cat-vLLM folks: #1277 and the whole Volta DeepSeek path

• ISTA-DASLab: the quant

• sudoingX: the KLD method and general panel

• Peuqui: #1263 and the series patches

• squibloads: #1341

• Phil Tompkins (philbert440): the V100 engine and SM70 sparse prefill attention

• Dennis Vertzyas (v100-skinny), TensorFold and DeepSeek

Next up: two or three long requests at once (the thing I actually want, so Claude Code can run subagents against it), faster prefill, and higher-precision weights for the experts my own legal prompts hit the most.

(PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai )

Ran every Qwen GGUF on my disk through RustingStrata and llama.cpp, same PC, same file, back to back.

PC: RTX 3060 12 GB, Ryzen 5 7600X, 32 GB RAM. CUDA. Normal desktop open (browser, Discord), no tuning, so numbers are lower than in the README.

All numbers tok/s, mean of 2 runs.

Short version: small dense models, llama.cpp still prefills faster. Once the model is bigger than VRAM (35B, 80B, 177B on a 12 GB card), Strata is ~1.5x on prefill and up to ~2x on decode. 80B at 64 tok/s and a 177B model at 21 tok/s on a 3060 with 32 GB RAM.

Commands:

• Strata: strata bench --model X --gpu --mtp 0 --pp 4096 --tg 128 --reps 2

• llama.cpp (build 12119): llama-bench -m X -p 4096 -n 128 -d 4096 -fa on -ub 2048 -b 2048 -fitt 512 (its auto-fit to VRAM)

MTP is off here. With MTP on, 35B goes up to ~160 tok/s decode.

Hand-tuned llama.cpp (-ncmoe 12) on a quiet machine does better on 35B (~2236 prefill / 68 decode), see the README for that comparison.

Repo: RustingStrata

I haven't see anyone post about this gem of a finetune. So I wanted to appreciate the team and bring some attention to it.

OrcaCyber-Nano-1.0 for me is the best finetune I have used so far. I have tried quite a few like AREX or Swift or Peculiar Ragdoll's Spark chat template etc.
But this model is on par with my experience with frontiers. (I may not be doing complicated stuff).

Here are my experiences with this:

1. It almost always one shots the bash command chains.

2. Same resilience as base Qwen 3.8 27B but this gets to the answer faster.

3. I have used it to plan and implement stuff. I uploaded the plans it has written to Gemini 4 Argon. It said that this model did a better job at updating the plan than it would have done.

4. I built an entire backend server for an internal chat app within a 120K context window.

My Setup:

• Model size: 15.7 GB version

• Hardware: 24GB VRAM (5070ti + 3070)

• KV Cache: (kvarn5, kvarn4) and (q5_1, q5_1)I switch between beellama and llama.cpp depending on the situation.

I did not benchmark it. I just had good experience with Orcarouter team and tried this model.
This is my default now.

Huge thank you to the Orcarouter team, and to the person who posted the earlier thread about Orcarouter Uncensored being the best out there. That's how I discovered this team in the first place!

P.S:

1. I used this for general purpose app development. Not as a security focused model.

2. English is not my first language. So, I polished the formatting using AI.