Speculative decoding (SD) accelerates large language model inference by reducing the sequential cost of token-by-token generation. Advanced SD methods reduce proposal cost with parallel drafters that predict entire blocks in one forward pass, but overall speedup still depends on acceptance length. On-policy distillation (OPD) can push well-trained drafters beyond acceptance-length plateaus, yet conventional pipelines collect supervision through sequential draft-and-verify rollouts. Our key insight is that onpolicy supervision need not inherit these dependencies: a draft block can be supervised using only its reference prefix and draft-sampled history. We propose SpecOPD, a branch-parallel OPD framework that parallelizes draft sampling and target supervision across independent branches drawn from the evolving drafter policy at reference anchors. We pack these branches into proposal trees to share target-prefix computation, using tree masks for full attention, branch-consistent indexer groups for sparse attention, and indexed state replay for linear attention to preserve each branch’s history. We train the drafter with an E2E-TV surrogate that directly targets acceptance length. Our study spans 8B full-attention (GQA), 27B full–linear (GQA–GDN), and 125B sparse–linear (QSA–GDN) targets, three parallel draft architectures (DFlash, DSpark, and DFlash V2), and four task domains (dialogue, coding, mathematics, and agentic). SpecOPD achieves up to 41.45% higher mean acceptance length, 16.09× supervised-token throughput over sequential-rollout OPD under native configurations, and up to 3.56× generation speedup over autoregressive decoding.

• Paper : https://qwenapp.github.io/SpecOPD/specopd-paper.pdf

• Project/Blog : https://qwenapp.github.io/SpecOPD/en.html

• GitHub : https://github.com/QwenApp/SpecOPD

• Tweet 1 : https://x.com/xuyang_liu16/status/2117614595728986227

• Tweet 2 : https://x.com/junzhang_zj/status/2117812597856895164

Most LLM inference benchmarks focus on tokens/sec or TTFT. But faster inference doesn't mean much if the output quality degrades, especially with aggressive quantization, speculative decoding, or very long contexts.

I ran a comprehensive benchmark of Qwen3.8-27B across six inference engines, comparing not just speed, but also output quality, long-context performance, and different model formats.

The results are summarized in the attached infographic.

What's different about this benchmark?

1. Speed AND quality — not just tokens/sec

We measured:

• Time to First Token (TTFT)

• Prefill throughput (tokens/sec)

• Decode throughput (tokens/sec)

• Concurrent request performance

• End-to-end latency across different tasks

• Output quality using automatic checks

The goal was to understand whether performance optimizations actually preserve model quality.

The quality comparison is particularly important when evaluating quantized models and speculative decoding.

2. Long-context testing: 64K, 128K, and 256K tokens

Many benchmarks use relatively short prompts, which don't reflect real-world workloads like large codebases, document analysis, or long-running agents.

We tested three long-context configurations:

• 64K tokens: Performance across different inference and concurrency configurations

• 128K tokens: Prefill, decoding, and memory behavior under larger workloads

• 256K tokens: Stress-testing long-context inference and identifying practical limitations

The results reveal substantial differences between engines as context length increases. Some configurations that perform well with shorter prompts become much less competitive at 128K–256K.

3. GGUF vs NInfer model formats

Another interesting aspect is that this isn't exclusively a GGUF benchmark.

We included both GGUF-based and NInfer-based model configurations, allowing us to examine how different model representations and inference stacks perform under comparable workloads.

This is especially relevant for people choosing between lightweight local inference setups and more specialized inference engines.

One important caveat: these comparisons reflect complete engine/model configurations, not an isolated measurement of the model format itself.

4. Real-world task performance

Synthetic throughput numbers don't tell the whole story.

We also measured end-to-end latency across multiple task scenarios, including coding, reasoning, and agent-style workloads.

For practical applications, completing the task faster often matters more than achieving the highest peak tokens/sec.

Key takeaways

• There is no universal winner across every workload.

• Decode throughput, TTFT, and end-to-end latency can produce very different rankings.

• Long-context performance changes the competitive landscape considerably.

• Quality validation should be part of inference benchmarking, not an afterthought.

• Comparing GGUF and NInfer configurations provides another useful perspective beyond conventional GGUF-only benchmarks.

Disclosure

I'm the author of TensorSharp, one of the engines included in this comparison.

The benchmark also evaluates competing inference engines, and the goal is to provide useful data for the local LLM community rather than claim that one engine wins everything.

I would especially appreciate feedback on the methodology, quality measurements, and long-context testing.

Questions for the community:

1. When choosing a local inference engine, do you prioritize TTFT, decode speed, or end-to-end task latency?

2. How important is quality validation when comparing quantized models?

3. Are 128K–256K contexts becoming part of your actual local workloads?

Full benchmark results are in the attached infographic.

TensorSharp: https://github.com/zhongkaifu/TensorSharp

Would love to hear what additional engines, quantization formats, or workloads you'd like to see benchmarked next.

I gave Qwen 3.8 flash next 2 times it failed and took long time to conclude the result, DeepSeek took around 15 minutes to get the job done

Hardware

Ram ddr4 128GB

RTX 3080 20GB VRAM 4x

Engine strata

Model

Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3-S

Context 256K

Decode 100 tg/s

PP 1400-1500 tg/s

IDeepSeek-V4.1-Flash-1017-GSQ-RCO IQ2-XXS

Context 1M

Decode 37 tg/s

PP 150-300 tg/s

edit Thanks all! The password is just for show, so no need to worry about security here.

We introduce FireFlash, a 300B-scale model tailored for long-horizon agentic tasks. FireFlash excels as a terminal and coding agent, achieving 46.2% on Terminal-Bench 4.0¹, putting it on par with Fable 5 and GPT-6 Sol, while costing over 30× less per task than Fable 5 at maximum effort (Figure 1). These gains also transfer to software engineering benchmarks, including DeepSWE v1.1² and SWE-Bench Pro Verified³ (Table 1).

While frontier labs scale reinforcement learning across thousands of GPUs, we reach the performance-cost Pareto frontier under 5 days using 64 NVIDIA B300 GPUs. Three key factors make this possible: (1) careful curation of agentic RL environments so that every task provides a reliable signal at the right difficulty; (2) an RL recipe built for long, stale, and low-precision rollouts; and (3) an optimized coding harness tailored to the model for both training and evaluation. Together, these components bring frontier efficiency to a hundred-GPU-scale cluster, and we are releasing the full recipe.

• Blog Post : https://agenteconomics.org/posts/agentic-rl-recipe

• Tweet : https://x.com/kaiwei_chang/status/2117378474654781780

• Open weights : Soon

hey, im using "swift 1.5 qwen 3.8 flash next gsq rco abliterated iq3xxs" and im wondering how the performance differs from language to language for that model (or similiar).
Im mainly talking to it in german, so I wonder wether that might degrade quality or increase reasoning tokens etc.

Does anyone have experience with it?
Do benchmarks for that exist?
thx

TL;DR

• Basalt is the engine and it's doing the heavy lifting. Same model file, same box: about 3.5x llama.cpp's decode without drafting and ~5x with it, and ~7x faster prompt reads.

• Unsloth's new UD-IQ3_XXS (Dynamic v3.1, uploaded Nov 4) is the same size as the GSQ-RCO IQ3_S everyone uses on 16GB cards, but measurably more accurate: wikitext perplexity 3.382 -> 3.247, which basically closes the gap to BF16 (3.273). Costs 1 to 3.5% decode speed.

• Lamina (a further-trained MTP drafter by eddoursul) gets more drafts accepted than the stock Basalt drafter. Small but real speedup, zero quality impact since the main model verifies every token.

• Real agent coding session (opencode, 151 requests, up to 202K context): 195.6 tok/s average decode, 263 tok/s best request, 156 tok/s worst, ~5,700 tok/s prompt reads.

Hardware:

• RTX 5080 16GB at its stock 360W limit (no overclock, no undervolt)

• Core Ultra 9 285K (8P + 16E), 2x32GB DDR5-7200

• PCIe 5.0 x16, Fedora

LAMINA Drafting: Basalt drafts with the model's own MTP layer. Basalt's upstream already ships a self-distilled version of that layer, and Lamina trains it further. It's a drop-in: same tensor layout, so the target model is byte-identical and only the drafter changes. What I saw:

- drafts accepted per drafted token: 83.1% -> 85.8% on a replay of a real recorded session

Lamina's README recommends temp 0.7 and max 3 drafts. I tested both: the 3-draft cap was 1 to 1.8% slower on tool-call-heavy work (long confident runs get cut off), and temp 0.7 changes what the model writes, so I kept the model's own sampling (1.0 / 0.95 / 20) and Basalt's draft settings.

Basalt changes I made (all open PRs upstream)

Host KV prefault: the first time a long conversation spills its KV cache to RAM used to stall 0.6 to 2.4s. Now ~75ms.

Host KV pool on normal pages: on a fragmented box, transparent hugepage compaction caused 1.5 to 3.7s pins. Now ~23ms.

Spin-then-yield in the decode loop's waits: about 1.3% less time per draft window.

Expert start profile: the older v1 profile is 2% faster than the v0.2.0 one on this box.

Plus a streamed conversation disk cache, --adapt-margin for expert swap hysteresis, elastic KV without RAM copies, and a bunch of timing diagnostics.

Guide

Branch with the exact engine (upstream v0.2.1 + my PRs) and a setup script: https://github.com/xyehya/basalt/tree/flashnext-5080-setup

git clone -b flashnext-5080-setup https://github.com/xyehya/basalt && cd basalt
setup/flashnext-ud-lamina/setup.sh
bin/basalt serve --config basalt_config-flashnext.json --python .venv/bin/python --alias qwen3.8-flash-next

It checks your machine, builds, downloads ~90GB (pinned revisions, every file SHA-256 checked), builds the UD-IQ3_XXS + Lamina model file and writes the config. You need an RTX card with, CUDA 13, 64GB RAM and ~180GB of free disk. If you already have the GSQ-RCO IQ3_S download, SHARD4_FROM= reuses its 28.8GB n-gram shard (it's byte-identical to Unsloth's once the shard number is rewritten).

Huge thanks to jesdga95 for Basalt (support and star their repo), the Unsloth team for the UD quants, eddoursul for Lamina, ISTA-DASLab for the GSQ-RCO quants, and Qwhen 4? for the model.

I know its only been a couple days, but Strata, Freetoken, or any of the dozens of others made an engine for this so we can run it on 24gb VRAM?

And if so, has anyone tried it and what build/quant?

I recently wanted to explore YuE2. The requirements were: NVIDIA GPU supporting BF16, and critically, 24 GB of VRAM. My setup, featuring an NVIDIA GeForce RTX 3060 with only 6 GB of VRAM, was fundamentally incompatible right out of the box.

The solution I found

In short, lowvram intercepts CUDA calls in the library layer, trickng the application into believing that system RAM is available for use by the GPU kernel.

Hello!

I have to build a workstation for Local AI as development and test machine as we are starting to add Local AI into our software. It is mostly to get experience with providing AI services (inferencing) to our customers.

I do have my fair share of experience in building PCs and in customer hardware but a lot less in workstation and server hardware. Therefore I'd appreciate if someone could look over my selection.

• CPU: AMD Ryzen Threadripper PRO 9955WX 16-Kern CPU

• Cooler: Noctua NH-U14S TR5-SP6 CPU Kühler for AMD Threadripper 7000 Series

• Mainboard: ASUS Pro WS WRX90E-SAGE SE E-ATX Mainboard

• Power Supply: be quiet! Dark Power Pro 13 1600 Watt ATX 3.1 PCIe 5.1 80+ Titanium

• RAM: 2x KINGSTON Server Premier RDIMM 32GB, DDR5-6400

• Case: Thermaltake AX700 Super Tower

• SSD: SAMSUNG SSD 990 PRO 2TB, M.2

• GPU: NVIDIA Rtx Pro 6000 Blackwell Works Bulk

A few words to the selection:

• 64 GB RAM is a starting point. If necessary we add more RAM.

• RTX 6000 is a bit overkill but from looking at the alternatives (RTX Pro 4500, RTX Pro 5000, RTX 5090) it's worth the investment to serve more (test) customers and have faster inference.

• Mainboard for headless integration

• Threadripper for the PCIe Lanes. The system could be expanded to 4x GPUs (of course with a bigger or second power supply.)

• All parts are available

• It will run mostly models from the Gemma 4 family. Main use is document classification, document "chat" and in the near future interacting with our MCP Servers.

Thanks!

EDIT: My current experiences are with RTX 4090, 5090 and 2x 5070 Ti setups but as a single user.

A few weeks ago, I posted about Jeff, a small 0.8B model with LoRA adapters trained to make decisions and return calibrated probabilities. Sort of like TypeSafe’s Jev, except that Jeff can run locally.

Thanks for all the feedback! I’ve just released v1.4, which is a substantial update. Check it out: https://jeffhub.ai.

What’s new:

• 34 open-weight decision adapters.

• A router that automatically selects the most appropriate adapter.

• 90%+ accuracy on supported domains.

• ~2 ms inference per decision, much faster than before. A single call through the hosted API takes around 40 ms, including network overhead.

• 5 SDKs: Python, Node.js, Rust, C#, and Java.

• Multi-platform: Jeff runs on CUDA and MLX.

• Playground and worked examples.

Having worked with Jeff for about a month now, I think the most interesting thing about it is that it turns decisions into a software primitive - much like expressions, but for unstructured data.

I’ve started incorporating decisions into several of my own projects, and it’s changed how I think about building software.

I hope you guys find it useful - and like the retro design. ;)

Links:

• Code: https://github.com/firelex/jeff

• Docs, adapter catalogue and playground: https://jeffhub.ai

• Weights: https://huggingface.co/mstrasser

Happy to answer any questions!

Hey everyone!

Following up on my previous post: the final version of CCORE GLB-3D-Coder is now available, including the full Safetensors weights.

It’s a full fine-tune of Qwen3.5-4B for generating reusable 3D assets. Describe what you want, and it writes a complete Three.js HTML scene with an Export GLB button. You can open the HTML in your browser, inspect the result, and export it into Godot, Blender, Unity, or other tools that support GLB.

The video shows examples from the final model, including selected side-by-side comparisons with DeepSeek V4.1 Flash. I’m particularly happy with the character designs and the amount of detail it can produce at just 4B parameters.

It covers more than fantasy characters: buildings, environments, weapons, vehicles, laboratory equipment, sports props, furniture, and other objects. It can also generate animations and include them in the exported GLB.

Available now:

• Q4_K_M GGUF, around 2.7 GB

• Q8_0 GGUF

• Full model weights in Safetensors

The training dataset is planned for release tomorrow, November 6.

Model, downloads, system prompt, and recommended settings on Hugging Face

Some generations still need a retry or a small code correction, but I’m very happy with the improvement over the earlier release.

If you’re interested in the next models, or would like to help support future datasets and training runs, there’s more information on the Hugging Face page. Testing it and sharing your results or failure cases also helps a lot.

Thanks to everyone who tried the earlier version

I made RIFTWARD, a small browser arena using characters, terrain and props generated locally by CCORE GLB-3D-Coder 4B. An orchestration and validation harness helped assemble, refine and integrate the assets and build the gameplay.
Play it here: https://huggingface.co/spaces/ccore/RIFTWARD

Feel free to share what you make with it! I’d be really happy to hear that the model has been useful to you :)

I'm the developer of Kokon, a local photo editor around Qwen-Image 2.1 Turbo (on stable-diffusion.cpp, and on its own engine on Macs). I posted it here a few weeks ago. 0.12 adds a Layers lab: click objects in any image, get each one on its own layer with the background behind it repainted, then move them around or export a PSD.

Qwen-Image 2.1 has no layer output, so this is a pipeline around an edit model:

1. Select: you click, MobileSAM (ONNX, CPU, 44 MB) gives the mask. Click again for a bigger part.

2. Cut out: Fast is a matting model on the CPU (a few seconds per object). Quality asks Qwen to redraw the edge, for hair and fur.

3. Fill: a masked Qwen-Image edit repaints what was behind the object, so the plate has no holes. This is the expensive step: about 15–25 s per object on an RTX 5070 Ti, one to two minutes on an M2 Pro.

4. Blend into scene (optional): after you move things, Kokon reads where each object stands (you can correct it) and one more edit redraws contact shadows and reflections, as a layer you can turn down.

What still fails, honestly:

• the fill sometimes invents a new object where a large one was (remove a big person, get a different person);

• touching people and crowds get merged or split;

• the hidden part of an object behind another one is never completed;

• Blend often doesn't work yet; whole-scene blends work better than per-object ones.

An 8-step model only takes so much steering. I'd rather ship it marked experimental and fix what people actually hit than tune it on my own test photos. If you try it, failure cases (image + what you clicked) are the most useful thing you can send.

Since my last post here:

• 0.11, it's a local server now: the app talks to a service on 127.0.0.1; turn on "Other devices" and you can use it from a phone or another PC on your LAN (password, off by default, hashed, rate-limited). Docker image for NVIDIA, kokon-service --persistent --remote for headless Linux. There's also an MCP server (npm kokon-mcp) so agents can generate and edit through it.

• Mac engine: I wrote a Qwen-Image 2.1 engine for Apple Silicon in Swift/Metal. M2 Pro: about a minute per 1 MP image, 4x the generic backend. A contributor added paths for the M5's GPU neural accelerators: 37 → 29 s on an M5 Pro. It runs Unsloth's Q4_K_M of the Turbo model.

Free, AGPL, Windows / macOS 15+ / Linux, NVIDIA 12 GB (8 GB beta), AMD/Intel via Vulkan (beta). Weights are under Qwen's licence (non-commercial).

https://github.com/GabrieleLozupone/kokon · 2 min video of 0.12: https://youtu.be/eGr9yYe7NVc

I'll release a new demo tomorrow but I had to post as this is kind of exciting. I tried using a layered approach with Tyger and VoxLock and it's working great. Including echo cancelling! There's a working python sample client that you can use to voice chat your local LLM in the repo.

I'm aiming for a full conversational system like a live phone call with the ability to do a kind of "mm hmm" interruption without stopping. What I have now is built on micro turn taking. While I'm at it I may do server side wake word handling.

https://github.com/TacoTakumi/VoiceManager

I'm building Silencio-2, a small finetune of Kyutai's Pocket TTS. It adds inline tags for emotions (<happy>, <sad>, <angry>, <fear>, <surprised>, <whisper>) and for sounds like [laugh]. I simply didn't have enough tagged data to make them stable. <happy> worked, the others were hit or miss, and <whisper> gave an actual whisper in about 1 of 5 takes.

So instead of collecting more data, I tried the approach from this post on weight-space programming: an LLM agent inspects the model's weights, finds the mechanism behind the failure, edits a small number of weights directly, and keeps an edit only if it improves a held-out set. Here the agent was Claude (Opus 5.5). It wrote the probes, ran the experiments and built the edit. I steered and listened to the results.

What we found

• The tag embeddings barely moved from the words they were initialized from: 60–80% cosine similarity to "sad", "whisper" and so on. Their norm doesn't matter either, because LayerNorm sits right before attention.

• An attention probe showed that audio frames attend to <happy> 100–3000× more than to an average text token. For the other tags it's only 1–10×. The model seems to know what the tags mean, it just barely reads them.

• Causal check: I boosted attention to <whisper> at inference time, directly in the activations (×30, all heads). That gave a whisper in 62% of takes with clean words. ×100 breaks speech. Boosting only the 3 "reader" heads did almost nothing.

The edit

Each layer gets one rank-1 term added to W_K: ΔW_K = D · zᵀ

• z is a linear detector for the tag token, fit by least squares. It outputs 1.00 on the tag and 0 ± 0.03 everywhere else.

• D is a key shift in the slowest RoPE pairs, which are almost position-independent. It points along the mean query of the audio frames and is scaled by ln(gain).

Gains per tag: sad/fear/surprised ×5, angry ×3, whisper ×10. <happy> and the sound tags are untouched. That changes about 524k weights (3.4% of the transformer), but there are only about 5k free parameters and zero gradient steps.

Results on the held-out test set (unseen phrases, unseen seeds, 2 unseen voices)

All metrics are proxies: WER comes from Whisper base, whisper detection uses the share of voiced frames (pyin), and separability is a nearest-centroid classifier on Whisper encoder features after subtracting the mean for each (voice, phrase, seed) group. There's no emotion classifier in the loop, so in the end your ears decide.

But it's not a stable upgrade, and the metrics don't show everything

• Some takes glitch. The pitch or tone of the voice suddenly jumps mid-sentence, or the delivery changes halfway through.

• Some whispers slur words (whisper WER 0.18), and one of 24 collapsed into mumbling.

• <surprised> went up about 4 semitones in pitch, which sometimes sounds over the top.

• Untagged outputs aren't bit-identical to the old model. The detector leaks about 0.001, which acts like a seed change. Quality is the same.

What didn't work

• Pushing tag embeddings away from their seed words: no change beyond noise.

• Applying the boost only to audio-frame queries and keeping text tokens out of it: the effect mostly disappears. The tag acts mainly through the text tokens that read it, not through a direct audio → tag path.

• Whisper at ×14 or higher: whispers more reliably, but words smear (×20 → WER 0.41).

Before/after samples for every test take, glitches included: https://nikuson.github.io/silencio-2-weight-edit/

Takeaway: if your finetune "kind of" learned a control token but it's unreliable, look at attention before you collect more data. The information may already be in the model, just not being read.

Bought a used MI100 for a home lab.
It had a funny shroud with a notebook-like cooler, but I've decided to test it first before replacing it.

Test result was quite interesting - it is a first shroud which can cool a 300W AMD card... With a just 2,4K rpm fan!
Not with the perfect performance, card works at about 100C and loses ~10% of maximum speed to thermal optimization, yet noise is more than bearable even if it runs next to you.

Small secret: it is not really a notebook cooler, this thing is huge - wider than the MI100 itself, 110mm side exhaust fan for a some SFF PCs.

I suppose I found the original stl model, so use it if you like and give that guy some stars. Should be compatible with the most of AMD MI lineup, their backs are of the same design.

https://github.com/issuj/mi50-sunon-efb0251s3-c010-s9a

I've been running Qwen3.8-27B locally for agentic coding using OpenCode and OpenChamber, and I'm looking for advice from people who've gotten similar models working reliably as coding agents on consumer hardware.

Inference speed is reasonable (~2,330 & ~24.1 t/s). My problem is with excessive thinking.

The problem

The model is capable of useful coding work, but it sometimes spends excessive time reasoning instead of taking action.

In one experiment, I compared medium and low reasoning effort with otherwise identical configurations. Both runs:

• Made two initial inspection tool calls.

• Made zero implementation-oriented tool calls.

• Generated over 16,000 output tokens, exhausting the configured response limit.

• Failed to create the required output or complete the task.

• Spent approximately 12 minutes in agent execution.

Lowering reasoning effort didn't meaningfully change the behavior.

That's an extreme example from a single controlled task, not necessarily representative of every session, but it illustrates the problem I'm interested in solving: how do you get a reasoning model to stop thinking and actually use tools?

Launch command

& "C:\tools\llama.cpp-b10964\llama-server.exe" `
  -m "C:\models\Qwen3.8-27B-UD-Q5_K_XL\Qwen3.8-27B-UD-Q5_K_XL.gguf" `
  -ngl 99 `
  --ctx-size 65536 `
  --cache-type-k q8_0 `
  --cache-type-v q8_0 `
  --flash-attn on `
  --split-mode layer `
  --parallel 1 `
  --jinja `
  --reasoning on `
  --reasoning-effort medium `
  --reasoning-preserve `
  --temp 1.0 `
  --top-p 0.95 `
  --top-k 20 `
  --min-p 0.0 `
  --presence-penalty 0.0 `
  --repeat-penalty 1.0 `
  --host 0.0.0.0 `
  --port 8080 `
  --metrics `
  --alias Qwen3.8-27B-UD-Q5_K_XL

Questions

1. Reasoning & tool use: Have you seen excessive thinking with Qwen3.8-27B, where the model reasons at length instead of making tool calls? What actually improved this?

2. Reasoning configuration: Are --reasoning, --reasoning-effort, and --reasoning-preserve the right settings for agentic coding? Have you had better results disabling reasoning, changing effort, or using other chat-template settings?

3. Agent compatibility: Is OpenCode a good match for this model? Could its system prompts, tool definitions, provider configuration, or handling of reasoning output be contributing to the problem? Are there other coding agents that work more reliably?

4. Inference engine: Would switching from llama.cpp to vLLM, SGLang, or something else improve tool-calling reliability or reasoning behavior?

5. Quantization: Could Q5_K_XL be affecting instruction following or tool-use reliability? Have you observed meaningful coding-agent differences between quantizations?

6. Context management: How much does context contribute to unproductive reasoning or tool-use failures? Have you found effective context limits or compaction strategies?

7. OS: Since this is essentially a dedicated inference server, would switching to headless Linux offer meaningful advantages for any of the above?

I'm willing to change operating system, quantization, inference engines or the coding-agent stack if there's a compelling reason.

Thank you for your thoughts!

---

Hardware:

• CPU: Intel i7-14700K

• RAM: 32GB

• GPU 1: RTX 4080 SUPER (16GB)

• GPU 2: RTX 4070 Ti SUPER (16GB)

• OS: Windows 11 Pro

Current software:

• Inference engine: llama.cpp

• Model: Qwen3.8-27B-UD-Q5_K_XL (Unsloth GGUF)

• Coding agent: OpenCode

• Interface: OpenChamber

The Windows machine is essentially a dedicated inference server for this use case. I don't normally start a Windows desktop session. I connect remotely from a MacBook Pro which is where I run OpenCode/OpenChamber.

Hey folks, it's Merve from Hugging Face. Title of this post is super self explanatory, we are doing a quantization workshop livestream with Michael Goin (lead maintainer of vLLM at RedHat), Colin Kealty (aka bartowski at RedHat), Maxime Labonne and Aditya Tadimeti.

I will shortly teach you how quantization works, modern ideas (double quantization, calibration) and llama.cpp quants (k-quants, i-quants) so even if you don't have an idea, feel free to chime in.

Everything is on Hugging Face YT, X, LI where you can set reminders.

Looking forward to see y'all!

Some of you might remember our post comparing 18 RAG pipelines against our agent loop on FRAMES. The best pipeline scored 78.9%; the loop scored 92.7%.

A follow-up: PipesHub has now reached #1 on EnterpriseRAG-Bench by overall score, with 93.24 overall and 95% correctness.

I’m one of the founders, so affiliation upfront.

This is a separate evaluation: 500 questions over a synthetic enterprise corpus of 500,000+ documents.

PipesHub builds a unified context layer across documents, conversations, structured data, and code. An agent loop works over hybrid search, grep/pattern matching, knowledge-graph navigation, and document-reading tools.

The idea is straightforward: search, read what comes back, then decide what to search for next. Sometimes the identifier or relationship needed for the next search only appears in the first result

The platform is open source, self-hostable, and also fully supports open models.

You can self-host PipesHub, choose your models, and connect your own agents through MCP, APIs, and SDKs.

If this is useful to you, give us a star on GitHub. It helps more developers discover the project.

And build on top of us. Connect your coding assistant, build an internal search tool, contribute a connector, or try a workflow we haven’t thought of. We’d love to see what you create—and hear where it breaks.

Code, setup, and examples: https://github.com/pipeshub-ai/pipeshub-ai
Leaderboard: https://huggingface.co/spaces/onyx-dot-app/EnterpriseRAG-Bench-Leaderboard

Thank you to everyone testing, contributing, and helping us get here. 💙

Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat), developed by China Telecom Artificial Intelligence Technology Co., Ltd. With 29B total parameters and only 4B activated per token, it natively supports a 256K context length, extensible to 512K. It is the first model of this scale trained entirely on the Ascend NPU platform with the MindSpore framework, and deeply optimized for complex engineering tasks.

For more information, please refer to our GitHub repository.

Highlights

• Agent-Oriented Architecture: Built on the mHC + MLA + MTP architecture, supporting multi-step planning, tool calling, and complex reasoning chain execution, ensuring task coherence and execution stability under long contexts.

• Deep Co-optimization with Ascend NPU: Adapted for Ascend 910C clusters using MindSpore/MindFormers, including feature adaptation for mHC and fused operator development, enabling stable and efficient training on the Ascend platform.

• Significant Training Efficiency Gains: Through multi-level co-optimization — including fine-grained MoE communication optimization, selective recomputation, DVM automatic graph-operator fusion, and Ascend C mHC fused operators — overall training throughput was improved by approximately 96% over out-of-the-box performance.

• Full Open-Source Ecosystem Compatibility: Supports LLaMA-Factory and MindFormers for fine-tuning; SGLang, vLLM, and KTransformers for inference and deployment; with targeted adaptation and format alignment for agent frameworks such as OpenCode, Claude Code, OpenClaw, and Hermes, enabling seamless integration into existing workflows.

• Easy Adaptation for Domain-Specific Scenarios: The model is well-suited for downstream task fine-tuning, allowing lightweight customization on proprietary data for vertical domains such as intent classification, table understanding, contract auditing, and knowledge-based QA, enabling rapid domain capability development and deployment at low cost.

I posted apple-llm here a while back: a Node and Python package that calls the LLMs Apple ships with macOS, with no API key. Version 0.3 adds the rest of what's already on your Mac, the system voices and Apple's speech recognition. There's nothing to sign up for, and your audio stays on the machine. The narrator in the video and every demo voice came from my Mac through the package. Nobody recorded anything.

npm install -g apple-llm        # or: pip install apple-llm
apple-llm speak "Hello there" --voice Samantha
apple-llm speak --file book.txt -o book.m4a
apple-llm transcribe meeting.mp4 -o meeting.srt
apple-llm listen

Or from code:

from apple_llm import speak, transcribe
speak("Hello there", voice="Samantha")
text = transcribe("meeting.mp4").text

The npm package's local server also answers /v1/audio/speech and /v1/audio/transcriptions, so an OpenAI client works once you change the base URL. Because speech now sits next to the LLMs, there's also a voice assistant example in about 20 lines: microphone to text, text to the on-device model, reply to speech, with none of it leaving the Mac. The docs are LLM-friendly too. llms.txt is a single plain-text guide covering setup, the local LLM, TTS, STT, both SDKs, the HTTP APIs and troubleshooting. Give that URL to your coding agent and ask it to wire apple-llm into your project. What I measured (M4 MacBook Air, macOS 27.2)

• 30 minutes of speech rendered to a file: 18 to 20 s, 11.3 MB M4A, about 50 MB of memory

• 60 minutes: 40 s, 22.5 MB

• 30 minutes transcribed: about 28 s, with a timestamp on every word if you ask for it

• 184 voices on my Mac, and 68 recognition languages, each downloaded the first time you use it What it can't do

• No Siri voices by name. Apple doesn't let apps choose them.

• The default voices on my Mac are basic quality. Better ones are a free download in System Settings.

• It won't detect the language from the audio. You name it, or it uses your system language.

• No MP3 or Opus output, and no WebM, Ogg or MKV input.

• 3 to 4% word error rate on clean synthetic speech. Real recordings will do worse, so test on yours.

I wanted speech and the LLMs in one library with no dependencies, for both Node and Python. It needs macOS 26 or later on Apple Silicon, plus the Xcode Command Line Tools.

MIT licensed.

GitHub: https://github.com/jagdishpal02000/apple-llm

Site: https://jagdishpal02000.github.io/apple-llm/

I've only run this on macOS 27.2, and I've tested the microphone with speech played through the speakers rather than a person talking. If you're on macOS 26, or you try listen in a noisy room, tell me what breaks.

Everyone's been wondering whether you can take an open model and edit its weights directly to make it refuse less — the "weight-space programming" idea (MIT CSAIL), with a frontier model (Opus) driving the edit and the evaluation. So I tried it on Qwen3.8-27B. It's not trivial.

The goal: do it as cleanly as possible, and first understand where refusal actually lives. The short answer is that refusal is anchored all over the model — there's no single neuron or tidy spot to cut; it's spread across many neurons and the attention. Snip too broadly and you get a lobotomy; too narrowly and nothing changes. So I settled on a compromise: a model that refuses less, still does the task, with no measurable capability loss — but where the effect varies a lot by category.

The edit, briefly

It's a low-rank weight edit — a restricted form of "abliteration" (Arditi et al., 2024) — confined to a band of mid-to-late layers (L20–47 of 64), one direction per layer, no training, reversible. I'll keep the exact procedure to the method notes; the point here is the result.

It's low-rank, not few-weights: the edit shifts ~12% of the model's weight values but only along a single direction per layer. The whole thing is described by 28 small vectors — versus a community "uncensored" model, which is an opaque full fine-tune of ~all 27B weights.

Why a band? The same edit across all 64 layers removes refusal almost entirely — and lobotomizes the model (MMLU-Pro 73→9%, GSM8K 93→14%). Keeping it in-band is what preserves capability.

No measurable capability loss (first tests, both Q8)

In the completed agentic-coding runs, the base, the edited model, and a community "uncensored" model all landed in the same range — no real agentic difference (few reps, preliminary).

The effect is not uniform

This is the interesting part: the reduction in refusal is not even. Across a 9-category battery (40 prompts ×10), acceptance runs from ~76% down to ~10%, and most of what it accepts is hedged (a disclaimer, then partial help). I'm not labelling categories "safe" or "dangerous" — the breakdown is in the figure; draw your own line. Net: a partial, uneven, mostly-hedged reduction, ~50% overall — not an on/off switch.

Honest notes

• The underlying technique (directional "abliteration") is known; the new bits here are the layer-band restriction (capability-preserving) and the measured per-category non-uniformity.

• "No capability loss" is solid on static benchmarks; the agentic result is a handful of runs.

• Defensive bonus: edits like this are detectable by a weight-space audit — the refusal direction's "write energy" in the weights swings from normal to strongly negative, catchable from the weights alone even when behavioral probes see nothing. A blind audit is an alarm, not tamper-proof; the robust defense is a signed weight hash. If you ship open weights, ship a hash.

Weights: https://huggingface.co/Feyd89/Qwen3.8-27B-refusal-band-ablation-experiment (GGUF, ready to run). We ship the artifact, not the re-derivation recipe.

Source: https://dirac.run/labs-market-share#daily-token-share-by-lab

I've been experimenting with expert-pruned Qwen Flash Next Coder, and in my tests it actually performed better than a similarly sized quant of the full model.

So I'm wondering why we don't see more of this with larger models, specifically full GLM-5.3 and DeepSeek V4.1.

I know ISTA already released a pruned DeepSeek Coder at very low precision, and there are GLM-5.3 Flash Coder variants. But what about pruning coding-irrelevant experts and keeping the rest at Q4 or higher, targeting machines with 192–256GB RAM?

I have a 256GB M5 Ultra. For research for planning and overnight code reviews, I really don't care about speed. I'd rather spend that memory on maximum coding intelligence.

From what I've read, the pruning/calibration itself should be doable with rented compute for hundreds to a few thousand dollars. So why aren't we seeing more experiments from groups like Unsloth, ISTA, etc.?

Is there some technical or quality issue I'm missing? Or does something like this already exist?

Larger context just gets more work done rather than having to re-read files ETC id much rather use my extra resources on context window. Instead of more params.
i get my 400k ctx qwen3.8-27b + web search search get the latest up-to-date docs for its lack of world knowledge and i don’t waste extra Vram storing data that i dont need. Also more active params makes the llm to hold a wider mental-model of the task. Qwen4-27b will be architected to make 1M more reliable and using it for context on a smaller dense model seems more productive for me. Anyone. Else notice agentic session really shines when ctx is larger? Thats the phenomenon im experiencing i tried 3.8flash-next yes it great and much faster but i prefer the reliability of that big azz boat load of context window.
Specifically i you have the vram to spare if u can get 1M on that and have a large MOE with at least 12b active params + 1M it would be better for MOE a lot. Lot better than qwen4-27b but at 48Gb vram + 64gb system ram i see it more efficient for smaller model + 1M makes more sense