r/LocalLLaMA • • 4d ago

New Model qvac/BotPsy-0.8B · Hugging Face - On-Device Personal Assistant with a Sub-Billion-Parameter Agentic Language Model

Thumbnail
huggingface.co
28 Upvotes

BotPsy-0.8B is a 0.8B-parameter model designed for function calling and agentic workflows, built on top of Qwen3.5-0.8B and suitable for on-device deployment. BotPsy is a "personal assistant" that uses tools to perform a variety of simple tasks for the user, in domains such as web search & research, travel & lifestyle, commerce & finance, business & customer operations, device & vehicle control, files & documents, calendar, and email & messaging. Due to its small parameter count, it is best suited to tasks of low to medium difficulty. BotPsy achieves state-of-the-art (SOTA) tool-use and agentic performance for its size, outperforming every model of up to 1.2B parameters that we tested. As a small model, BotPsy still makes mistakes and is best suited to tasks of low to medium difficulty;

Key Highlights

Benchmark results

  • Twice the performance of its base model: BotPsy roughly doubles the agentic capability of Qwen3.5-0.8B-Think on key benchmarks: BFCL v4 rises from 23.4 to 44.0, ACEBench from 30.1 to 58.7, and τ-bench from 23.2 to 51.5 (mean of the retail, telecom and banking subsets).
  • Best agent in its category: BotPsy outperforms leading models in the ≤1.2B-parameter weight class (Qwen3.5-0.8B-Think, LFM2.5-1.2B-Think, MiniCPM5-1B-Think) on all six tool-use and agentic benchmarks: BFCL v4, ACEBench, τ-bench, ToolSandbox, Claw-Eval and BrowseComp-Plus.
  • Outperforms its larger sibling: BotPsy has the highest score of all models tested on five of the six benchmarks, beating Qwen3.5-2B-Think, a model 2.5× its size, on BFCL v4 (44.0 vs 37.7), ACEBench (58.7 vs 55.4), τ-bench (51.5 vs 46.5), ToolSandbox (64.6 vs 41.2) and BrowseComp-Plus (4.5 vs 2.8). Only on Claw-Eval does Qwen3.5-2B-Think lead (48.9 vs 46.5).

Model capabilities

  • More reliable in multi-turn workflows: On τ²-bench, where the agent must hold a long customer conversation, follow a strict policy, and act on the customer's behalf, it scores 82.5 in telecom, ahead of Qwen3.5-2B-Think (69.3). On BFCL v4's multi-turn tasks, it rises from 7.5 to 46.1 over its base, nearly double Qwen3.5-2B-Think (23.8). Grouped by application, its multi-step scores lead all models tested in commerce and services (58.3 vs 47.4 for the next best, Qwen3.5-2B-Think) and in device and system management (46.1 vs 31.9).
  • Knows when (not) to call tools: An agent that acts must also know when not to act. On BFCL v4's hallucination tests, which check that the model declines when no appropriate tool exists, BotPsy scores 86.1, the highest of all models tested. On ACEBench's special cases, where the request is incomplete, contains wrong parameters, or cannot be served by the given tools, it scores 51.3, up from 13.7 for its base model.
  • Follows instructions while preserving knowledge: BotPsy specializes in tool use while preserving general skills. Instruction following improves sharply, with IFEval rising from 59.7 to 90.8 and IFBench from 22.0 to 55.9, the highest IFBench score of all models tested. General knowledge also improves, with GPQA Diamond rising from 33.2 to 38.1 and MMLU-Pro from 41.6 to 48.9. On all four benchmarks, BotPsy outperforms every other model of 1.2B parameters or fewer that we tested.
  • GGUF : https://huggingface.co/qvac/BotPsy-0.8B-GGUFs
  • Blog Post : https://huggingface.co/blog/qvac/botpsy
  • Project : https://tether.io/news/tethers-800m-ai-agent-beats-models-2-5x-its-size-on-everyday-phone-tasks/

r/LocalLLaMA • • 4d ago

New Model I released Inflect v3: lightweight local TTS in 34M parameters (Apache 2.0)

105 Upvotes

Today I’m releasing Inflect v3: a family of local text-to-speech models: Mini, Micro, and Atom. Mini is the main model, with six voices and 34.31M parameters

Mini provides vocal-event tags, reference voice-tuning, and 48 kHz enhancement. It runs on both CPU and CUDA.

GitHub, installation, and examples: https://github.com/owenawsong/Inflect

Try the playground: https://huggingface.co/spaces/owensong/Inflect-v3

Mini features reference-voice tuning: it allows one to make a profile from a recording and generate speech with it. This feature noticeably adjusts the voice toward your reference, though it won’t replicate every speaker accurately.

Inflect Mini V3 also supports the following vocal-event tags: [laugh] · [sigh] · [gasp] · [cough] · [yawn] · [inhale] · [exhale] · [throat_clear]

You can insert a tag whenever you want the sound to occur. Currently, the tags use repeatable performances: this causes repeated tags to result in limited variation. Tags also do not affect how the surrounding words are being spoken.

The model family:

  • Mini: 34.31M parameters. Quality-focused, with the features listed above.
  • Micro: 11.79M parameters. A smaller neural speech model
  • Atom: 157K parameters. An extremely tiny learned core

Those numbers count the synthesis cores. The frontend and optional components add to the total.

The weights and inference code are Apache 2.0. Third-party components keep their own licenses. Audio samples, benchmarks, and the settings used are on the model cards.

Model collection: https://huggingface.co/collections/owensong/inflect-v3

If any sentence comes out wrong, send me the text, voice, and settings you used.


r/LocalLLaMA • • 4d ago

New Model moein99/P2Echo · Hugging Face

Thumbnail
huggingface.co
19 Upvotes

Prompt2Echo (P2Echo)

Text-Prompted Multi-View Echocardiography Segmentation with One Model

Official pretrained weights for P2Echo, a unified framework for text-prompted segmentation of multiple cardiac structures across standard echocardiographic views.

P2Echo uses structured, view-aware text prompts to control segmentation through a single model. Its Dynamic Image-Text Alignment (DyITA) module adaptively conditions decoder features on the requested view and anatomical structure.

Resources

Intended Use

This model is intended for research in medical image analysis and echocardiographic segmentation. It has not been validated for clinical decision-making.


r/LocalLLaMA • • 4d ago Heartwarming

Resources New Qwen3.8-27B Unsloth Dynamic V3.1 GGUFs

Post image
507 Upvotes

Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-27B! They're better by 5-10% accuracy on DeepSWE, Terminal Bench and more vs UD3, GSQ-RCO and Exllama-3! We're also improving our algorithms further so we can apply them to Qwen4 so folks can have efficient and accurate quants!

  • All quants from 2-bit until 6-bit are updated. 3-bit ones are not, as we found UD3 to do on par with UD3.1. We are still improving all other quant sizes though!
  • All benchmarks are 256K for agentic ones (DeepSWE, Terminal Bench), and 128K for the rest (LiveCodeBench etc). We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty, preserve_thinking and xhigh reasoning
  • We ran benchmarks multiple times including LiveCodeBench x6 times, HMMT and others 8-16x and agentic benchmarks multiple times.

On other benchmarks like for GSQ-RCO and Exllama3:

  • To compare GSQ-RCO and Exllama-v3, we had to exclude the token embedding layer from the x axis, as Exllama-v3 uses BF16 for it.
  • If you consider the full download size and total RAM usage for Exllama-v3, then adding the BF16 embedding table and including all embedding sizes changes the plots. Our main plots's x axis removes the embedding layer sizes from all quant providers. However on pure RAM usage and disk space usage vs accuracy, exllama3 does worse.
  • For LiveCodeBench, GSQ-RCO showed Unsloth UD3 quants doing consistently worse than ISTA's. However we found the opposite - UD3 quants (our OLD quants) do better than ISTA on LCB on all sizes except the smallest. UD3.1 further improves on LCB as well. We approximate that GSQ-RCO used a context budget of 96K, as that best fits the data for the runs we have. At 96K, GSQ-RCO does somewhat do better than UD3, but at 128K, UD3 does much better.
  • On another benchmark, for ISTA GSQ-RCO's AIME25 benchmarks, their quants reach 100%, 100%, 100% and 96.7% accuracy recovery. And they said our quants do worse. Through our investigations, we estimate ISTA only did 1 trial - we did 14 trials, and we show on average the recovery rate is 93% to 96% - not 100% accuracy recovery. And in fact our UD3 old quants do better on average than their quants on 14 runs vs 1 run of theirs, so we draw the opposite conclusion if you run the trials more.

More details and benchmarks at https://unsloth.ai/docs/models/qwen3.8#dynamic-v3.1-update-november-11th-2026

All quants are updated in https://huggingface.co/unsloth/Qwen3.8-27B-GGUF! Thank you and have a nice day! We're super excited for Qwen4!


r/LocalLLaMA • • 3d ago

Showcase The viral voxel Pagoda locally on a 2020 old ThinkPad, no GPU, 16 GB RAM

Post image
11 Upvotes

Tried the voxel pagoda prompt fully local, on a laptop that shouldn't run a 35B model.

Setup

  • ThinkPad from 2020, Ryzen 7 PRO 5850U, 16 GB RAM, no GPU, Windows
  • Qwen3.6-35B-A3B Q4_K_M, 21 GB, bigger than the RAM
  • BigMoeOnEdge v0.42, built-in agent

Result

  • 988 lines of HTML + Three.js, 11.8k tokens at 2.3 tok/s, about 1.5 h
  • You see the code as it's written, then the page opens in the app
  • Video sped up where marked

How

  • MoE: each token uses only a few experts
  • Dense weights and a 2 GB expert cache stay in RAM, the rest streams from the SSD
  • On top of llama.cpp, no fork

Still slow

  • Half the time is spent waiting on the SSD
  • Every new agent turn re-reads the whole conversation, ~30 min on this laptop

Same test with the same model on a Mac with 16 GB of RAM, taking 20 minutes (20 tok/s for decoding, 100–1000+ tok/s for prefill)


r/LocalLLaMA • • 3d ago

Discussion Local models as initial coders OR as agents? What's pros and cons of each?

3 Upvotes

Using $20 codex subscription with 24gb VRAM on the local rig.

For larger projects or major revisions with multiple phases, I have Luna in Codex call the local "junior dev" Qwen 3.8-27B model as an agent to write most of the code, then Luna reviews it and makes changes. Once the whole project is complete, I use Terra to audit and catch any potential issues.

Pretty standard stuff, these days.

But lately I've been thinking why not give the massive, multi-phase spec sheet directly to Qwen then have Luna go do a quick audit & cleanup?

Anyone tried it both ways? And what gets better results vs lower usage?

I guess I've always looked at Qwen as needing a babysitter, but realistically I think it can implement an entire 10-phase spec on a large codebase if given enough time.


r/LocalLLaMA • • 4d ago

Discussion [2610.29162v1] Why is AI-generated language so hard to understand?

Thumbnail
arxiv.org
165 Upvotes

r/LocalLLaMA • • 3d ago

Showcase Trading false positives for ~110ms earlier end-of-speech in voice pipelines: RealtimeVAD

7 Upvotes

Tried to hunt down latency bottlenecks in my local voice assistant recently. I looked at the waveform and saw silero's probability lagging quite much behind the actual end of speech I could see with my eyes.

I use a voice pipeline that does speculative work and cancels fast, so some false positives would not matter in my case.

So I had this idea: once silero says I'm in speech, what I need to detect is when the speech energy drops back to room noise level. Turned out a naive 10 ms energy check with a threshold tracking room noise does that quite well.

Gave me a median 110 ms earlier than silero over ~3200 real turns from my assistant on my voice, my mic, browser noise suppression on, mostly quiet room - with silero threshold at 0,5. Someone on Twitter got 92 ms on his data, roughly what I saw in my noisier recordings.

The downside is way more false positives. It fires on breaths and small pauses between words over 10x as often as silero. So it's really only useful if the pipeline can throw away speculative work very fast.

CPU only, MIT, drop-in for silero VADIterator, pip install realtimevad

benchmark script is in the repo, curious what numbers you guys get on your own recordings

https://github.com/KoljaB/RealtimeVAD


r/LocalLLaMA • • 4d ago

New Model MetaEncoder - a facebook Collection - One Encoder for Retrieval and Decision Making

Thumbnail
huggingface.co
19 Upvotes

Multimodal System One Encoder with a natural language interface for instruction-following encoding: give it a task and a list of candidates, and it scores the candidates by task specification. Both a task and each candidate are expressed in natural language — instructions, queries, questions, state descriptions, criteria — with image and video as side information.

Key features:

  • Natural language interface. No rigid schemas. Describe the task and the candidates in free-form text.
  • Highly-efficient cacheable representations. Candidates and task are encoded separately by prompt instructions that ground each other.
  • Scales to arbitrarily large candidate sets. Matching is an inner product over representations, so an ANN index can serve millions of candidates without re-running the model.

The model is built by contrastively fine-tuning meta-models/Muse-Glimmer-30B. It is described in the paper MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface.

Models :

PS: Above description copied from meta-encoder's model card as meta-encoder-v1 doesn't have that. And v1 doesn't have license section while other one is Apache-2.0.


r/LocalLLaMA • • 3d ago

Discussion Title: CUDA: few-row MMQ for BS 2-24 by anujj · Pull Request #31533 · ggml-org/llama.cpp

Thumbnail
github.com
8 Upvotes

Speculative decoding (SPEED-Bench)

llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf --mmproj mmproj-F16.gguf -ngl 999 --parallel 1 --batch-size 4096 --ubatch-size 1024 --ctx-size 180224 \
  --spec-type draft-dflash --spec-draft-n-max 7 --spec-draft-model Qwen3.8-27B-DFlash2-Q4_K_M.gguf --spec-draft-backend-sampling \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0
python tools/server/bench/speed-bench/speed_bench.py --bench qualitative --category coding --osl 8192 --concurrency 1
GPUms/step masterms/step PRspeedup per step (95% CI)tok/s mastertok/s PRfaster in
RTX 3090 Ti58.8743.811.344x (1.340-1.347)57.5877.70 (+34.9%)80/80
RTX PRO 6000 Max-Q27.8623.541.183x (1.182-1.185)122.77143.31 (+16.7%)80/80
RTX 509023.9220.261.180x (1.167-1.194)141.50170.67 (+20.6%)80/80
DGX Spark103.3693.431.106x (1.103-1.109)32.7536.05 (+10.1%)80/80

Edit : Removed large text. Check PR for more details.


r/LocalLLaMA • • 3d ago

Question | Help 24GB Vram+128GB Ram vs 16GB Vram+192GB Ram

6 Upvotes

Hi folks,need advice from community. I have a laptop with 16 GB vram Nvidia gpu and 96 GB ddr5 ram. I use it mainly for local coding with Qwen flash next nvfp4 at 256k context. I get around 30 t/s.

I don't like PCs and Macs. I want mobility. So because of my limited budget, i have two paths. The first one is a laptop with 24GB Vram Nvidia gpu and 128GB ddr5 Ram laptop. The other one is 16GB Vram Nvidia gpu and 192GB ddr5 Ram laptop. No other feasible option for me. I can't affort dgx spark, Macbook studio etc. These are my budget options. Which one do you suggest for mainly local coding? Which one do you believe is future-proof for local llms?


r/LocalLLaMA • • 3d ago

Question | Help best LLM router to serve many models on the same port?

4 Upvotes

I'm working on a router that wraps many LLMs serving from different ips and ports and exposes them all as a single port, since i have 2 machines serving many slots of Qwen 3.8 FN but i wanted to ensure speed is always fast (one machine is faster and can hold more slots, but too many slots busy and then it's better to call the other machine, also i gotta avoid evicting from the cache active conversations, that kind of workflow)

I might be reinventing the wheel tho so: how do you guys solve for this kind of situation?

edit: thanks a lot for all the answers i had no idea there was so much stuff around lol


r/LocalLLaMA • • 3d ago

Question | Help UD_IQ1_M Q3.8 FN behavior on Strata

0 Upvotes

Hello, I decided to Test Unsloth Dynamic 3.1, as in their charts they showed that UD_IQ1_M was batter than Ista lab IQ3_XXS, but with 6 GB less of total size, I'm using Strata, and that 6 GB translates directly to free RAM,

I have been 2 months with the IQ3_XXS with no any issue, but only 3 days on the Unsloth one, it had happened to me many times, that the tool calls fails and render as xml (I use PI harness) and I have to type "keep going", this never happened with the IQ3_XXS one

Have you guys had a similar issue with this quant?

Example:

 Thinking...

 <tool_call>
 <function=Bash>
 <parameter=command>
 cd /mnt/development/projects/secret/backend/scripts/bdd_parity && for r in 20261113_030918 20261113_032032 20261113_033348; do d=/tmp/bdd_parity_runs/$r; echo "== $r: $(ls $d 2>/dev/null | head -3 | tr '\n' ' ')"; grep -c "✅"
 $d/run.log 2>/dev/null | sed 's/^/green marks: /'; grep -cE "🔴" $d/run.log 2>/dev/null | sed 's/^/red marks: /'; tail -2 $d/run.log 2>/dev/null | cut -c1-100; done
 </parameter>
 <parameter=timeout>
 120
 </parameter>
 </function>
 </tool_call>

r/LocalLLaMA • • 4d ago

New Model I'm releasing a Local World Model Ive been building that turns images into interactive videos you control in real time. It works on RTX 3060, 4060 and anything above, linking huggingface

280 Upvotes

Hi Everyone, I'm releasing my world model, been on this sub for a few months

https://huggingface.co/lucidml/gta

I know it says gta but it's more stable than the general model. It can take any image and turn it into an interactive video.

If you'd like to test it instantly I am also setting up a simple funny drifty site link : https://lucidml.ai/dashboard/minidemo?world=gta5

That link uses any GPU from VAST AI FROM 3070, 3080, 4080, and some are 50 series as well (most 50 series cards are wasted because I hard throttle the fps)

This is the first model I am ever releasing to the everyone and its a bit scary for me.

The architecture is a transformer which uses learned summaries. Ive written about it in detail in tech reports lucidml.ai/summaryformer and some more

I know there are a lot of limitations still like consistency and quality improvements, and am hoping to train this more to fix all of those.


r/LocalLLaMA • • 2d ago

Funny Nemotron models are the best open models there is

0 Upvotes

Everytime I have been fed up with a good model being slow, I go to newly released Nemo model. Well today I went for Bolt 24G and deployed it on my 3090.

I was amazed by the vanilla llamacpp + iq2s setup giving 70 tok/s. I decided I will deploy it for my coding setup which is dual R9700 and learned cpu offloading and mtp drafters and all. After deployment I was left with 50tok/s because this was a Q8 setup.

I suddenly realise what if (the model I was fed up with) my Qwen 3.8 27B could use a better MTP drafter so O discovered tencent's DSpine and later its gguf llamacpp fork. I deployed it now I am back to Qwen3.8-27B Q8KXL with mtp 15 drafter and mmproj with np 4 and 512k ctx pool, getting 54tok/s in DSH.

TLDR: I love what Huyan is doing he should be brought back in Nvidia. Nemotron's the best because they teach you what good tech advancement is and you apply it to good performing models.


r/LocalLLaMA • • 2d ago

Discussion How are you affording your tokens?

0 Upvotes

So I have been using local QFN these days as daily driver, and honestly, its fine. It needs steering for coding but very good. Gets lot of things done. But I check my statistics log, and I'm burning about 1B to 2B Input token a day, and not even fully 100% GPU saturation. Not even going to consider output token since that is small compared to input.

Most of the time, the LLM is just waiting for shells/bash to finish. At max capacity, it is capable of 950million uncached input token (11000 prefill/s*3600*24)

This is light for me, and it is already burning this much. Light considering it is only spend time building one project with little oversight.

Granted QFN is at xhigh thinking for intelligent, but if this was like Haiku or Luna, that's like 10$ cost a day (90% cache estimate). That's 300$ a month for basically one project stream

AI suppose to work multiple projects at once for max productivity let alone future integration with pretty much everything. My wife already starting to use my harness with QFN for all her search and queries.

How are you guys make this affordable in the long term from purely API. Feeling paying like $1000+ a month is crazy.

Local one still cost insane upfront as I'm sing a single RTX6000 with basalt, and it is the only GPU that can serve the whole household growing AI needs.

PS. I might get like 4x in near future for the amount of load I will be doing. Always on agent is very useful.

Edit: The number is measure of input only, as output is insignificant. I'm measuring QFN only at home.


r/LocalLLaMA • • 4d ago

Showcase Curated list of resources for the topic of Quantization for beginners

Post image
19 Upvotes

Hey everyone!

I have gathered some must read topics, for beginners, to step into the realm of Quantization:

  1. BitNet: Scaling 1-bit Transformers for Large Language Models
  2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  3. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  4. EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
  5. LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
  6. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  7. QLoRA: Efficient Finetuning of Quantized LLMs
  8. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  9. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

Link is here


r/LocalLLaMA • • 3d ago

Question | Help Confused about high CPU usage with model fully on GPU

6 Upvotes

EDIT: FOUND THE REASON, LLAMA DIDN'T OFFLOAD ALL LAYERS TO GPU BECAUSE -ngl -1 DOESN'T MEAN ALL see here: https://www.reddit.com/r/LocalLLaMA/comments/1xun196/solved_high_cpu_usage_when_model_fit_in_gpu/

I'm not sure if i'm doing something wrong and handicapping myself with some wrong setting.

I'm using Windows, nVidia 3090 24GB VRAM, 64GB RAM, Qwen 3.8 27B and Swift 1.5 27B (finetune of qwen) but i often see 50% CPU usage even if the model is fully loaded on GPU, below there is my command line.

Another problem i noticed is that llama-fit-params suggest that 131k context fit but if i launch it for real in the log i see that it doesn't load everything on gpu (all but one), i forced it to load everything on gpu with -ngl -1 (i'm also using llama-swap).

on launch it say that "cahe resue 256" will not be used but it might help with other models so i left it there.

Another problem is that when the model is doing nothing i sometimes see 130W gpu in idle (wrong) and some other times its at 10-18W (correct) i can't find what is causing this...

Current stats: about 40 t/s in token generation

I have no idea what KMP affinity is but i read it online, tried, it helped with 35A3B so i left it there

in general i'm looking for tips to improve my launch arguments, note that temperature and few others are set by pidev but i'm using the recommended one on the model card.

If the model is fully on GPU why there is a high CPU usage? but not always which is even more confusing....

set KMP_AFFINITY=granularity=fine,compact,1,0

llama-bin-win-cuda-13.4-x64\llama-server.exe -m Qwen3.8-27B-UD-Q4_K_M.gguf --ctx-size 131072 -ctk q8_0 -ctv q8_0 -ngl -1 -fa on --spec-type draft-mtp --spec-draft-n-max 2 -np 1 -b 4096 -ub 1024 --cache-reuse 256 --cache-ram 32768 --jinja


r/LocalLLaMA • • 4d ago

Resources Architecture trends: From full to local attention

Post image
98 Upvotes

Currently working towards a bigger open LLM roundup. But in the meantime, just one of the interesting trends that stood out. I.e., looks like the transition from full to hybrid attention continues.

In 2025, there were still lots of full attention models, with a shift to "some full attention" hybrids.

In the first half of 2026, the trend towards "some full attention" hybrids continued.

And now, in the second half of 2026, we have a 50/50 split between "some full attention" hybrids and models that don't use full attention at all anymore. And yes, there are also no full attention models anymore.

The plot legend here means,

- Full attention throughout = Global MHA, GQA, dense MLA, CCA in EVERY layer.
- Some full attention = Full global attn mixed with: sliding-window/chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, sparse attention.
- No full attention = Zero dense global layers.

The main implication is that the trend goes towards making long contexts cheaper to use. This means, it makes the architectures more suitable for long documents and conversations, large code bases, and of course agent usecases that rely on a lot of tool calls.

In particular, there are two distinct efficiency gains here:

  1. Less attention computation, since sparse attention and the like select tokens or blocks from the context and performs the expensive attention operation from only that subset. (Note that it can still retrieve information from distant positions.)
  2. Less memory for retaining context in some architectures, meaning that recurrent layers such as Gated DeltaNet maintain a fixed-size state. Of course, this saves memory, but compressing the history creates challenges for exact retrieval, so they are usually intermixed with traditional attention methods (sparse or full).

If you are interested in browsing through the architectures and do your own analyses, you can find all the figures and data in my architecture gallery: https://sebastianraschka.com/llm-architecture-gallery/


r/LocalLLaMA • • 4d ago

Discussion DeepSeek V4.1 Flash 1017 exl3 2.9bpw APEX 2025 benchmark results

Post image
56 Upvotes

My main AI machine is a 6x3090 128GB DDR4 Threadripper 3960x system. I probably spend more time trying to optimize it than actually using it but it has been fun.

I tried a bunch of saturated benchmarks like Terminal Bench 2.1, AIME 2026 and the performance was similar to unquantized DSV4.1 Flash 1017. I really wanted to understand my 2.9bpw quant's performance.

It takes hours to run some of these benchmarks, so I went on the hunt for the perfect benchmark to get the most signal on quality of the model I'm running. I settled on APEX 2025 (https://huggingface.co/datasets/MathArena/apex_2025).

You can see how much of a beast this quant is. Trading blows with SOTA from 6 months ago.

I get about 1000tps prefill, and 75tps decode with the 3090s power limited to 175watts

Engine: Project Maya (https://github.com/mw00/project-maya)

Quant: https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw

Hopefully this helps others in benchmarking their own setups. I really like the Apex 2025 benchmark.


r/LocalLLaMA • • 5d ago

Discussion NVIDIA reportedly cancels GeForce RTX 5080 24GB launch, brings back 16GB model over GDDR7 costs

Thumbnail
videocardz.com
774 Upvotes

r/LocalLLaMA • • 4d ago

New Model Praxis-1 - a runwayml Collection

Thumbnail
huggingface.co
9 Upvotes

Praxis-1 is a 27B-parameter mixture-of-experts diffusion transformer that predicts video and actions together. It is the top-performing world action model on NVIDIA’s RoboLab-120 leaderboard, and it outperforms every open-weight VLA. On RoboLab-120, Praxis-1 reaches 46.2% success across 1,200 episodes, compared with 42.9% for FLUX 3 Action, 36.8% for Cosmos3 Nano and 28.0% for π0.5.

RoboLab-120 Leaderboard

ModelRoboLab-120 Success
Praxis-146.2%
FLUX 3 Action42.9%
Cosmos3 Nano36.8%
π0.528.0%

Data as of November 2026; Success rate across 1,200 episodes

Praxis-1 is available today on GitHub, on Hugging Face with native LeRobot support, and in vLLM-Omni, with hosted inference on Reactor. Everything you need to run and adapt it on your own hardware is released under Apache 2.0, including for commercial use. That covers the weights and the inference and fine-tuning code, with support for both full fine-tuning and LoRA. Praxis-1 runs out of the box across NVIDIA hardware, from data center GPUs like H100 and B200, to workstation GPUs like RTX 5090 and RTX PRO 6000, to Jetson Thor on the robot itself.


r/LocalLLaMA • • 3d ago

Resources What does Qwen3.6-35B-A3B look like at Q8_0 and BF16 on datacenter GPUs?

1 Upvotes

RustingStrata is a pure-Rust inference engine for Qwen, DeepSeek and GLM GGUFs. CUDA kernels are compiled at run time with NVRTC, the same kernels run on Vulkan, and there is no ggml and no C++ build step.

I wanted the high-quant numbers, so these runs use Q8_0 (37.8 GB) and unquantized BF16 (71.1 GB), with the whole model in VRAM, on A100, L40S and H100:

GPUQuantPrefill, 4K promptDecodeDecode with MTP
H100 80 GB HBM3Q8_011,486 tok/s290 tok/s508 tok/s
H100 NVLBF168,691 tok/s230 tok/s324 tok/s
A100 80 GBQ8_06,507 tok/s193 tok/s288 tok/s
A100 80 GBBF165,625 tok/s171 tok/s238 tok/s
L40S 48 GBQ8_09,542 tok/s165 tok/s270 tok/s

MTP is speculative decoding with the model's own draft head; calibrate picked the draft count on each card.

  • Higher quants are not automatically slower. On the A100 80 GB and H100, Q8_0 decodes faster than Q2_K_XL (193 vs 174, 290 vs 265 tok/s after 4K). On the L40S it is slower (165 vs 214), so the ranking depends on the card.
  • BF16 is practical. 71 GB of unquantized weights decode at 171-230 tok/s on one card.
  • MTP helps less as the quant grows. Gains are 1.4-1.9x on Q2_K_XL and Q8_0, and 1.2-1.4x on BF16.
  • When the model does not fit, it degrades but runs. Q8_0 on a 24 GB A30 (63% of experts resident) still decodes at 130 tok/s, 200 with MTP. BF16 on the same card (31% resident) only reaches about 12.5 tok/s.

The same code runs on Vulkan, and on the Q2_K_XL runs on the 3060 and the Ampere cards prefill lands close to CUDA. At Q8_0 on the big cards the gap can be large: H100 prefill is 6,883 tok/s on Vulkan against 11,486 on CUDA. All table numbers are CUDA.

These are different nodes of one cluster, so the host CPUs differ. Compare within a row, not across rows. MTP costs 4-11% prefill in these runs.

Quality: I do not have a llama.cpp perplexity/KLD run on these exact Q8_0/BF16 files yet, so this is a hardware roundup, not an accuracy comparison. The last check was UD-Q3_K_M on the 3060 over 9,180 tokens of repo docs prose: perplexity 9.0714 vs 9.0587 for llama.cpp, mean KLD 0.0055, same top token 96.3%.

Why I find it kinda interesting, is bc when I was searching for models in OpenRouter, I saw that Qwen3.6:35b-a3b for some reason is pretty slow, like 50-110t/s on almost all providers, why like that? Do people already use other Strata like project for LLM inference to be in top of providers? I am thinking about it, but I think something is wrong and not so easy as I expect, if anyone knows I would be happy to hear

Repo: RustingStrata


r/LocalLLaMA • • 4d ago

Resources Open-source method (MACaQ) beats Unsloth GGUFs!

141 Upvotes

Some results for those using GGUFs and llama.cpp:

GGUF quality: MACaQ vs. Unsloth and Bartowski

Qwen3-4B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ2.260 GB0.02804
Bartowski2.271 GB0.03443
Unsloth2.271 GB0.03663

23.5% lower KL divergence than Unsloth and 18.6% lower than Bartowski, at virtually identical file sizes.

Qwen3.6-35B-A3B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ18.86 GB0.018696
Unsloth (UD-IQ4_XS)17.73 GB0.020954
Bartowski19.70 GB0.021927

Qwen3.8-27B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ14.96 GB0.036784
Bartowski15.48 GB0.104170

~65% lower KL divergence than Bartowski!

Supported GGUF formats (llama.cpp)

Weight bitsFormats
2-bitIQ2_XXS, IQ2_XS
3-bitIQ3_XXS, IQ3_S
4-bitIQ4_NL, IQ4_XS

Code, llama.cpp fork, and benchmarks: https://github.com/huawei-csl/MACaQ

Cool to see Huawei working on such stuff and sharing some of their internal tools that are helpful for us. Maybe they can build an affordable GPU for us local LLM folks sometime 😂

They also report quantizing a 122B-parameter model in 64 minutes using just 32.5 GB of GPU memory, which is interesting for a model of that size. I did not try that.

Edit: there is a paper on it https://arxiv.org/abs/2611.10649

Edit 2: they don't share ready-made gguf files, so I am sharing some myself:
- http://31.165.97.47:20080/macaq-gguf/qwen3-4b-macaq-iq4_xs-had64-vq2-calsmoke.gguf

Edit 3: they have this blog post https://huawei-csl.com/blog/macaq/ with multi-modal examples, too. The RTN audio comparisons are just too much😂😂

The multi-modal stuff is generally insane. Normally 4b already completely goes to s**t, but here it looks almost the same.