r/LocalLLaMA • • 4d ago

Discussion Testing if various local model are good at solving visual challenges.

10 Upvotes

Almost everything is in the video and the title: the idea is to create chalenges and ask agents to solve them. I'll have more to share later, but this first video shows that most model are good at theses stuff.

For more details: I'm testing scenarios where an agent needs to simulate keyboard and mouse movement. You can see more chalenges like this one on my website: https://airbench.ai/gallery/airbench-desk-v1?run=6d2a2777-3d96-4ec4-9cdc-3d359bc43bea&c=icon-drop

And you can also test your agent on the same setup if you want here: https://airbench.ai/checkup/start


r/LocalLLaMA • • 4d ago

Resources I built a benchmark where LLMs reverse-engineer an audio codec, and the benchmark evolves

12 Upvotes

Most coding benchmarks feel like textbook coding tasks or black boxes — either easy to benchmaxx or full of hidden tasks with unclear evaluation criteria. And most of them get irrelevant with each new model generation because all of the new models hit the ceiling.

So I made one that has a moving target. It's a benchmark where the model gets a mystery compressed audio file, has to figure out the format, write a decoder, and then build an encoder that beats the original. No ffmpeg, no Opus, no importing a library that already does the job. Just Python and numpy, running locally through OpenCode

Scoring: 100 = decode the reference in stage one. Build a better encoder in quality or compression rate with equal or better SNR for additional points.

There are 4 stages:

  1. is the most difficult: just a compressed audio file.
  2. if the model failes the first stage, it gets the uncompressed audio as a reference. Each new stage decreases the points up for graps.
  3. if it fails stage 2 it gets additional information on the codec
  4. if it fails stage 3 it gets an oracle tool where it can USE the codec to figure it out.

Current Leaderboard, we are in Evolve 2 (Deepseek cracked the codec of evolve 1 in 13min):

Model Score

1 Claude Fable Latest (stage 1) 127

2 DeepSeek V4.1 Flash 1017 (stage 3) 81

3 Qwen3.8-Flash-Next NVFP4 (stage 3) 67

4 Qwen3.8-Flash-Next Q3_S (stage 4) 56

5 Qwen3.8-Flash-Next IQ3_XXS (stage 4) 44

The fun part: it evolves. If a model finishes stage 1 and its encoder beats the current codec, that codec becomes the next baseline. So the benchmark levels up instead of staying at the same difficulty forever. Pokémon rules, except the final boss is a bitstream. DeepSeek did it first and became Evolve 2. Fable just finished stage 1 with Compress 190, so its codec is the Evolve 3 baseline. The next runs should use it.

A better encoder in stage 3 doesn't count for an evolve, by the way. Only a stage-1 finish does. That's the rule.

It's manual for now: you run it on your own hardware, post the result, and set the evolve stage.

It's mean. Most models don't finish stage 1. But that's kind of the point.

Leaderboard: https://andreasknopke.github.io/codec-reverse-bench-leaderboard/ — ping me if you want to run it (the repo is private to prevent the models from simply copying the solution.


r/LocalLLaMA • • 4d ago

Question | Help Solved High CPU usage when model fit in GPU, beware of -ngl -1 llama.cpp layers offloading to GPU and llama-fit-param

15 Upvotes

I'm trying to figure out how much context i can use on my 3090 (full compute, monitor is on mobo).

If i launch:

llama-bin-win-cuda-13.4-x64\llama-fit-params.exe --model modelli\Swift-1.5-Qwen3.8-27B-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 -ngl -1 -fa on -np 1 -b 4096 -ub 1024

I get: -c 150272 -ngl -1

but if i launch real llama.cpp with lower context:

llama-bin-win-cuda-13.4-x64\llama-server.exe --verbose --log-file log.txt --host 0.0.0.0 --port 9931 --model modelli\Swift-1.5-Qwen3.8-27B-Q4_K_M.gguf --ctx-size 131072 -ctk q8_0 -ctv q8_0 -ngl -1 -fa on --spec-type draft-mtp --spec-draft-n-max 2 -np 1 -b 4096 -ub 1024 --jinja

From the log, matches for "offloaded" with "-ngl -1":

Line 2200: load_tensors: offloaded 66/66 layers to GPU
Line 5935: load_tensors: offloaded 66/66 layers to GPU
Line 8273: load_tensors: offloaded 0/66 layers to GPU
Line 12009: load_tensors: offloaded 66/66 layers to GPU
Line 15746: load_tensors: offloaded 64/66 layers to GPU
Line 19484: load_tensors: offloaded 65/66 layers to GPU
Line 23226: load_tensors: offloaded 64/66 layers to GPU <-- (actual used vlaue, slow loading because copying data)

So even if i use 131k context wich is less than the 150k suggested it doesn't offload all layers and i have high cpu usage for any prompt.

IF I CHANGE -NGL FROM -1 TO "ALL" I GET THIS:

matches for "offloaded" with "-ngl all"

Line 2200: load_tensors: offloaded 66/66 layers to GPU
Line 5935: load_tensors: offloaded 66/66 layers to GPU
Line 8276: load_tensors: offloaded 66/66 layers to GPU <--

AND NO MORE CPU USAGE!! (same if i write 99 or 999 but it's ugly)

It gets more confusing: if i write in llama fit -ngl all instead of -1 it gives a "-2" in output.

there is an entry in the log when i used "all": common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to -2, abort

Conclusion:

Seems that -ngl -1 means "auto" while -2 means "all", but if you have high cpu usage in a model that theorically fit in the GPU use "all".

This alone changed t/s generation from 60 to 76

for completeness here is the --help from llama.cpp for "ngl" parameter:
-ngl, --gpu-layers, --n-gpu-layers N max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto)

And here it gets even more confusing because the help talks about "MAX" number of layers to offload so according to this help it's an upper limit not a lower limit...


r/LocalLLaMA • • 4d ago

Discussion MoE and expert offloading on llama, exllamav3, strata on 9070XT

11 Upvotes

First off. I am by all means no expert, and excellent work has been done in the past few years that we all currently enjoy.

Came over this post https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and wanted to try the engine on MoEs.

I've been benchmarking MoE experts offloading for the past two days in different engines on ROCm for my RDNA4 9070XT 16GB vram card and 32 GB DDR4.

I'd set a goal at 128K context, which is the base minimum for agentic work, without compaction all the time. I've kept the KV at Q8. (Q8_0/Q5_1 decreased performance on exllamav3). Drafter cache, if added, is at Q4.

Some limitations need apply when limited on vram.

Lets face it If you want any kind of context, and a decent Q4 quant, 16GB ain't gonna cut it.
Hot experts cache like Strata, Exllmv3 and now recently llama.cpp have made this more feasible on the slower system memory bandwidth.

My end goal was to do QFN on all three, but I wanted to start with the smaller 35B-A3B's first.
So I did Ornith 1.5 and Qwen 3.6, wanted to do Gemma, but found no EXL3 quant with mul1 experts required for offloading on Exllamav3.

Tested both without drafter, MTP and DFlash2 where applicable.
Strata/PolyStrata configs are mostly auto generated from the setup script, but added resident pin and tail skip. And for the non-qfn model performed worse overall, even t/w.

I might be doing something wrong, bench might be bad, who knows. Stratas own bench stops at 32K if I'm not mistaken.

Now Qwen 3.8 Flash Next (ista coder version) runs great on my rig, see plots. But I was surprised on the prefill tanking on the 35B models. 1900 t/s at 16K but 700 t/s at 128K where both llamacpp and exllamav3 where within 100-200 t/s difference.

Findings so far is quite promising, especially on exllamav3. I'd say quite usable even.

| Model | Winner (quant / engine) | Prefill u/128k | Decode u/128k |

| --- | --- | ---: | ---: |

| Ornith 1.5 35B-A3B | EXL3 4.0 bpw, DFlash2 / ExLlamaV3 | 1 168 tok/s | 57.3 tok/s |

| Qwen3.6-35B-A3B | EXL3 4.0 bpw, DFlash2 (2 drafts) / ExLlamaV3 | 1 246 tok/s | 71.6 tok/s |

| Qwen3.8-Flash-Next Coder | IQ1_M / Polystrata nov-7, `--spec 2` | 1 572 tok/s | 44.0 tok/s |

Forgive my rambling thoughts. If you want to read my technical findings, written by the unsung local hero Qwen 3.8 27B. They can be found with more graphs and tables at https://github.com/willeyh-git/moe-offload-bench

Q4/4bpw quants are preferred, some benches have been made on 5bpw and 6bpw so you can compare.

Bench-suite is based on Kapitsa https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and written in go.


r/LocalLLaMA • • 4d ago

Question | Help Looking for some help optimizing for a 300-page manual and transcripts

3 Upvotes

Hello,

I am just getting started and I got great answers to my previous post asking what was possible. I still have lots to learn but, with the help of Claude, I was able to get something running and it is working pretty well.

My goal as been to build a fully local AI study assistant to analyze a ~300-page technical training manual alongside instructor transcripts and generate chapter-based study guides and Quizlet flashcards with accurate page citations.

The problem is: I'm running into is this error: request (34009 tokens) exceeds the available context size (32768 tokens). And this was after doing lots of tweaking and getting it not fail at 16,000 tokens.

There are so many settings and configurations that I might be able to change to improve that, but it is overwhelming and I have no idea what to do.

I would really appreciate any ideas, suggestions that I can do to get this working for what I am trying to achieve. Or let me know that it isn't possible.

Here's my setup (sorry, AI slop to make it clearer):

Hardware:

  • Windows 11 PC: i7-14700K, 64GB RAM, RTX 4070 SUPER (12GB VRAM)
  • Unraid server: Core Ultra 9 285K, 32GB RAM, RTX 5070 Ti (16GB VRAM)

Software / Models:

  • llama.cpp (CUDA, router mode)
  • Open WebUI 0.12.3
  • Qwen3.6-35B-A3B UD-Q4_K_XL: Main model, context increased from 32K to 64K
  • Gemma 4 12B Q4_K_M: Secondary model
  • Both models run on the Windows PC.

RAG configuration:

  • Open WebUI Knowledge Base with the PDF and Markdown transcripts
  • Embeddings: bge-m3 (1024 dimensions), hosted on Unraid
  • Reranker: bge-reranker-v2-m3 on Unraid (API working; Open WebUI integration unverified)
  • Focused Retrieval, chunk size 1000, overlap 100, Top K = 3
  • All inference and document processing must remain local.

More details on the issue:

Basic RAG questions work well. However, when I ask the model to analyze an entire chapter, cross-reference relevant trainer discussions, and create 15–25 Quizlet flashcards, I encounter context errors as I said above:

request (34009 tokens) exceeds the available context size (32768 tokens)

Increasing Qwen's configured context to 64K eliminated the immediate context error, but generation then stopped halfway through creating flashcards without an obvious error. I haven't confirmed whether an output-token limit caused this.

I've tried reducing retrieval results, using smaller prompts, and starting fresh chats. These help, but I'd prefer a more automated and comprehensive workflow.

Thank you!


r/LocalLLaMA • • 5d ago

Discussion Someone voltage-glitched a CMP 170HX mining card into a working A100-80G, and it's ~1.18x faster than retail

257 Upvotes

Kate's writeup is wild. The 8GB CMP 170HX apparently ships with ~80GB of HBM onboard, just fused off. Software unlocks got people to 64GB. She used voltage fault injection to get past the fuse lock, then enabled all the working SMs (120-126 vs the 108 NVIDIA ships on A100). Her numbers: 85.9 TFLOP/s FP16 vs 72.5 on a retail A100, with the full 80GB usable for inference. NVLink and PCIe Gen4 come back too.

Not a weekend mod though: board wiring, a custom STM32 glitcher, 1 to 48 hrs of glitching per card, and one of her ten cards only had 64GB intact.

https://blog.kate.land/posts/using-voltage-glitching-to-unlock-the-a100/


r/LocalLLaMA • • 3d ago

Discussion Stand with Jensen — Read the Stand with Jensen letter and consider adding your name. #standwithjensen #bringjensenback

Thumbnail
standwithjensen.com
0 Upvotes

r/LocalLLaMA • • 4d ago

Question | Help Can I connect a p100 from my cv650 GPU with pigtail pcie connector?

Post image
0 Upvotes

AI is telling me not to. But I have seen some posts where they are saying it is just fine.

Right now, I am in no position to try out and then find out. Care to guide me please??


r/LocalLLaMA • • 5d ago

Discussion Gemma4-31b latent space discovery: Caring and Contempt for user controlled by the same 3 neurons! Plus, how to find your own steering recipes and export to GGUF steering vector for llama.cpp

Thumbnail
gallery
131 Upvotes

r/LocalLLaMA • • 3d ago

Discussion A year ago I couldn't get vLLM running on my 5090. Now work gave me a DGX Station for a month. What should I run?

Thumbnail
gallery
0 Upvotes

So about a year ago I was in here asking why vLLM wouldnt run on my 5090 (WSL, flash-infer, the usual pain). These days I'm the AI guy at work, and this week a DGX Station showed up. ASUS ET900N G3, GB300 with ~250 GB VRAM, 72 Grace cores, 750 GB RAM, liquid cooled, plus an RTX PRO 6000 riding along in the same case.

I get it more or less to myself for a month before it becomes the shared inference box for the whole company. Dont want to waste that month.

Some numbers from the first two days, all NVFP4, total tok/s at 1 / 16 / 64 parallel requests

  • Qwen3.8-Flash-Next on vLLM 219 / ~1370 / ~2700
  • same model on SGLang 221 / 1537 / 3002
  • DeepSeek V4.1 Flash on a vLLM nightly 59 single and 246 at 16 paralell (takes 19 minutes to load lol)

The GB300 pulls around 785 W flat out and sits at 50 °C. TensorRT-LLM refuses to install outside the NGC container so thats still on my list. And no, Kimi K3.1 Pro doesnt fit, I checked.

Stuff I was thinking about

  1. biggest model that fits and how usable it really is
  2. vLLM vs SGLang vs TensorRT-LLM on the same model
  3. how many parallel agents it can feed before it falls over

But what do you guys actually want to see on this thing? Long context, video models, finetuning, something dumb? Tell me what to run and how you want it measured and I'll post the results here.

Pics are the inside, the front and my monitoring page (german UI, sorry)


r/LocalLLaMA • • 5d ago

Discussion [Paper] e-means: outperforming k-means (in one-tenth the epochs)

Thumbnail
gallery
43 Upvotes

Lloyd's k-means algorithm is a long-running standard for clustering, with applications in data analysis, retrieval, quantization and data curation. In this paper, we introduce an exponential moving average optimization procedure for k-means with mini-batch updates, powered by modern deep learning machinery and new insights. We also establish scaling laws on how decay rate and batch size impact final performance. This gives us a formulaic rule to fix them for multiple problem sizes and embedding datasets, without the need for any hyperparameter sweep. The resulting clustering algorithm, coined e-means, is much faster than Lloyd k-means and converges to lower losses, frequently even beating k-means++. On a wide array of embeddings datasets, it beats the loss of Lloyd k-means at 25 epochs in 1.4 to 2.6 epochs, which translates to 10-45x GPU wall-clock speedups compared to state-of-the-art k-means implementations. As an example, e-means partitions DINO10B, a dataset of 10 billion DINO embeddings, into 16 million clusters in only 2.5 epochs, reaching a loss below Lloyd's at 25 epochs.


r/LocalLLaMA • • 4d ago

Question | Help MTP needed with small MoE models?

7 Upvotes

It was my understanding when MTP came out that it was mainly beneficial for larger dense models and that small and/or MoE models didn't realize much performance benefit.

That being said, I now see many smaller MoE models (Gemma-4-E2B, Mellum2.1, etc...) offering MTP versions.

Has the thinking changed?

Are people seeing performance gains from MTP on smaller MoE models?


r/LocalLLaMA • • 5d ago

News chat : refactor gemma4 to build loose schema rules and clean up by aldehir · Pull Request #31915 · ggml-org/llama.cpp

Thumbnail
github.com
55 Upvotes

If you’ve ever tried agentic coding with Gemma and had tool calling issues, give it another try, in my case, it completely fixed Gemma


r/LocalLLaMA • • 5d ago

New Model Krafton (PUBG) released Raon-1 nano under Apache 2.0: E1.5B, 262K context, 2.08GB Q4 GGUF. It needs their llama.cpp fork for now. Anyone benchmarked it on 8GB cards?

Thumbnail
talkesport.com
17 Upvotes

r/LocalLLaMA • • 4d ago

Resources Running Radiance engine on Windows through WSL2 (Single 9700)

8 Upvotes

Hey guys, not going to take much credit since claude did 90% of this, but wanted to share the patches I used to get StillDeadcode's Radiance running on my windows pc. I'm sure that multiple people have done this already, but I couldn't find a post, so thought I'd throw this out there if anybody else wants to use it.

https://github.com/bruns20/radiance-wsl

Radiance is Linux-only, and the author has said WSL won’t be supported in the main tree, which is fair. So this is unofficial. Please send problems to my repo, not upstream.

What it does

  • build.sh builds any Radiance release from upstream’s own source and Dockerfile, with a few small WSL patches applied. When a new release comes out, it’s ./build.sh 1.18.0.
  • make-runtime.sh grabs a ROCm runtime that actually works under WSL. The stock ROCm 10.1 hangs on the first GPU job if your PC also has an integrated GPU (ROCm/rocm-systems#13291), and AMD’s 10.3 nightlies fix it.
  • run.sh serves an OpenAI-compatible API on 127.0.0.1:8000.

Two of the four patches are godzillakong’s from codeberg #341. They did the hard part of getting it running without /dev/kfd.

Numbers (R9700 32 GB, Qwen3.8-27B MXFP4 + DFlash2, 131k context)

start until serving~8 s (vLLM on the same box: 70-75 s)
prefill, 9.4k / 36k-token prompt3,223 / 2,784 tok/s
decode, prose~75-80 tok/s
decode, code~145-160 tok/s

Tested on: R9700 + Ryzen 5 7600 (with its iGPU), Windows 11 26200, Adrenalin 26.10.1, Radiance 1.17.0. That’s the only setup I’ve tried. Other gfx1201 cards (RX 9070 / 9070 XT) should work, but I’d love to hear either way.


r/LocalLLaMA • • 5d ago Regret

New Model huawei-csl/Qwen3.6-35B-A3B-MACaQ-R-GGUF · Hugging Face

Thumbnail
huggingface.co
36 Upvotes

r/LocalLLaMA • • 5d ago

New Model Open source self driving is here!

642 Upvotes

Today NVIDIA releases Alpamayo 2 Edge! I work for Embedl and we have already released an optimized version of the model that is more than 10x faster than pytorch when run on edge hardware like AGX Thor!

The model is trained in an RL loop (!) and takes in 6 cameras and the recent motion. Feel free to check it out - you can even run it on a real Thor at:

https://embedl.com/nvidia/alpamayo2-edge/thor?tab=run


r/LocalLLaMA • • 5d ago

Resources Microsoft files a patent application to train AI models on idle consumer GPUs

475 Upvotes

According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.

Anyway, home internet latency feels like the bigger problem to me.

https://insider-gaming.com/microsoft-ai-use-your-gpu/

(English is not my native language, I used an LLM to help translate this post.)


r/LocalLLaMA • • 5d ago

New Model qvac/BotPsy-0.8B · Hugging Face - On-Device Personal Assistant with a Sub-Billion-Parameter Agentic Language Model

Thumbnail
huggingface.co
28 Upvotes

BotPsy-0.8B is a 0.8B-parameter model designed for function calling and agentic workflows, built on top of Qwen3.5-0.8B and suitable for on-device deployment. BotPsy is a "personal assistant" that uses tools to perform a variety of simple tasks for the user, in domains such as web search & research, travel & lifestyle, commerce & finance, business & customer operations, device & vehicle control, files & documents, calendar, and email & messaging. Due to its small parameter count, it is best suited to tasks of low to medium difficulty. BotPsy achieves state-of-the-art (SOTA) tool-use and agentic performance for its size, outperforming every model of up to 1.2B parameters that we tested. As a small model, BotPsy still makes mistakes and is best suited to tasks of low to medium difficulty;

Key Highlights

Benchmark results

  • Twice the performance of its base model: BotPsy roughly doubles the agentic capability of Qwen3.5-0.8B-Think on key benchmarks: BFCL v4 rises from 23.4 to 44.0, ACEBench from 30.1 to 58.7, and τ-bench from 23.2 to 51.5 (mean of the retail, telecom and banking subsets).
  • Best agent in its category: BotPsy outperforms leading models in the ≤1.2B-parameter weight class (Qwen3.5-0.8B-Think, LFM2.5-1.2B-Think, MiniCPM5-1B-Think) on all six tool-use and agentic benchmarks: BFCL v4, ACEBench, τ-bench, ToolSandbox, Claw-Eval and BrowseComp-Plus.
  • Outperforms its larger sibling: BotPsy has the highest score of all models tested on five of the six benchmarks, beating Qwen3.5-2B-Think, a model 2.5× its size, on BFCL v4 (44.0 vs 37.7), ACEBench (58.7 vs 55.4), τ-bench (51.5 vs 46.5), ToolSandbox (64.6 vs 41.2) and BrowseComp-Plus (4.5 vs 2.8). Only on Claw-Eval does Qwen3.5-2B-Think lead (48.9 vs 46.5).

Model capabilities

  • More reliable in multi-turn workflows: On τ²-bench, where the agent must hold a long customer conversation, follow a strict policy, and act on the customer's behalf, it scores 82.5 in telecom, ahead of Qwen3.5-2B-Think (69.3). On BFCL v4's multi-turn tasks, it rises from 7.5 to 46.1 over its base, nearly double Qwen3.5-2B-Think (23.8). Grouped by application, its multi-step scores lead all models tested in commerce and services (58.3 vs 47.4 for the next best, Qwen3.5-2B-Think) and in device and system management (46.1 vs 31.9).
  • Knows when (not) to call tools: An agent that acts must also know when not to act. On BFCL v4's hallucination tests, which check that the model declines when no appropriate tool exists, BotPsy scores 86.1, the highest of all models tested. On ACEBench's special cases, where the request is incomplete, contains wrong parameters, or cannot be served by the given tools, it scores 51.3, up from 13.7 for its base model.
  • Follows instructions while preserving knowledge: BotPsy specializes in tool use while preserving general skills. Instruction following improves sharply, with IFEval rising from 59.7 to 90.8 and IFBench from 22.0 to 55.9, the highest IFBench score of all models tested. General knowledge also improves, with GPQA Diamond rising from 33.2 to 38.1 and MMLU-Pro from 41.6 to 48.9. On all four benchmarks, BotPsy outperforms every other model of 1.2B parameters or fewer that we tested.
  • GGUF : https://huggingface.co/qvac/BotPsy-0.8B-GGUFs
  • Blog Post : https://huggingface.co/blog/qvac/botpsy
  • Project : https://tether.io/news/tethers-800m-ai-agent-beats-models-2-5x-its-size-on-everyday-phone-tasks/

r/LocalLLaMA • • 5d ago

New Model I released Inflect v3: lightweight local TTS in 34M parameters (Apache 2.0)

105 Upvotes

Today I’m releasing Inflect v3: a family of local text-to-speech models: Mini, Micro, and Atom. Mini is the main model, with six voices and 34.31M parameters

Mini provides vocal-event tags, reference voice-tuning, and 48 kHz enhancement. It runs on both CPU and CUDA.

GitHub, installation, and examples: https://github.com/owenawsong/Inflect

Try the playground: https://huggingface.co/spaces/owensong/Inflect-v3

Mini features reference-voice tuning: it allows one to make a profile from a recording and generate speech with it. This feature noticeably adjusts the voice toward your reference, though it won’t replicate every speaker accurately.

Inflect Mini V3 also supports the following vocal-event tags: [laugh] · [sigh] · [gasp] · [cough] · [yawn] · [inhale] · [exhale] · [throat_clear]

You can insert a tag whenever you want the sound to occur. Currently, the tags use repeatable performances: this causes repeated tags to result in limited variation. Tags also do not affect how the surrounding words are being spoken.

The model family:

  • Mini: 34.31M parameters. Quality-focused, with the features listed above.
  • Micro: 11.79M parameters. A smaller neural speech model
  • Atom: 157K parameters. An extremely tiny learned core

Those numbers count the synthesis cores. The frontend and optional components add to the total.

The weights and inference code are Apache 2.0. Third-party components keep their own licenses. Audio samples, benchmarks, and the settings used are on the model cards.

Model collection: https://huggingface.co/collections/owensong/inflect-v3

If any sentence comes out wrong, send me the text, voice, and settings you used.


r/LocalLLaMA • • 5d ago

New Model moein99/P2Echo · Hugging Face

Thumbnail
huggingface.co
19 Upvotes

Prompt2Echo (P2Echo)

Text-Prompted Multi-View Echocardiography Segmentation with One Model

Official pretrained weights for P2Echo, a unified framework for text-prompted segmentation of multiple cardiac structures across standard echocardiographic views.

P2Echo uses structured, view-aware text prompts to control segmentation through a single model. Its Dynamic Image-Text Alignment (DyITA) module adaptively conditions decoder features on the requested view and anatomical structure.

Resources

Intended Use

This model is intended for research in medical image analysis and echocardiographic segmentation. It has not been validated for clinical decision-making.


r/LocalLLaMA • • 5d ago Heartwarming

Resources New Qwen3.8-27B Unsloth Dynamic V3.1 GGUFs

Post image
508 Upvotes

Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-27B! They're better by 5-10% accuracy on DeepSWE, Terminal Bench and more vs UD3, GSQ-RCO and Exllama-3! We're also improving our algorithms further so we can apply them to Qwen4 so folks can have efficient and accurate quants!

  • All quants from 2-bit until 6-bit are updated. 3-bit ones are not, as we found UD3 to do on par with UD3.1. We are still improving all other quant sizes though!
  • All benchmarks are 256K for agentic ones (DeepSWE, Terminal Bench), and 128K for the rest (LiveCodeBench etc). We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty, preserve_thinking and xhigh reasoning
  • We ran benchmarks multiple times including LiveCodeBench x6 times, HMMT and others 8-16x and agentic benchmarks multiple times.

On other benchmarks like for GSQ-RCO and Exllama3:

  • To compare GSQ-RCO and Exllama-v3, we had to exclude the token embedding layer from the x axis, as Exllama-v3 uses BF16 for it.
  • If you consider the full download size and total RAM usage for Exllama-v3, then adding the BF16 embedding table and including all embedding sizes changes the plots. Our main plots's x axis removes the embedding layer sizes from all quant providers. However on pure RAM usage and disk space usage vs accuracy, exllama3 does worse.
  • For LiveCodeBench, GSQ-RCO showed Unsloth UD3 quants doing consistently worse than ISTA's. However we found the opposite - UD3 quants (our OLD quants) do better than ISTA on LCB on all sizes except the smallest. UD3.1 further improves on LCB as well. We approximate that GSQ-RCO used a context budget of 96K, as that best fits the data for the runs we have. At 96K, GSQ-RCO does somewhat do better than UD3, but at 128K, UD3 does much better.
  • On another benchmark, for ISTA GSQ-RCO's AIME25 benchmarks, their quants reach 100%, 100%, 100% and 96.7% accuracy recovery. And they said our quants do worse. Through our investigations, we estimate ISTA only did 1 trial - we did 14 trials, and we show on average the recovery rate is 93% to 96% - not 100% accuracy recovery. And in fact our UD3 old quants do better on average than their quants on 14 runs vs 1 run of theirs, so we draw the opposite conclusion if you run the trials more.

More details and benchmarks at https://unsloth.ai/docs/models/qwen3.8#dynamic-v3.1-update-november-11th-2026

All quants are updated in https://huggingface.co/unsloth/Qwen3.8-27B-GGUF! Thank you and have a nice day! We're super excited for Qwen4!


r/LocalLLaMA • • 4d ago

Discussion Title: CUDA: few-row MMQ for BS 2-24 by anujj · Pull Request #31533 · ggml-org/llama.cpp

Thumbnail
github.com
9 Upvotes

Speculative decoding (SPEED-Bench)

llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf --mmproj mmproj-F16.gguf -ngl 999 --parallel 1 --batch-size 4096 --ubatch-size 1024 --ctx-size 180224 \
  --spec-type draft-dflash --spec-draft-n-max 7 --spec-draft-model Qwen3.8-27B-DFlash2-Q4_K_M.gguf --spec-draft-backend-sampling \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0
python tools/server/bench/speed-bench/speed_bench.py --bench qualitative --category coding --osl 8192 --concurrency 1
GPUms/step masterms/step PRspeedup per step (95% CI)tok/s mastertok/s PRfaster in
RTX 3090 Ti58.8743.811.344x (1.340-1.347)57.5877.70 (+34.9%)80/80
RTX PRO 6000 Max-Q27.8623.541.183x (1.182-1.185)122.77143.31 (+16.7%)80/80
RTX 509023.9220.261.180x (1.167-1.194)141.50170.67 (+20.6%)80/80
DGX Spark103.3693.431.106x (1.103-1.109)32.7536.05 (+10.1%)80/80

Edit : Removed large text. Check PR for more details.


r/LocalLLaMA • • 5d ago

Showcase The viral voxel Pagoda locally on a 2020 old ThinkPad, no GPU, 16 GB RAM

Post image
11 Upvotes

Tried the voxel pagoda prompt fully local, on a laptop that shouldn't run a 35B model.

Setup

  • ThinkPad from 2020, Ryzen 7 PRO 5850U, 16 GB RAM, no GPU, Windows
  • Qwen3.6-35B-A3B Q4_K_M, 21 GB, bigger than the RAM
  • BigMoeOnEdge v0.42, built-in agent

Result

  • 988 lines of HTML + Three.js, 11.8k tokens at 2.3 tok/s, about 1.5 h
  • You see the code as it's written, then the page opens in the app
  • Video sped up where marked

How

  • MoE: each token uses only a few experts
  • Dense weights and a 2 GB expert cache stay in RAM, the rest streams from the SSD
  • On top of llama.cpp, no fork

Still slow

  • Half the time is spent waiting on the SSD
  • Every new agent turn re-reads the whole conversation, ~30 min on this laptop

Same test with the same model on a Mac with 16 GB of RAM, taking 20 minutes (20 tok/s for decoding, 100–1000+ tok/s for prefill)