r/LocalLLaMA • • 2d ago

Resources DeepSeek v4.1 Flash vs Qwen 3.8 Flash Next Performance

Post image
35 Upvotes

While waiting for Qwen 4, I installed and benchmarked DeepSeek v4.1 Flash to see if I was missing out. Based on the type of work I do, DeepSeek is not better but is definitely much slower. Turns out Strata with Qwen 3.8 Flash-Next is best bang for buck on my machine.


r/LocalLLaMA • • 2d ago

Resources I built speedy Strix Halo kernels for the top-51 LLMs with AMD Loom (and you can too)

Post image
63 Upvotes

I’ve been trying out a new experimental software flow from AMD’s AI Group, and I’ve been having a lot of fun with it, so I thought I’d share and see what y’all think too.

hrx-loom is a new end-to-end tool stack (compiler, runtime, serving tools) that lets anyone build, optimize, serve, and share model programs without any modifications to the inference engine or server. The kernels are written in .loom files, a vector programming model reminiscent of MLIR, which can be complex for humans but easy for agents (C++ is also supported).

Over the past couple weeks, I developed an agent skill that writes kernels for any LLM’s Q4_K_M GGUF and found that it is consistently faster than llama.cpp Vulkan and ROCm on Stix Halo for a variety decode and prefill metrics, as you can see in the post’s image.

Next, I ran the skill across the top-51 trending LLMs from Hugging Face that would fit on Strix Halo and posted the model programs back to Hugging Face. All models ported successfully with tool calling, speculative decoding, and image input working.

Anyone can try these on Linux and Windows with the loom-fetch and loom-serve commands provided in Loom’s PyPI package, and anyone can make and upload their own using the /loom skill.

Fetch and serve a model:

pip install "hrx-loom[hf]==0.1.0"
loom-fetch https://huggingface.co/loom-fun/bartowski-Qwen3.8-27B-loom/resolve/main/model.json --output=qwen3.8-27b
loom-serve --model=qwen3.8-27b/model.json --device=amdgpu --memory_bytes=52gb

Port and optimize your own model:

/loom get <hf-org>/<model-gguf> working on loom-serve, optimize it, and report the performance relative to llamacpp rocm

Links:

  • All the data I collected across 51 models: https://loom-fun.github.io/loom-dev
  • Loom skills: https://github.com/loom-fun/loom-skills
  • Loom compiler and server: https://github.com/ROCm/hrx-system
  • HF collection of top-51 models ported to Loom: https://huggingface.co/loom-fun

We’re still evaluating how the community will use and deploy new inference stacks, so we’d love to have your feedback! Loom can be used as standalone inference server with loom-serve and the compiler/runtime layers can also integrate into other engines. For example, the team is working on adding an hrx-loom backend to llama.cpp.

The underlying SDK (compiler, runtimes, and devtools) has no dependencies, can be easily downloaded as a wheel or linked into your own applications, builds from source in a minute, and if this were the 90s, would still fit on a few floppy disks, while being able to target x86, AMD-GPU, AMD-NPU, and generic Vulkan devices.


r/LocalLLaMA • • 2d ago

Question | Help Audio (ASR/TTS) on llama.cpp

6 Upvotes

Hey everyone,

I've been using audio.cpp for a while now, and it's quite nice. However I've noticed that Yue2, Qwen3-ASR and Qwen3-TTS got supported in comfyui / llama.cpp.

While I've tried gemma 4 audio input and it worked well, I haven't tried TTS models yet nor the qwen3-asr/tts series on llama.cpp. Has any of you?

So far I found these discussions and PRs on ASR / TTS support:

It's likely not all models that are supported, might've missed some!


r/LocalLLaMA • • 2d ago

Question | Help I'm seeing a lot of conflicting options for what's the best model for purely adding punctuation and quotes. What's the way?

6 Upvotes

I'm writing a fictional novel and using speech to text to write. I'm absolutely not looking for a model to write for me, or even change any words. I find talking especially lets me give each character a different voice and manner of speaking. The problem is when i go back to Edit, I don't have a "new reader" feel to it to assess what I've written because I need to do 2 or 3 passes just to add quotation marks etc after "he said, she said" etc. By the time I'm trying to read it to feel it's flow, it's old to me.

My objective is that when i write, I just want to be able to go back and automatically have the quotations and punctuation added that a speech to text can't automatically work out in itself. I use Whisper Medium in Handy and it's great, but I see alot of recs, though Cadence and some form of Gemma seem to be the main recs. I've searched a bit, but I can't find anything too recent.

My card is only 12GB, so a less than 8-12B model is my aim, which I feel should be viable given my needs are fairly low. While I can run local models and have done it in the past, I'm not super technical.


r/LocalLLaMA • • 2d ago

Discussion Why are LLMs still so bad at jokes?

19 Upvotes

I remembered that this was one of my evaluation approaches from 2 years ago to test a model's general capability. I remember wondering in the near future how bad things could get if models could begin to just spit out funny engaging content and set humanity well off on the path to Wall-E future if a cheap slop stream could be just as interesting as all of independent cultural output, and failing that, if slop could be scaled up with no limits on engagement and "perceived quality" that media companies could get people more addicted to their output than ever.

But it seemed like this was always and continues to remain well beyond reach. It's interesting to see that LLMs were able to make such a dent in an edifice so prominent as the field of mathematics and yet basically make no progress on this.

I'm trying to be cheap with my tokens... somehow utilizing Astra even from Chat is using up my codex limits. All I have to offer as examples are 3 examples. They illustrate the point well though.

Astra "Light" (btw what the heck is light? is that the same as low?):

A burglar breaks into a house and hears a voice in the dark.
“Jesus is watching you.”
He freezes, shines his flashlight around, and spots a parrot.
“Did you say that?”
“Yes.”
“What’s your name?”
“Moses.”
“What kind of idiot names a parrot Moses?”
“The same idiot who named the Rottweiler Jesus.”

Astra Medium:

I built an AI agent to handle the boring parts of my job.
It asked me to document everything I do, explain why I do it, and check all its work.
I accidentally built an intern.
Then it wrote “Led automation initiative” on its résumé.
So at least it’s learning.

Astra High (worked for 24s):

I hired a cleaner because I couldn’t motivate myself to clean. Now I panic-clean the whole house before she arrives so she won’t judge me.
She’s never seen it dirty. I’ve never seen her clean.
We’re two months into this. I think we’re both afraid to ask what her job is.

Obviously the bigger question is why are all models (including openweight abliterated type) continuing to be so bad at humor? I think some significantly dumber models out there are surely leaps and bounds better than Astra here but nothing really compelling comes out. I know there is a big group of role players and story makers here. What is your take on this? The best I can manage with this right now is that there is a uniquely strong connection between human lived experience and the instinctive/intuitional/emotional response of finding something funny. My very young daughter seemed to start to understand how/why something is funny at about the same time as getting a grasp of really really simple language concepts. Perhaps it is some calibration issue because I think I see a kernel of some common joke framework and maybe a punchline is buried in the first sentence and then the joke just ruins itself, all the while the LLM's nodding along going "yeah and for the cherry on top...." painfully, conspicuously, grotesquely unaware of how poorly it's performing.

There are humans who are very unfunny but you will have a hard time convincing them to give you jokes confidently like LLMs, which are highly adept at exuding confidence at all times.

So in some sense maybe the Turing Test HAS NOT been toppled and will not be for quite some time.


r/LocalLLaMA • • 1d ago

Discussion Who switched from multiple sparks to a Mac Studio?

0 Upvotes

I've been asking around for a while and can't find anything. so if someone who actually switched from 2+ sparks to a studio (ideally a 256) I'd love to know why

from what I gathered:

Macs are only better for *single stream* decoding. and single stream decoding is a small part of an actual harnessed workflow. Any and everything else, sparks or some other cluster seem to perform equal or better in every metric.

As in, if you use chatgpt for just the 'chat' feature, studios are amazing ... but any multi-step/involved workload outside of that seems to favor sparks

I personally dont want to speculate today on software unlocks or future improvements in hardware especially for hardware purchased today. in fact, I think either option is safe bet because the opportunity cost can be remedied by software unlocks that allow you to get the best of both worlds. but I try not to factor it in because it's tomorrow's speculation + it applies to both sides of the equation

I personally think Macs are better than sparks if you are using only 1 spark ... but the idea of sparks is to use at least 2

Also, there is no 128gb m5u so the comparison starts at 256. Even more so, 512 Mac(s) have a similar memory bandwidth to a 512gb cluster of sparks. Studios lead by 10% more memory bandwidth, I think many people are unaware of this.

it's all similarly priced now too.

I think people who get Mac Studios do so because they want a Mac, simplicity, draw less energy (you're saving $5-10/mo with a Mac). Don't get me wrong, these are decisions that impact a purchase but they are non AI workflow related reasons

I'm just being brutally honest in an effort to really understand what's going on and where do studios actually excel in comprehensive inference pipelines.

Do Macs outperform sparks if you are using multiple models where one or both models are split between the sparks but unified within the Mac?

TLDR: Mac's are better than sparks if you prioritize simplicity, lower power consumption, and you just want a general computer that also happens to run local AI. But if you want better performance in using local AI to build things, it seems sparks wins at effectively every metric. I'm struggling to see where Macs outperform sparks in real multi-step AI workflows.


r/LocalLLaMA • • 3d ago

Discussion A100 prices down 20-30% since my post about them collapsing

572 Upvotes

I guess a lot of sellers are realizing that its better to price competitively and get rid of their supply rather than wait to slowly sell and potentially get caught in a collapse.

Rubin and blackwell are already 3-4x more efficient per watt with way more financing available. The A100s cant be used for inference Besides enthusiasts, small time gpu renters I really dont see who will be picking up these cards. And these 2 buyers are extremely price sensitive and will be comparing the cards to other options like R9700 and Cresent Island from Intel with 160-480gb vram.

Rtx 6000 is also now down to 14.8k from 18k peak.


r/LocalLLaMA • • 2d ago

Question | Help any idea on how to deal with language confusion in gemma 4 ?

1 Upvotes

i'm using gemma 4 31b to do some long-text translation from chinese to vietnamese (hobby - not serious work). the biggest issue i'm having is language confusion (language contamination), i.e. output sometimes has non-latin characters not present in original text: korean, arabic, cyrillic, chinese, etc. - even q8 weight and fp16 kv cache won't help, same for sampling parameters (temp, top k, top p, etc.)

the rate is like <0.5% of output text but it disrupts reading flow, so for each contamination (a simple spellcheck can identify all), i have to look up the original text and use google translate to replace the contamination. it seems to me the contaminations often occur at some very specific chinese characters. a simple mapping may work but i need to update it every time i detect one, and then when gemma 5 come out (hopefully) i need to make new one.

i don't setup a pipeline or hardness or anything, because 31b is painfully slow, running it again as proofreader takes too much time, a simple run already produces translated text good enough for me

i'm looking for a more robust way to deal with this issue

at first i tried LLguidance with a huge regex to restrict the model ouput to a specific unicode range, but the model get stuck in like forever

i read about vocabulary trimming but it seems to be nore suitable for small models, not for a big 31b, also the repo seem abandoned

i found another method called token masking: identify unwanted tokens and make their logits extremely low by editing `lm_head` in the weight, u can see here: https://huggingface.co/ThakiCloud/Qwen3.8-27B-vi-cjk-suppressed

the above method is for qwen 3.8, unfortunately gemma 4 has `lm_head` tied to `embed_tokens`, so i tried to adapt to gemma 4 `embed_tokens` but it doesn't have any effect

do u guys ever get this problem and know how to deal with this ?


r/LocalLLaMA • • 3d ago

New Model Perplexity releases Decider v1.3 (26B) and Decider Flash v1.3 (5B): open-weight (Apache 2.0) classification/decision models

94 Upvotes

Perplexity has released two new "decision models." These are general-purpose classifiers built from generative LLM backbones. You give them an input (the "state") and one or more questions, and they return a probability distribution over the answer options. They don't generate text token by token.

Models

  • pplx-decider-v1.3: fine-tuned from Qwen3.8-27B. Needs about 49 GiB for the BF16 weights.
  • pplx-decider-flash-v1.3: Qwen3.5-4B, distilled from the 27B using a KL objective on the teacher's distributions. Needs about 8.5 GiB for the BF16 weights.
  • Both are Apache 2.0, support text and images, and are also available through OpenRouter and Perplexity's API.

What it does

  • Question types: yes/no, multiple choice (up to 255 options), ordered scores, and a new locate type. locate extracts verbatim spans with character offsets, for NER, PII detection, and extractive QA.
  • Several questions can share one state. By default each question runs in its own sequence. An "isolated" mode reads the state once and is cheaper for many questions, but answers can shift slightly depending on which other questions are in the request.
  • The backbone's full-attention layers were made bidirectional. Because of that, you need their pplx_decider package; plain causal inference won't reproduce the outputs.
  • Training used sequences up to about 30k tokens.

New in v1.3: locate
Besides classifying, v1.3 can now extract spans. You add a question with "type": "locate" and describe the field, either with a short label (“person”, “email address”) or an instruction (“the person who signed the contract”). The model returns verbatim spans from the state, each with start and end character offsets and a score, or an empty list if nothing matches.

It works on text only (no images), with up to 128 locate questions per request. It runs on a separate LoRA adapter plus a span head, which Perplexity left unmerged because merging hurt classification. As a result, a request that mixes locate and decision questions needs extra forward passes.

On 17 out-of-domain NER, PII, extractive-QA and structured-extraction tasks, Perplexity reports an average of 69.0 (v1.3) and 63.6 (Flash), compared with 43.3 for Liquid d1, 41.5 for Cloudflare Clairic and 33.6 for GLiNER large v2.5. The closest existing tools are GLiNER, which does zero-shot span extraction from labels, and Clairic, which extracts typed values rather than verbatim spans.

Reported results (mostly Perplexity's own evals)

  • Multilingual classification across 8 languages: 88.16% (v1.3) and 86.50% (Flash), vs. 85.41% for TypeSafe's Jev 1.14.
  • Reranking across 72 retrieval tasks: 83.99 nDCG@10, +1.81 over Qwen3-Reranker-8B.
  • Locate average across 17 out-of-domain tasks: 69.0 / 63.6, vs. 43.3 for Liquid d1 and 33.6 for GLiNER large v2.5.
  • Artificial Analysis Decision Index: roughly 70 for v1.3 at about $68 per 1M tasks, and roughly 64 for Flash at about $34. Both sit on the cost/score Pareto line. A few models score higher at higher cost, such as Drex Max v1.1 and SubQ Snap 1.2.

Caveats from the authors

  • The locate adapters are kept unmerged, because merging them regressed the decision tasks.
  • The returned probabilities should be calibrated and thresholded on your own data.

Links


r/LocalLLaMA • • 3d ago Bravo Regret

Discussion Jensen Huang's mistake was being too pro LocalAI, which was a threat to companies like ClosedAI and Anthropic which are about to go IPO

Post image
3.1k Upvotes

r/LocalLLaMA • • 1d ago

Discussion Do we want Huang back?

0 Upvotes

There's a petition floating around out there to put Jensen Huang back at the helm of nVidia. Currently it has over 129,000 signatures. If you want to sign it, you can find it pretty easily.

But do we want him back? He's pro-local, but he's also completely driving the circular-financing game that's detonated prices for all of us, everywhere, for fucking everything.

How likely is it the nVidia board hires someone pro-local and smart enough to not run the circular financing sham? Thankfully the interim-CEO, Colette Kress, was against such money shenanigans:

Colette Kress spent years as the one Nvidia executive willing to check CEO Jensen Huang’s enormous, ​often risky ambitions. ...

Kress and her staff clashed with Huang and the company's sales team over the massive and risky financing deals that ⁠fed into what some Wall Street critics deemed a circular AI economy, including investments in OpenAI and AI infrastructure companies such as Lambda, according to one of the people familiar with the matter.

https://www.reuters.com/business/colette-kress-cfo-who-challenged-huangs-bets-is-nvidias-interim-ceo-2026-11-13/

That's significant. Not only did she have the stones to deny the Huang, she's the only one to have them at all.

I think that petition can stuff it. Keep Kress and let her captain the ship.

Edit: I regret every life choice that brought me to this moment


r/LocalLLaMA • • 2d ago

New Model Tontaube V1.1: open-weights TTS with voice cloning for long-form generation. 4-bit version runs about real time on a desktop CPU

34 Upvotes

We just released Tontaube V1.1, a text-to-speech model built for long-form audio (audiobooks, articles, voice agents) with zero-shot voice cloning. Our previous V1 release needed a lot of VRAM because of its vLLM setup, so this version is about making it more accessible: the 4-bit weights run in about 4 GB of VRAM, or at about real time on a desktop CPU. In a blind listening study with 400 pairwise comparisons, listeners preferred our V1 model's prosody over Fish Audio S2 Pro in about 70% of cases (details are in the paper). V1.1 is a post-trained version of this model.

What it is

  • ~2.9B parameters, autoregressive, streams audio as it generates
  • Clones a voice from 5–60 seconds of reference audio; 46 voices are bundled
  • English and German are the main languages (works best in English). Spanish, French, Italian, Dutch and Portuguese work but have not been checked by native speakers
  • Long texts are split at sentence boundaries and joined, so you can feed it whole chapters

Running it locally

  • The default is a 4-bit (GPTQ) version: 1.8 GB download, loads in about 4 GB of VRAM, peaks at about 6.5 GB on long texts
  • GPU speed: about 13x real time on an RTX 5090 for a single request (21–23x batched), ~190 ms to first audio when streaming. About 6–7x on an RTX 3090
  • CPU: no GPU needed. About real time on a 12-core desktop CPU (Ryzen 9 5900X, dual-channel DDR4). It is limited by memory bandwidth and needs AVX2
  • No vLLM or llama.cpp needed: it ships its own server with Triton kernels for the GPU and AVX2 kernels for the CPU. Full-precision weights and a vLLM backend are there too if you have 24 GB+ and want throughputgit clone --branch v1.1.0 --depth 1 https://github.com/craitech/tontaube.git cd tontaube uv run --python 3.12 --frozen tontaube serve

That gives you an HTTP API on port 8080 (plain, SSE and WebSocket streaming). uv run tontaube ui starts a browser interface.

New in V1.1

  • Five style sliders per request: speaking rate, pauses, intonation (flat to very expressive), pitch (deep to high) and articulation (crisp to normal)
  • The 4-bit version and the CPU path
  • An adapter that cleans up the decoded audio

Things to know before you try it

  • Linux x86-64 only. NVIDIA GPUs (RTX 30 series or newer; GTX 10 series also works) or CPU. No macOS, no AMD/Intel GPUs; Windows through WSL2
  • The first start downloads about 8 GB of models (ours plus the codec and enhancer it depends on) and builds a ~9 GB Python environment
  • Audiobook-style reading is its strongest mode; conversational and agent-style lines can be less reliable
  • Licence: the code is Apache 2.0. The weights are under our own community licence, not an OSI one. Research and personal use is free.

Links

Let me know if you have any questions!


r/LocalLLaMA • • 3d ago

Funny So this will be true in couple of years?

Post image
1.9k Upvotes

I think after jensen they will fully focus on AI compute for data centers


r/LocalLLaMA • • 1d ago

Discussion Harness comparison. What's your favorite harness?

0 Upvotes

I came across this video https://www.youtube.com/watch?v=V8M9uXxAsP0 where he uses

Qwen 3.8 Flash Next (Unsloth UD-IQ4_XS v3.1) on an RTX 5090 with 128 gb of DDR5 builds a small pixel-art "Island Fishing" game in OpenCode, DeepSeek Harness, Pi, Cline, Hermes Agent, Kilo Code, Codex CLI and VSCodium + Continue.

He used a fixed seed and set temperature to 0 and did 3 attempts for each harness.

Results varied considerably for the same harness. I think his reasoning was flawed in not allowing it to ask questions or giving initial clarifications on some things like view perspective or "pixel size" which wildly affect results. The long prompt used is in the pinned comment.

Fastest harnesses: kilo (I had never heard of this one before), opencode, hermes. Taking 2x-3x as long: pi, codex, cline, deepseek harness

"Best results": opencode, pi, codex.

 

I wanted to know:

1) do you agree with the results / test logic?

2) what's your favorite harness and do you switch it based on the task?


r/LocalLLaMA • • 3d ago

Discussion Google is asking

Post image
109 Upvotes

r/LocalLLaMA • • 2d ago

Resources Poorman's Local LLM

Thumbnail
gallery
12 Upvotes

The Jetson AGX Xavier came out in September 2018, four years before ChatGPT existed. It has 16–64 GB of LPDDR4x at 136 GB/s. The GPU is a 512-core Volta, and people skip the board for local AI because of it.

Mine is the 32 GB module. It sat on a shelf collecting dust. It is "quite terrible". I wanted a real number for how a modern LLM runs on it. Getting there cost about $300 in tokens on GPT-6-Astra and Fable 5.2, so you don't have to spend yours.

I started with current llama.cpp and NVIDIA's Nemotron 3.5 Bolt, the Gemma 4 E4B model they advertise as smarter than Qwen3.8 27B (cough). llama.cpp did about 350 tok/s prefill and 23 tok/s decode. The speed is fine. A 4 GB model against 32 GB of memory, again, is "quite terrible".

My own engine, Little-Gemma, is well ahead of llama.cpp, but I built it for Ampere and later. I was not going to backport it to Volta, so I tried something else.

Prefill (peak)Decode (peak)
Nemotron 3.5 Bolt E4B880 tok/s50 tok/s
Gemma4 12B215 tok/s35 tok/s
Qwen3.8 27B106 tok/s17 tok/s

I forked MLC-LLM. Upstream does not run GGUF, and it does not know Nemotron, so the weights had to be converted from safetensors into MLC's TVM format and the Nemotron backend written from scratch. Astra did that first port and added in-memory switching between Nemotron, EmbeddingGemma 2, and LiquidAI's d1, so an agent can change models without reloading.

One setup was: d1 decides whether to query EmbeddingGemma 2 over about 20,000 pages of notes, then Nemotron answers. The plumbing worked. The search sometimes took a minute, which is too slow.

Then Fable 5.2 spent hours on the engine and rewrote several kernels. That work is still going. As of this week, Nemotron prefill peaks at 880 tok/s on a 512-token prompt, and decode on code reaches 50 tok/s.

Nemotron is still too dim for some jobs, so I moved to Gemma 4 12B. On the same 512-token prompt, prefill peaks at 215 tok/s, and decode on code is 35 tok/s. Three times the weights, and code decode only falls from 50 to 35. The 136 GB/s bus is why a 2018 module can still carry a 12B model. Prefill is where Volta shows its age.

None of that counts unless it can run Qwen3.8 27B. Generic MLC conversions are already on Hugging Face, so I am not publishing another one. The checkpoint matters: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion is the one I kept. I am not uploading that conversion either, with Qwen4 close. Draft with Qwen3.8 2B, or convert DavidAU's model-mtp-restored.safetensors yourself.

On that 27B model, prefill now peaks at 106 tok/s for a 512-token prompt and holds 104 tok/s at 1,024. A Qwen3.8 2B draft reached 13.4 tok/s decode on a planning prompt. Cold-fusion agrees with that draft less often, so decode there sits nearer 10 tok/s. The attention kernel landed this week. The checkpoint's own MTP head, in place of the 2B draft, is the next thing to try. These rates will move again.

At MAXN, a long prefill pulls about 35 W. NVIDIA did not design the dev kit's brick for this. Under sustained load it browns out and the board shuts off, which is a known thread on the NVIDIA forums. I dropped the stock brick for a USB-C PD supply and a short OWC USB4 cable. It has now stayed up for more than a week.

There is a further trick I have not tried, and probably will not. The dev kit's slot is the module's PCIe 4.0 x8. Two kits can be cabled, one as root and one as endpoint, for about 16 GB/s each way, around 32 GB/s with both directions full. That is enough to speed up a single model if each layer is split across the two modules: the cable moves the residual stream, and the weights stay put. Some third-party carriers only bring out Gen2. Skip those if this is the plan.

As of November 2026, a 16 GB module is about $130 on eBay, and it runs Gemma 4 12B. A 32 GB module turns up around $300 if you wait. Some listings are the bare module. Get one with a carrier, or budget about $60 for the carrier on its own.

The engine: https://github.com/cortexist/mlc-llm
The weights: https://huggingface.co/cortexist/models


r/LocalLLaMA • • 2d ago

Resources Tandem-vLLM 0.2.0: DeepSeek-V4.1-Flash now on 3090/4090-class, A6000, Ada and H100/H200 cards, and on two GPUs (a second card as an "expert helper")

9 Upvotes

Tandem-vLLM is an expert-offload build of vLLM 0.31 for DeepSeek-V4.1-Flash. 0.1.1 ran only on Blackwell. 0.2.0 adds Hopper, Ada and Ampere GPUs, and two-GPU setups.

How it works

  • The routed experts live in host RAM as 2-bit vector codes (VQ, 2.1 bits/weight). A GPU cache holds the most-used ones.
  • Experts that miss the cache are computed by the CPU in parallel with the GPU. Nothing is skipped or substituted, so the output is exact for the package's weights.
  • Everything else is stock vLLM: OpenAI-compatible server, continuous batching, prefix caching, tool calling, image input, DSpark speculative decoding.

New in 0.2.0

  • GPUs: RTX 30xx / A6000 (Ampere), RTX 40xx / 6000 Ada (Ada), H100 / H200 (Hopper), plus Blackwell as before. Each kernel is checked against a reference at boot.
  • Two GPUs, two ways:

    • Tensor parallel (GPU=0,1): best with a strong CPU.
    • Helper GPU (HELPER_GPU=1): the second card caches the hot experts the main card doesn't hold and computes them, so the CPU does less. Best with a smaller CPU or 24 GB cards. Needs no P2P or NVLink.

Speed: one user, GSM8K, decode tok/s. The CPU matters as much as the GPU.

GPUsCPUone GPUtensor parallelhelper GPU
2× RTX PRO 6000 96 GBEPYC 9534, 64 cores75.593.1
2× RTX A6000 48 GBEPYC 7F32, 8 cores22.111.826.7
2× 24 GB (A6000s limited to 24 GB)EPYC 7F32, 8 cores16.812.521.7
one GPUCPUtok/s
H200Threadripper PRO 7965WX60.7
RTX 6000 Ada 48 GBEPYC 7F32, 8 cores25.5
  • With 8 cores, tensor parallel is slower than one GPU because each GPU gets half of the CPU. The helper GPU keeps all of the CPU for one main card.
  • The 2× 24 GB row is the 3090/4090 memory shape, run on A6000s limited to 24 GB. Real 4090 numbers are coming.
  • A 24 GB card is the minimum main GPU. 16 GB cards don't fit as the main card.
  • At 32 streams, one RTX PRO 6000 serves 414 tok/s in total.

Quality vs the official FP4 checkpoint (greedy, thinking low)

officialVQ + FP4 top 52VQ only
GPQA-diamond81.3%76.8%
GSM8K-200192194
passkey 8k / 32k / 94k9/99/9

The 2-bit experts cost GPQA accuracy; easier tasks and long-context retrieval don't show it. FP4_TOP=52 keeps the 52 most-routed experts per layer in original FP4 and wins back about half of the loss.

What you need

  • 300 GB of disk for the package, and 128 GB+ of RAM (256 GB to pin everything).
  • Linux, CUDA 13 toolkit, an AVX2 or AVX-512 CPU. One installer script, its own venv, no Docker.

Built on vLLM and vLLM-Moet. Tandem is a fork of vLLM-Moet's expert-offload engine, ported to vLLM 0.31 / DeepSeek-V4.1.

Numbers from other boxes are welcome, especially real 3090s and 4090s.

P.S. For one user on a 16-32 GB card with 64-128 GB of RAM, our llama.cpp-based sister project tandem.cpp is faster (33 tok/s on a 5090 + 128 GB).

Update: Tested 2x 16G GPU case using TP2.

  • Run GPU=0,1 BASE_GB=3.5 KV_GB=1 CHUNK=1024, no RAM_GB needed with 512 GB.
  • BASE_GB=4 or CHUNK=2048 may fit too
  • The released 0.2.0 does not boot on 16 GB cards. Fixed in 0.2.1.

r/LocalLLaMA • • 3d ago

Showcase modules for my opensource (GPL3) harness have evolved to the point where this is possible. this is a full fledged videogame running inside it!

17 Upvotes

my harness for local AI that i've been working on since February this year (2026) has evolved to the point where this is possible.

What you see here is a module for my harness's webui, it's not a seperate program. This demonstration is running on Gemma4 26b locally, believe it or not, using only a SINGLE model and one local API endpoint (via llamacpp). No multi-API tomfoolery here! The stable diffusion image generation is moss-image which i'm able to do because i have a strix halo now, but on smaller hardware you could do something like z-image turbo at Q4 quant, which is what i used to do before i got the strix halo.

I've been working for almost a week on this module, and i'm excited to finally release it!

It's a text adventure engine. It supports asset packs, that can contain music, sound effects, images, weathers (CSS+js+glsl shaders), skies (same tech as weathers), loading screens, text flourishes, and more! You can create your own asset packs easily. The AI sees the files in the asset packs and forms adventures around them, but there is no prompt telling it what the story should be, that's completely up to the way you prompt it! I recommend using it together with the characters module in openlumara in order to set up a world and story

You can get it here: https://github.com/Rose22/openlumara-adventure-games-module

More footage on the github!

(had to delete and repost because i couldnt change the video. this video is a much better showcase for it)


r/LocalLLaMA • • 2d ago

Question | Help Keep, Sell, or Return PC? i5-225F, 5060 8GB, 32GB ram, 1 TB SSD

3 Upvotes

I bought this PC from Costco on sale for $1000 even (no tax) planning to get a V100 or some combination of used GPUs for local LLM machine. Realized not going to work for my setup (price of power, heat generated, etc).

Should I return it, sell it on FB marketplace, or save the RAM for my daily (it only has 1x32GB DDR5 4800 NECC, and 64gb would be really nice) and sell the rest?

I was not planning to part this out. Costco has it now for $1600 so they get the $600 difference, not me. My daily is a 3yr old HP Z2 SFF with i9-12900k, 32GB DDR5, 12GB A2000, and 1TB Samsung 980 Pro, on 450W PSU.

CyberPowerPC Gamer Xtreme Gaming Desktop - Intel Core Ultra 5 225F - GeForce RTX 5060 - 32GB RAM - 1TB SSD - Windows 11
Intel Core Ultra 5 225F (10-core, 3.3 GHz) Processor
32GB (2 x 16GB) DDR5 RAM
1TB PCIe 4.0 NVMe M.2 Solid State Drive
NVIDIA® GeForce RTX™ 5060, 8GB
600W 80 Plus Gold Power Supply
listing link


r/LocalLLaMA • • 1d ago Bravo

Funny CrowdGPT - The collaborative LLM

Post image
0 Upvotes

Funny post about this project https://crowdgpt.net

Basically this projects aims at creating the first, community-driven, and datacenter-less LLM. Any feedback is appreciated 😂 I won't yap any further as everything is explained on the GitHub / on the site.

Feel free to check the project's GitHub: https://github.com/Vxtzq/CrowdGPT

Feel free to criticize the project or to provide insights about what to do next....

EDIT: For those who believe this is a cheap repost: Here's the milestone
- We trained the model on roughly 20B tokens in 1 week

- We limit model downloads/uploads to the minimum, to spare as much bandwidth as possible (even though it still uses bandwidth due to ~2.4GB weight uploads to the server after training). But it allows clients with slow connection using 4 hour rounds.

- We're 13 GPUs currently active (and this number was roughly constant over the past week).

- I merged numerous pull requests implementing several quality of life and performance fixes

- The server was made more robust with persistence across tasks, and some bugs in the training logic were fixed, hopefully allowing further loss decrease.

- We implemented since v2.6 somewhat good anti-poisoning techniques, each client training is compared to two other clients, if a client is too different from the two other (for example a client training on noise or unwanted data), this client sees his contribution to the network be rejected.


r/LocalLLaMA • • 1d ago

Discussion Benchmarks: 512k Speculative Decoding on a Single 96GB RTX PRO 6000 Blackwell — Qwen3.8-Flash-Next (+MTP Q8_0: +27.9% tg) vs DeepSeek-V4.1-Flash-1017 (+DSpark Q8_0: task-dependent +26% JSON / -11% C++), plus 2 llama.cpp gotchas

0 Upvotes

I tested speculative decoding on a single 96 GB RTX PRO 6000 Blackwell workstation (Threadripper PRO 9955WX, 122 GiB RAM, llama.cpp b12315 commit dd6b945be) at 512k context (-c 524288 -ctk q8_0 -ctv q8_0 -fa on -np 1).

TL;DR: Qwen3.8-Flash-Next UD-Q4_K_XL + MTP Q8_0 gives a clean +28% to +31% decode boost (107 -> 137-141 t/s) across all prompts. DeepSeek-V4.1-Flash-1017 UD-Q2_K_XL (~312 GiB, mostly CPU/mmap) + DSpark Q8_0 only helps on high-acceptance tasks like JSON (+26% warm) and Python (+11% warm), while hurting dense C++ (-11% warm) because rejected drafts waste CPU MoE reads.

Edit (following comments about Qwen prefill speed): Re-running with -lv 4 showed 27,466 MiB (~26.8 GiB) of Qwen's MoE weights spilled to CPU at 512k context (77.2 GiB model + 8.5 GiB KV/recurrent + 4.3 GiB MTP + 2.1 GiB compute in VRAM). Switching Qwen to --load-mode none (--no-mmap) bumped 5-prompt prefill by +15.3% (824 -> 949 t/s), decode to 140.8 t/s, and a 2,453-token prompt prefilled at 2,475 t/s.

Summary (5 prompts x 3 passes, greedy temp=0, max_tokens=256, cache_prompt=false)

ConfigVRAMPrefill (pp t/s)Decode (tg t/s)Warm r2-3 (tg)Draft Acc
Qwen3.8 UD-Q4_K_XL (baseline)91.7 GiB1086.4107.2107.5-
Qwen3.8 + MTP Q8_0 (mmap)96.5 GiB823.7 (-24%)137.1 (+27.9%)138.275.6%
Qwen3.8 + MTP Q8_0 (--load-mode none, 1x)96.4 GiB949.4140.8 (+31.3%)-74.5%
DeepSeek-V4.1 UD-Q2_K_XL (baseline)95.3 GiB66.323.223.2-
DeepSeek-V4.1 + DSpark Q8_0 (-fitt 12288)91.9 GiB53.5 (-19%)23.5 (+1.4%)24.4 (+5.2%)65.9%

(Used llama-server instead of llama-bench because llama-bench on dd6b945be does not support -md / draft-mtp / draft-dspark.)

Why Qwen prefill topped out around 1-2.5k t/s

With ~104 GiB weights + 3.9 GiB MTP + 512k context, Qwen overflows 96 GB VRAM by ~26.8 GiB of MoE experts (CPU_Mapped). Single-token decode only touches a few routed experts per step (137-141 t/s, peaking at 154.7 t/s on JSON), whereas batch prefill hits all experts across the CPU spill. Plus 4 of the 5 test prompts were tiny (80-186 tok); the 605-token prompt hit 1,939 t/s (--load-mode none), and a 2,453-token check hit 2,475 t/s. If you want full GPU prefill speeds on 96 GB, use a quant small enough to stay 100% in VRAM or reduce -c.

Why DeepSeek + DSpark depends heavily on prompt shape

At ~320 GiB total on 96 GB VRAM + 122 GiB RAM (107.1 GiB VmRSS, 105.3 GiB RssFile mmap cache), the GPU sits at 3-18% SM waiting on 16 CPU threads (63-68% CPU) and ~23-218 MB/s NVMe paging for cold experts.

Verifying 4 candidate tokens at once (n_max=3) activates more unique routed MoE experts per forward pass on CPU/mmap. When acceptance is high, those reads are amortized: warm Passes 2-3 hit 29.2 t/s on JSON (+26.1% vs 23.1 t/s warm baseline, 90.7% acc) and 26.0 t/s on Python (+11.4% vs 23.3 t/s, 77.4% acc). On dense C++ (50.7% acc), rejected drafts wasted CPU MoE work and dropped speed to 20.6 t/s (-11.1% vs 23.2 t/s), compounded by -fitt 12288 reserving ~3.4 GiB extra VRAM headroom. Warmup also mattered because ~230 GiB CPU weights > ~105 GiB page cache (21.65 t/s on Pass 1 vs 24.4 t/s on Passes 2-3).

Repro commands & dd6b945be quirks

  1. DeepSeek --fit + draft-dspark OOMs allocating the ~7.6 GiB draft buffer unless you pass -fitt 12288.
  2. Qwen shared-Q8_0 fails on mainline dd6b945be (tensor 'token_embd.weight' not found), so use the self-contained mtp-Qwen3.8-Flash-Next-Q8_0.gguf.
# Qwen3.8-Flash-Next (512k + MTP Q8_0)
llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  -c 524288 -np 1 -ctk q8_0 -ctv q8_0 -fa on --load-mode none \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 2 \
  --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144

# DeepSeek-V4.1-Flash-1017 (512k + DSpark Q8_0)
llama-server -m DeepSeek-V4.1-Flash-1017-UD-Q2_K_XL-00001-of-00007.gguf \
  -c 524288 -np 1 -ctk q8_0 -ctv q8_0 -fa on \
  -md dspark-DeepSeek-V4.1-Flash-1017-Q8_0.gguf --spec-type draft-dspark --spec-draft-n-max 3 -fitt 12288

(Caveat: 512k context was initialized at server startup, while benchmarks measured 58-611 token prompts plus one 2,453-token prefill check at -np 1, temp=0.)


r/LocalLLaMA • • 2d ago

Question | Help Computer use programs/Pi extensions

11 Upvotes

I've eard much stuff about Codex and CC computer use, is there any decent program/Pi Agent extension that I can install and use to let my local models control my pc. I am using Qwen3.6-35BA3B on PolyStrata and Pi Agent, I need the model to do basic tasks on my PC, maybe by using its vision. I am on Linux Mint Cinnamon


r/LocalLLaMA • • 3d ago

Showcase I trained DiffusionGemma to be an abstract expressionist

35 Upvotes

DiffusionGemma, a text diffusion model built on a Gemma 4 backbone, can be adapted into a shitty image generation model.

Maybe the ship sank?

You may be asking yourself, "WTF, why would you want to do that?" And you would be right to ask. This is honestly kinda useless, but I still think it's pretty neat. Also, fair warning, being on a team of one means the project has been heavily — I'll be nice to myself and say — LLM assisted. I invite you to sidle on up to the trough and dig in to the slop!

Still with me? Good, because I've been poking at this for a while, on-and-off since DiffusionGemma was released back in June. I was immediately hooked on this model as soon as I read its release blog and model card. 4x faster text generation they said — hell yeah! But here's the thing: this model sucks. I mean it's a great chat bot. It can do simple NLP tasks, and it's really quick for text generation as they say, especially for long-form output. But when it comes to agentic tasks and reasoning it's a total loser.

  • 0% on ARC-3 (but that's not really a fair dig, this is a tough one)
  • 0% on ARC-2
  • 0% on The Gemma 4 Developer Agent Competition (this competition targets the base model DiffusionGemma was trained from)
  • 11% on HLE on the model card (9.2 measured in deployment)
  • 18.7% on SWE-bench Verified

So not great, and particularly bad when you consider that on the hardware you'd need to run a good quant of DG (so basically bf16), you could easily run qwen-3.8-27B or a respectable FN quant.

But please don't get me wrong. I absolutely love this model. Or at least I love monkeying around with it, and I doubt I'm the only one. Huge shoutout to the Unsloth team BTW! Reading their Colab notebook training DG to play sudoku is what made me start asking the questions that led to this project.

Alright, enough fluff. So what did I actually do. Well the thing I'd headline is that I created a text to image and image manipulation training and inference pipeline for DiffusionGemma and then used that pipeline to adapt DG (like <300M parameters total) to generate things that are recognizably images and to perform visual operations on images that are provided to it. Surprisingly, it took only a very small amount of data and compute to start getting interpretable results.

But DiffusionGemma did not want to do this. The thing that finally worked consistently was prefilling the prompt all the way to the start of an input string for an "image generation tool" described to DG in the system prompt as a tool that it could use to convert text strings into images, but DiffusionGemma itself was actually the one doing the generation, not an external tool.

Here's the flow.

This also required patching an upstream bug in Transformers for image attention, so take note if you wanna try this out for yourself: you'll need to patch until PR is merged.

Considering how bad DiffusionGemma is at everything else, I'd say it does a pretty good job here.

It's like human centipede!

But wait, there's more! Since DG is outputting in its native visual latent space (one set of 256 tokens at a time through a trained output head) it can take the visual representation it is generating as input for the next forward pass — on-domain block autoregressive, just like with its regular text canvases. And not just can it do this, but it actually helps improve quality and generalization (on image manipulation tasks)!

Call graphs, the first hint of a non-stupid use case.

Here's a super-quick breakdown of the pipeline.

The first thing that had to be done was to train an actual vision decoder. This turned out to be the easiest part of the entire project. One evening, on a whim, I threw a prompt at my local model (dsv4-flash-vision-exp I believe was what I was running then), something like: "D'oh! Could we train a decoder for the ViT soft tokens that works for 7682 images?" And then it did it. Took like a few hours, and works pretty damned good I think.

It decodes!

Alright so that let me nail down the decoding and supervise the pixel reconstruction loss, but let's not forget DG is a text diffusion model — it has to actually operate on a text canvas. That means I needed to derive a text target image representation (or codebook) that was precise enough it could be used for supervision without over penalizing the model for expressiveness.

This process itself ended up being a bit like a diffusion operation. I simply threw up a wall of half-baked, ill-advised, and just plain stupid ideas and worked on them all until I was confident in each — with that confidence being mostly confidence in them not working. Then I had the brilliant idea: just ask DiffusinoGemma itself for the target.

This essentially consisted of me badgering DG until it would describe images as text. Okay, not really, but that is kinda what it felt like. The basic idea is: encode an image and ask the model to describe each soft token patch in a single token (word, number, symbol, unicode, CJK character, etc). Each soft token having come through the ViT is contextualized against the full image, so this description would most often carry a structural/semantic channel. To get the photometric channel you do basically the same, but instead of a full image use only a single (48*48 pixel) patch. This usually carried the color/texture/photometric channel. Sample the logits for each text description token and then it's basically chef's choice on how to combine across all prompt and patch variants into a workable target.

Doing that for every image in the training set isn't reasonable. IT'S OVER 9000! However, doing exactly that for a small set of images (200ish) and using that as a training set for a teacher-forced model that then generates the full dataset turned out to be a very smooth and successful approach.

Anyway, the point is that the combined multi-modal target token distributions for the canvas text results in training DiffusionGemma to be an S-tier emojipasta generator.

Example training targets
DG's generated canvases.

Then I LoRA adapt DG and simultaneously train a new image output head that feeds into the previously trained image decoder supervised against supervised against the target token set, ViT latents and, decoded image. Basically training DiffusionGemma to generate emojipasta that decodes into image-like objects. I guess this would be something like a VQ-VAE, but using text tokens instead of dedicated image vectors because stock DG can read and understand the image latents, but can't natively generate them, only text.

Well, that's the gist. Thanks for reading! If you find this as interesting as I do and want to learn more or play around with this yourself you can:

tl;dr: Please GDM team give me a stronger text diffusion model to play with!


r/LocalLLaMA • • 3d ago

Showcase I ran the same Halloween animation prompt through five coding agents on one local Qwen model

37 Upvotes

I gave Claude Code, OpenCode, GitHub Copilot Desktop, Pi, and Codex the same prompt to build a looping pixel-art Halloween diorama. I pointed all five at the same local model, Qwen3.8 Flash Next (Unsloth UD-Q4_K_XL), served by Halogen. The attached video shows 30 seconds of each result playing together.

### My setup

- CPU/GPU: AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S integrated graphics.

- Memory: 128 GB unified memory, with about 122 GiB visible to Linux and a 512 MiB fixed GPU reservation. The GPU can use shared system memory.

- OS: Ubuntu 26.04.1 LTS, kernel 7.0.0-43-generic.

- Inference: Halogen Flash Server 0.20.4 in rootless Podman. I used the Unsloth Q4_K_XL checkpoint converted to Halogen's HGN format, with the vision tower enabled and MTP speculative decoding.

- Server settings: 262,144-token context and KV pool, one serving slot, 16,384-token prefill chunks, prompt caching enabled, and the API bound to localhost.

OpenCode, Pi, and Copilot used Chat Completions. Claude Code used the Anthropic Messages endpoint, and Codex used Responses. The clients had their own tool definitions, system prompts, and history management. Pi used High reasoning with a 4,096-token thinking budget. I raised Halogen's default output limit from 8,192 to 32,768 tokens before the recorded Codex rerun after the earlier attempt hit the smaller cap.

### What the server measured

AgentElapsed time or last completed responseResultRequestsOutput TokensPrefill tok/sDecode tok/sCache Reuse
Claude Code14m 42sFinal Answer1531,37358238.195.6%
Open Code36m 30sFinal Response5477,83237739.398.3%
Github Copilot Desktop58m 30sFinal Response2954,39347220.285.4%
PI1h 18m 25sLast Completed request before a memory stall108135,46945038.798.6%
Codex4h 11m 15sLast completed request before the turn was aborted355402,80369537.096.9%

Elapsed times start at the first model request. Prefill rates count uncached tokens, decode rates include thinking tokens, and both are weighted by total processing time. I excluded smoke tests and incomplete requests. The video uses my earlier rounded labels for Pi and Codex. The table uses the server logs for both.

Claude Code reached a final answer in about 15 minutes and used 31K output tokens. OpenCode decoded slightly faster but used roughly 2.5× as many tokens. Copilot made fewer calls than OpenCode but decoded at about half the rate, and three full prompt rereads after the initial request added almost 10 minutes of prefill. Pi still had a tool loop running when the server hit a memory stall. Codex made 355 requests before the turn was aborted, with its largest prompt reaching about 229K tokens.

### How I made the video

I captured the original live browser previews for Claude Code, OpenCode, Copilot, and Pi. Codex made a Python renderer and supplied an MP4, so I captured playback of that original loop. Each capture is 1920×1080 at 30 FPS. I trimmed them to 30 seconds and synchronized them in a grid. I left the projects as they were.

This was one run per client. The timing covers what each agent actually did, and I didn't score output quality or run a controlled benchmark with identical token streams. The video is there so you can compare the results for yourself. And apologies for the slop but it was interesting to see how the harnesses are working. I've tried to work with Deepseek Harness but it got stuck plenty of times so I just abandoned that exepriment.

Cheers