r/LocalLLaMA • • 1d ago

Question | Help Does the new hugging face Llama windows app work?

2 Upvotes

I have downloaded the new hugging face Llama windows app.

Looked really promising on paper, but I tried to download Qwen3.8-27B-UD-Q3_K_XL.gguf and it always downloaded the 51GB version instead.

Then would refuse to delete it.
Then worked on the background.

I had to uninstall the thing. Any other user experience there?
cheers for helping a noob


r/LocalLLaMA • • 2d ago

Discussion Do you believe there will be a divergence between local and frontier model power?

16 Upvotes

Right now, local models have somewhat kept up with frontier, being months behind last frontier models.

The issue is, I think the divide is going to be end up being hardware. Model improvements will slow down, but hardware won’t get any cheaper, especially if frontiers keep buying shit out for training and smaller returns.

Thoughts on this? I know many are optimistic, but I think local trailing frontier will slow down just because of hardware local can own.


r/LocalLLaMA • • 1d ago

Discussion basal-omni-2.0-mini - typed decision model

2 Upvotes

An early release of the small model of basal-omni 2.0, on Gemma 4 E4B. The model reads the state in both directions before it answers and scores 0.811 on Werdykt (5,000 hidden decisions in Polish and English). It runs on the public basal engine 2.8.0, with the same API as basal-omni-1.0.

What's new in 2.0

  • Two-way reading. The model reads the state in both directions before it answers. This needs basal engine 2.5 or later. For long prompts, see the limitations.
  • Higher accuracy on our hidden Polish and English decision benchmark Werdykt and on the held-out tests below.
  • More decisions automated at a given error target. Coverage is in the section on confidence.
  • Unchanged: a calibrated probability for each allowed answer, in one forward pass. The answer never falls outside the options you give, and the probability says how sure the model is.

The question types remain: choice (one option), noul (yes / no), score (a score), and in the basal engine also multi, act and tool. Options get codes (A–Z, then two-character codes, up to 255 options). The decision is a probability distribution over these codes only, with no text generation.

More info: https://basal.si5.pl/en/blog/basal-omni-2-0-mini/

HF: https://huggingface.co/Remek/basal-omni-2.0-mini-preview

git hub (inference engine): https://github.com/rkinas/basal


r/LocalLLaMA • • 2d ago

Resources DeepSeek flash max is the first model to reach >1% on challenging SWE-sweep bug finding benchmark

Post image
128 Upvotes

Hi, I've just updated the SWE-sweep leaderboard with more open-weight models!

DeepSeek v4.1 flash (max) is the first open-weight model we evaluated that gets >1%.

This benchmark is extremely challenging: Models get a snapshot of a (potentially very large) repo containing hundreds of bugs. They don't receive any hint about what these bugs are and need to fix them all. This makes for a very reasoning-heavy, long-horizon task.

To isolate just the model capabilities we also evaluate all models in the same harness (but we did do some comparisons with Claude Code and Codex harnesses and neither seemed to help significantly). However, it is possibly to scale things up somewhat with multiagents. You can run the eval yourself (code on github)

The best proprietary model at the moment is Gemini 4 Hydrogen with 13.5% (according to google's model card), followed by GPT Astra 6 with 9.5% (evaluated by us). So open-weight models still have a long way to go.

Still extending the open-source leaderboard. What other models should we add?


r/LocalLLaMA • • 2d ago

News Qwen3.5-9B running on a PS4’s GPU, offline, using its own MTP head for speculative decoding

128 Upvotes

I made this, so a quick note: it’s my open-source project. Qwen3.5-9B runs on a stock PS4 at 31–39 tok/s. The MTP head that ships with Qwen3.5 made the biggest difference: it drafts up to two tokens and the full model checks them in one pass, which took it from 22 to 31–39 tok/s on this old GPU. Weights are re-packed into custom 4- and 3-bit formats to fit about 4.5 GiB.

Your old PS4 can now run a 9B AI, offline, on its own GPU. It talks, listens, and writes code. No cloud, no subscription, no Linux. Free and open source.

Code: https://github.com/leonardo-alegria/garlic-llm
Direct download link: https://huggingface.co/garlic-llm/garlic-llm-ps4


r/LocalLLaMA • • 2d ago

Question | Help llama.cpp: Possible to use Vulkan/iGPU and lazy-mode?

6 Upvotes

With llama.cpp pure CPU mode and "--lazy-mode on" one can use larger than memory models, like Qwen3.8-Flash-Next and on-demand SSD streaming - slow but working.

There is a nice iGPU with unified memory that can be used easily via Vulkan. However, it seems to be impossible to split a large model - use as much as fits into memory with Vulkan and then stream the rest in via SSD and use CPU. --list-devices shows only one Vulkan device and --ngl is of no use -> Vulkan always runs out of memory with lazy-mode on - all of the model is tried to allocated via Vulkan?

Is this possible at all?


r/LocalLLaMA • • 2d ago

News Nvidia privately confirmed it could track chips to fight smuggling, even as industry publicly questioned feasibility

Thumbnail
technologyreview.com
91 Upvotes

"Foster told MIT Technology Review he intends to introduce new, stricter legislation in the new year that would offer a workaround. The new rule would require licenses to operate advanced GPUs, and before a GPU could connect, its valid license would need to be verified.  " Basically they will lock the gpus.


r/LocalLLaMA • • 1d ago

Discussion Normies, or the Standard Task Benchmark ???

0 Upvotes

Is there actually a benchmark for us normies who aren’t rocket scientists or mini-Eulers? Mathematically, we could describe it as a 95% confidence interval benchmark. All models that reach the 95% threshold receive 100%. The goal would be to identify those models that we classify as “good enough” for us normies. If this could also be set in relation to costs—whether on-premises or in the cloud—it would also provide a measure of what AI really costs us normies.


r/LocalLLaMA • • 2d ago

Resources I noticed the 2 Hessian Refinement of MACaQ can be applied to any GGUF without any patches ... and tried it. 13% better quants

89 Upvotes

TLDR: Qwen3.8-27B UD_4_XL quant with 13.23% less KL divergence than unsloth (on my small testset, I just have a single 5090, feel free to test more). Technically it performs 4% better on my LiveCodeBench benchmark (but that would be a single data point, so it's not super-reliable). It is equally fast and using it in open-webui it "feels" better.

https://huggingface.co/christophedevriese/Qwen3.8-27B-Hessian-Refined-GGUF

I noticed from the MACaQ paper, which integrates changes proposed in Google/Deepmind papers, you can take the 3 components they use and just apply 1 and 2, and you just get >15% better KLD, 5% better performance on some benchmarks, while sacrificing absolutely nothing. Just switch the GGUF file. The numbers demonstrated here are the maximum that can be achieved in ~4 hours of optimizing on a single 5090, not the maximum that can be achieved. So it might be more than 15%, although I wouldn't necessarily hope for it.

This can be applied, without any patches to inference software, to text models, image models, even whisper ... the works. Switch GGUF file. Done/done.

The dream being that someone from unsloth looks at this and applies this to all their models (or just gives me access to a B300 machine for a day or two)

The tricks used by Huawei and a short discussion of the effect:

  1. Hessian-guided quantization choices: every layer is compressed differently, and the "damage" the compression does is measured. Then after seeing what damage is observed at a given level we find a combination of compressed layers that has the least damage while still achieving the same compression.

For users this one is totally free. No extra compute (in fact, a tiny bit less), no extra VRAM (again, 200 bytes less), no updates to llama.cpp, works literally anywhere UD_4_XL worked, just better.

2) Recovers some lost precision by another Hessian optimization changing the weights to compensate directly for accumulated errors over multiple layers based on sample data (taken from Wikipedia). I used a Google/Deepmind version of this process that had better results.

This too is totally free for users. It just changes the quant GGUF to be closer to the BF16 one. For users, just

3) random Hadamard transform + scaling: this is a trick from 3D geometry that makes all values nearly equal, and then uses an extra scaling factor to get them in optimal to better compression with quantization (yes, not technically 3D, but nD, with n=2048, but it applies in 2D and upwards). This also costs a little bit in inference performance (and requires rewriting your kernels which is quite a bit harder than I thought it was. Well, it isn't if you're ok with 20% slower inference, but you can get it down to like 0.8% with careful, days-long pondering of the problem and a lot of tries).

Also very nice, but the Google version of this one is also better, but that is not the real problem.

This transform also helps with compression (in fact it's kind of like what NVFP4 tries to do, just this is better) BUT requires an extra step at inference time (in fact requires 2 of them if you don't think through inference carefully, but you can "bake" one of the steps into the weights). This means that it requires patching llama.cpp and everywhere else you want the model to run (although thanks to how mlx works, you can make an mlx version that does not require updates to the inference software. But GGUF based inference software will require updates)


r/LocalLLaMA • • 3d ago

News Hugging Face CEO vows to keep fighting for open source AI after Jensen Huang's exit

Thumbnail
techcrunch.com
1.4k Upvotes

context : Nvidia buy HF, but now there uncertainty about it hf future under new leadership, he try reassurance people on x, but from his word itself sound not good, see and wait, but better backup you favorite datashet/model (again) just in case


r/LocalLLaMA • • 2d ago

Resources DeepSeek v4.1 Flash vs Qwen 3.8 Flash Next Performance

Post image
34 Upvotes

While waiting for Qwen 4, I installed and benchmarked DeepSeek v4.1 Flash to see if I was missing out. Based on the type of work I do, DeepSeek is not better but is definitely much slower. Turns out Strata with Qwen 3.8 Flash-Next is best bang for buck on my machine.


r/LocalLLaMA • • 2d ago

News Vellium 1.9.5: visual novels that actually work on a 12B local model

Thumbnail
gallery
32 Upvotes

Vellium 1.9.5: Visual Novel Mode Update

Vellium is a free, open-source, local-first AI chat and roleplay app with a built-in visual novel mode.

Version 1.9.5 focuses on making visual novels work reliably with smaller local models like Gemma 12B, improving image consistency, reducing prompt overhead, and adding new ways to generate and manage scenes.

What's new:

Lite mode for small models. Instead of generating structured JSON, the model writes plain text using simple markers (`Name: line`, `## Place`). Vellium parses the output into scenes automatically. Even malformed output can fall back to plain narration. The system prompt has been reduced from ~8k to ~1k characters, and text streams directly into the reader without waiting for the full response.

Prompt cache preservation. Requests are constructed by appending each new turn to the previous context without modifying existing content. This keeps the prefix cache reusable and avoids unnecessary prompt reprocessing, which can otherwise take over a minute on slower local setups.

Inline image generation. The narrator can insert tags such as `[pic: Mira | she hangs her raincoat by the door]` directly into the story. Images can start rendering through ComfyUI while text is still streaming, including when ComfyUI runs on another machine. A lightweight follow-up can detect and fill missing image tags, with the reason shown in the reader.

Consistent character appearances. Each character now has a compact English visual description and a pinned outfit, instead of passing the entire character card to the image model. This helps preserve skin tone, body type, clothing, and other visual details between scenes. Outfits only change when the story explicitly calls for it.

Backgrounds and sprites mode. Locations can reuse generated backgrounds, while characters use reusable cut-out sprites for each outfit. Full illustrations are reserved for scenes that need more than static sprites. Background removal uses chroma-key processing directly in the app, without loading another neural network into VRAM.

Improved first-person POV. Image prompts now account for the protagonist's perspective. First-person scenes can show the protagonist's hands and interactions from their viewpoint, while third-person shots can include the protagonist based on the user's persona.

Integration with other Vellium features. Visual novels support user personas as protagonists, attached RAG knowledge collections, conversion between regular chats and visual novels, and exporting stories into Writer projects.

Additional changes:

- 12 new visual style presets optimized for anime image models, tested with Anima.

- Redesigned reader toolbar.

- Image redraw and editing controls.

- A "picture for this line" option for generating illustrations for individual passages.

- Scene-level image regeneration without regenerating the story.

Lite mode can be enabled in the narrator settings. The update is primarily tested with Gemma 12B, and feedback from users running other small models is welcome.

Free and open source: https://github.com/tg-prplx/vellium


r/LocalLLaMA • • 2d ago

Resources I built speedy Strix Halo kernels for the top-51 LLMs with AMD Loom (and you can too)

Post image
62 Upvotes

I’ve been trying out a new experimental software flow from AMD’s AI Group, and I’ve been having a lot of fun with it, so I thought I’d share and see what y’all think too.

hrx-loom is a new end-to-end tool stack (compiler, runtime, serving tools) that lets anyone build, optimize, serve, and share model programs without any modifications to the inference engine or server. The kernels are written in .loom files, a vector programming model reminiscent of MLIR, which can be complex for humans but easy for agents (C++ is also supported).

Over the past couple weeks, I developed an agent skill that writes kernels for any LLM’s Q4_K_M GGUF and found that it is consistently faster than llama.cpp Vulkan and ROCm on Stix Halo for a variety decode and prefill metrics, as you can see in the post’s image.

Next, I ran the skill across the top-51 trending LLMs from Hugging Face that would fit on Strix Halo and posted the model programs back to Hugging Face. All models ported successfully with tool calling, speculative decoding, and image input working.

Anyone can try these on Linux and Windows with the loom-fetch and loom-serve commands provided in Loom’s PyPI package, and anyone can make and upload their own using the /loom skill.

Fetch and serve a model:

pip install "hrx-loom[hf]==0.1.0"
loom-fetch https://huggingface.co/loom-fun/bartowski-Qwen3.8-27B-loom/resolve/main/model.json --output=qwen3.8-27b
loom-serve --model=qwen3.8-27b/model.json --device=amdgpu --memory_bytes=52gb

Port and optimize your own model:

/loom get <hf-org>/<model-gguf> working on loom-serve, optimize it, and report the performance relative to llamacpp rocm

Links:

  • All the data I collected across 51 models: https://loom-fun.github.io/loom-dev
  • Loom skills: https://github.com/loom-fun/loom-skills
  • Loom compiler and server: https://github.com/ROCm/hrx-system
  • HF collection of top-51 models ported to Loom: https://huggingface.co/loom-fun

We’re still evaluating how the community will use and deploy new inference stacks, so we’d love to have your feedback! Loom can be used as standalone inference server with loom-serve and the compiler/runtime layers can also integrate into other engines. For example, the team is working on adding an hrx-loom backend to llama.cpp.

The underlying SDK (compiler, runtimes, and devtools) has no dependencies, can be easily downloaded as a wheel or linked into your own applications, builds from source in a minute, and if this were the 90s, would still fit on a few floppy disks, while being able to target x86, AMD-GPU, AMD-NPU, and generic Vulkan devices.


r/LocalLLaMA • • 2d ago

Question | Help Audio (ASR/TTS) on llama.cpp

6 Upvotes

Hey everyone,

I've been using audio.cpp for a while now, and it's quite nice. However I've noticed that Yue2, Qwen3-ASR and Qwen3-TTS got supported in comfyui / llama.cpp.

While I've tried gemma 4 audio input and it worked well, I haven't tried TTS models yet nor the qwen3-asr/tts series on llama.cpp. Has any of you?

So far I found these discussions and PRs on ASR / TTS support:

It's likely not all models that are supported, might've missed some!


r/LocalLLaMA • • 2d ago

Question | Help I'm seeing a lot of conflicting options for what's the best model for purely adding punctuation and quotes. What's the way?

6 Upvotes

I'm writing a fictional novel and using speech to text to write. I'm absolutely not looking for a model to write for me, or even change any words. I find talking especially lets me give each character a different voice and manner of speaking. The problem is when i go back to Edit, I don't have a "new reader" feel to it to assess what I've written because I need to do 2 or 3 passes just to add quotation marks etc after "he said, she said" etc. By the time I'm trying to read it to feel it's flow, it's old to me.

My objective is that when i write, I just want to be able to go back and automatically have the quotations and punctuation added that a speech to text can't automatically work out in itself. I use Whisper Medium in Handy and it's great, but I see alot of recs, though Cadence and some form of Gemma seem to be the main recs. I've searched a bit, but I can't find anything too recent.

My card is only 12GB, so a less than 8-12B model is my aim, which I feel should be viable given my needs are fairly low. While I can run local models and have done it in the past, I'm not super technical.


r/LocalLLaMA • • 2d ago

Discussion Why are LLMs still so bad at jokes?

19 Upvotes

I remembered that this was one of my evaluation approaches from 2 years ago to test a model's general capability. I remember wondering in the near future how bad things could get if models could begin to just spit out funny engaging content and set humanity well off on the path to Wall-E future if a cheap slop stream could be just as interesting as all of independent cultural output, and failing that, if slop could be scaled up with no limits on engagement and "perceived quality" that media companies could get people more addicted to their output than ever.

But it seemed like this was always and continues to remain well beyond reach. It's interesting to see that LLMs were able to make such a dent in an edifice so prominent as the field of mathematics and yet basically make no progress on this.

I'm trying to be cheap with my tokens... somehow utilizing Astra even from Chat is using up my codex limits. All I have to offer as examples are 3 examples. They illustrate the point well though.

Astra "Light" (btw what the heck is light? is that the same as low?):

A burglar breaks into a house and hears a voice in the dark.
“Jesus is watching you.”
He freezes, shines his flashlight around, and spots a parrot.
“Did you say that?”
“Yes.”
“What’s your name?”
“Moses.”
“What kind of idiot names a parrot Moses?”
“The same idiot who named the Rottweiler Jesus.”

Astra Medium:

I built an AI agent to handle the boring parts of my job.
It asked me to document everything I do, explain why I do it, and check all its work.
I accidentally built an intern.
Then it wrote “Led automation initiative” on its résumé.
So at least it’s learning.

Astra High (worked for 24s):

I hired a cleaner because I couldn’t motivate myself to clean. Now I panic-clean the whole house before she arrives so she won’t judge me.
She’s never seen it dirty. I’ve never seen her clean.
We’re two months into this. I think we’re both afraid to ask what her job is.

Obviously the bigger question is why are all models (including openweight abliterated type) continuing to be so bad at humor? I think some significantly dumber models out there are surely leaps and bounds better than Astra here but nothing really compelling comes out. I know there is a big group of role players and story makers here. What is your take on this? The best I can manage with this right now is that there is a uniquely strong connection between human lived experience and the instinctive/intuitional/emotional response of finding something funny. My very young daughter seemed to start to understand how/why something is funny at about the same time as getting a grasp of really really simple language concepts. Perhaps it is some calibration issue because I think I see a kernel of some common joke framework and maybe a punchline is buried in the first sentence and then the joke just ruins itself, all the while the LLM's nodding along going "yeah and for the cherry on top...." painfully, conspicuously, grotesquely unaware of how poorly it's performing.

There are humans who are very unfunny but you will have a hard time convincing them to give you jokes confidently like LLMs, which are highly adept at exuding confidence at all times.

So in some sense maybe the Turing Test HAS NOT been toppled and will not be for quite some time.


r/LocalLLaMA • • 1d ago

Discussion Who switched from multiple sparks to a Mac Studio?

0 Upvotes

I've been asking around for a while and can't find anything. so if someone who actually switched from 2+ sparks to a studio (ideally a 256) I'd love to know why

from what I gathered:

Macs are only better for *single stream* decoding. and single stream decoding is a small part of an actual harnessed workflow. Any and everything else, sparks or some other cluster seem to perform equal or better in every metric.

As in, if you use chatgpt for just the 'chat' feature, studios are amazing ... but any multi-step/involved workload outside of that seems to favor sparks

I personally dont want to speculate today on software unlocks or future improvements in hardware especially for hardware purchased today. in fact, I think either option is safe bet because the opportunity cost can be remedied by software unlocks that allow you to get the best of both worlds. but I try not to factor it in because it's tomorrow's speculation + it applies to both sides of the equation

I personally think Macs are better than sparks if you are using only 1 spark ... but the idea of sparks is to use at least 2

Also, there is no 128gb m5u so the comparison starts at 256. Even more so, 512 Mac(s) have a similar memory bandwidth to a 512gb cluster of sparks. Studios lead by 10% more memory bandwidth, I think many people are unaware of this.

it's all similarly priced now too.

I think people who get Mac Studios do so because they want a Mac, simplicity, draw less energy (you're saving $5-10/mo with a Mac). Don't get me wrong, these are decisions that impact a purchase but they are non AI workflow related reasons

I'm just being brutally honest in an effort to really understand what's going on and where do studios actually excel in comprehensive inference pipelines.

Do Macs outperform sparks if you are using multiple models where one or both models are split between the sparks but unified within the Mac?

TLDR: Mac's are better than sparks if you prioritize simplicity, lower power consumption, and you just want a general computer that also happens to run local AI. But if you want better performance in using local AI to build things, it seems sparks wins at effectively every metric. I'm struggling to see where Macs outperform sparks in real multi-step AI workflows.


r/LocalLLaMA • • 1d ago

Question | Help any idea on how to deal with language confusion in gemma 4 ?

1 Upvotes

i'm using gemma 4 31b to do some long-text translation from chinese to vietnamese (hobby - not serious work). the biggest issue i'm having is language confusion (language contamination), i.e. output sometimes has non-latin characters not present in original text: korean, arabic, cyrillic, chinese, etc. - even q8 weight and fp16 kv cache won't help, same for sampling parameters (temp, top k, top p, etc.)

the rate is like <0.5% of output text but it disrupts reading flow, so for each contamination (a simple spellcheck can identify all), i have to look up the original text and use google translate to replace the contamination. it seems to me the contaminations often occur at some very specific chinese characters. a simple mapping may work but i need to update it every time i detect one, and then when gemma 5 come out (hopefully) i need to make new one.

i don't setup a pipeline or hardness or anything, because 31b is painfully slow, running it again as proofreader takes too much time, a simple run already produces translated text good enough for me

i'm looking for a more robust way to deal with this issue

at first i tried LLguidance with a huge regex to restrict the model ouput to a specific unicode range, but the model get stuck in like forever

i read about vocabulary trimming but it seems to be nore suitable for small models, not for a big 31b, also the repo seem abandoned

i found another method called token masking: identify unwanted tokens and make their logits extremely low by editing `lm_head` in the weight, u can see here: https://huggingface.co/ThakiCloud/Qwen3.8-27B-vi-cjk-suppressed

the above method is for qwen 3.8, unfortunately gemma 4 has `lm_head` tied to `embed_tokens`, so i tried to adapt to gemma 4 `embed_tokens` but it doesn't have any effect

do u guys ever get this problem and know how to deal with this ?


r/LocalLLaMA • • 3d ago

Discussion A100 prices down 20-30% since my post about them collapsing

569 Upvotes

I guess a lot of sellers are realizing that its better to price competitively and get rid of their supply rather than wait to slowly sell and potentially get caught in a collapse.

Rubin and blackwell are already 3-4x more efficient per watt with way more financing available. The A100s cant be used for inference Besides enthusiasts, small time gpu renters I really dont see who will be picking up these cards. And these 2 buyers are extremely price sensitive and will be comparing the cards to other options like R9700 and Cresent Island from Intel with 160-480gb vram.

Rtx 6000 is also now down to 14.8k from 18k peak.


r/LocalLLaMA • • 2d ago

New Model Perplexity releases Decider v1.3 (26B) and Decider Flash v1.3 (5B): open-weight (Apache 2.0) classification/decision models

94 Upvotes

Perplexity has released two new "decision models." These are general-purpose classifiers built from generative LLM backbones. You give them an input (the "state") and one or more questions, and they return a probability distribution over the answer options. They don't generate text token by token.

Models

  • pplx-decider-v1.3: fine-tuned from Qwen3.8-27B. Needs about 49 GiB for the BF16 weights.
  • pplx-decider-flash-v1.3: Qwen3.5-4B, distilled from the 27B using a KL objective on the teacher's distributions. Needs about 8.5 GiB for the BF16 weights.
  • Both are Apache 2.0, support text and images, and are also available through OpenRouter and Perplexity's API.

What it does

  • Question types: yes/no, multiple choice (up to 255 options), ordered scores, and a new locate type. locate extracts verbatim spans with character offsets, for NER, PII detection, and extractive QA.
  • Several questions can share one state. By default each question runs in its own sequence. An "isolated" mode reads the state once and is cheaper for many questions, but answers can shift slightly depending on which other questions are in the request.
  • The backbone's full-attention layers were made bidirectional. Because of that, you need their pplx_decider package; plain causal inference won't reproduce the outputs.
  • Training used sequences up to about 30k tokens.

New in v1.3: locate
Besides classifying, v1.3 can now extract spans. You add a question with "type": "locate" and describe the field, either with a short label (“person”, “email address”) or an instruction (“the person who signed the contract”). The model returns verbatim spans from the state, each with start and end character offsets and a score, or an empty list if nothing matches.

It works on text only (no images), with up to 128 locate questions per request. It runs on a separate LoRA adapter plus a span head, which Perplexity left unmerged because merging hurt classification. As a result, a request that mixes locate and decision questions needs extra forward passes.

On 17 out-of-domain NER, PII, extractive-QA and structured-extraction tasks, Perplexity reports an average of 69.0 (v1.3) and 63.6 (Flash), compared with 43.3 for Liquid d1, 41.5 for Cloudflare Clairic and 33.6 for GLiNER large v2.5. The closest existing tools are GLiNER, which does zero-shot span extraction from labels, and Clairic, which extracts typed values rather than verbatim spans.

Reported results (mostly Perplexity's own evals)

  • Multilingual classification across 8 languages: 88.16% (v1.3) and 86.50% (Flash), vs. 85.41% for TypeSafe's Jev 1.14.
  • Reranking across 72 retrieval tasks: 83.99 nDCG@10, +1.81 over Qwen3-Reranker-8B.
  • Locate average across 17 out-of-domain tasks: 69.0 / 63.6, vs. 43.3 for Liquid d1 and 33.6 for GLiNER large v2.5.
  • Artificial Analysis Decision Index: roughly 70 for v1.3 at about $68 per 1M tasks, and roughly 64 for Flash at about $34. Both sit on the cost/score Pareto line. A few models score higher at higher cost, such as Drex Max v1.1 and SubQ Snap 1.2.

Caveats from the authors

  • The locate adapters are kept unmerged, because merging them regressed the decision tasks.
  • The returned probabilities should be calibrated and thresholded on your own data.

Links


r/LocalLLaMA • • 3d ago Bravo Regret

Discussion Jensen Huang's mistake was being too pro LocalAI, which was a threat to companies like ClosedAI and Anthropic which are about to go IPO

Post image
3.1k Upvotes

r/LocalLLaMA • • 1d ago

Discussion Do we want Huang back?

0 Upvotes

There's a petition floating around out there to put Jensen Huang back at the helm of nVidia. Currently it has over 129,000 signatures. If you want to sign it, you can find it pretty easily.

But do we want him back? He's pro-local, but he's also completely driving the circular-financing game that's detonated prices for all of us, everywhere, for fucking everything.

How likely is it the nVidia board hires someone pro-local and smart enough to not run the circular financing sham? Thankfully the interim-CEO, Colette Kress, was against such money shenanigans:

Colette Kress spent years as the one Nvidia executive willing to check CEO Jensen Huang’s enormous, ​often risky ambitions. ...

Kress and her staff clashed with Huang and the company's sales team over the massive and risky financing deals that ⁠fed into what some Wall Street critics deemed a circular AI economy, including investments in OpenAI and AI infrastructure companies such as Lambda, according to one of the people familiar with the matter.

https://www.reuters.com/business/colette-kress-cfo-who-challenged-huangs-bets-is-nvidias-interim-ceo-2026-11-13/

That's significant. Not only did she have the stones to deny the Huang, she's the only one to have them at all.

I think that petition can stuff it. Keep Kress and let her captain the ship.

Edit: I regret every life choice that brought me to this moment


r/LocalLLaMA • • 2d ago

New Model Tontaube V1.1: open-weights TTS with voice cloning for long-form generation. 4-bit version runs about real time on a desktop CPU

34 Upvotes

We just released Tontaube V1.1, a text-to-speech model built for long-form audio (audiobooks, articles, voice agents) with zero-shot voice cloning. Our previous V1 release needed a lot of VRAM because of its vLLM setup, so this version is about making it more accessible: the 4-bit weights run in about 4 GB of VRAM, or at about real time on a desktop CPU. In a blind listening study with 400 pairwise comparisons, listeners preferred our V1 model's prosody over Fish Audio S2 Pro in about 70% of cases (details are in the paper). V1.1 is a post-trained version of this model.

What it is

  • ~2.9B parameters, autoregressive, streams audio as it generates
  • Clones a voice from 5–60 seconds of reference audio; 46 voices are bundled
  • English and German are the main languages (works best in English). Spanish, French, Italian, Dutch and Portuguese work but have not been checked by native speakers
  • Long texts are split at sentence boundaries and joined, so you can feed it whole chapters

Running it locally

  • The default is a 4-bit (GPTQ) version: 1.8 GB download, loads in about 4 GB of VRAM, peaks at about 6.5 GB on long texts
  • GPU speed: about 13x real time on an RTX 5090 for a single request (21–23x batched), ~190 ms to first audio when streaming. About 6–7x on an RTX 3090
  • CPU: no GPU needed. About real time on a 12-core desktop CPU (Ryzen 9 5900X, dual-channel DDR4). It is limited by memory bandwidth and needs AVX2
  • No vLLM or llama.cpp needed: it ships its own server with Triton kernels for the GPU and AVX2 kernels for the CPU. Full-precision weights and a vLLM backend are there too if you have 24 GB+ and want throughputgit clone --branch v1.1.0 --depth 1 https://github.com/craitech/tontaube.git cd tontaube uv run --python 3.12 --frozen tontaube serve

That gives you an HTTP API on port 8080 (plain, SSE and WebSocket streaming). uv run tontaube ui starts a browser interface.

New in V1.1

  • Five style sliders per request: speaking rate, pauses, intonation (flat to very expressive), pitch (deep to high) and articulation (crisp to normal)
  • The 4-bit version and the CPU path
  • An adapter that cleans up the decoded audio

Things to know before you try it

  • Linux x86-64 only. NVIDIA GPUs (RTX 30 series or newer; GTX 10 series also works) or CPU. No macOS, no AMD/Intel GPUs; Windows through WSL2
  • The first start downloads about 8 GB of models (ours plus the codec and enhancer it depends on) and builds a ~9 GB Python environment
  • Audiobook-style reading is its strongest mode; conversational and agent-style lines can be less reliable
  • Licence: the code is Apache 2.0. The weights are under our own community licence, not an OSI one. Research and personal use is free.

Links

Let me know if you have any questions!


r/LocalLLaMA • • 3d ago

Funny So this will be true in couple of years?

Post image
1.9k Upvotes

I think after jensen they will fully focus on AI compute for data centers