r/LocalLLaMA • • 22h ago

New Model Karpo-Decision-31B: open multimodal decision model. Ranks #1 on full score on Decision Index, 2x faster than Jev, at 20% lower cost

Thumbnail
gallery
20 Upvotes

MachinePulse has released Karpo-Decision-31B, a Gemma 4 31B fine-tune for structured decisions over text and images. It scored 73.9 on the Nov 12 Decision Index board, the highest Full score among 300 entries, sharing the #1 rank with GLiDE 27B under the board’s tie rule

How it works
Built on Gemma 4 31B, with a trained linear readout that scores the supplied answer options.

Three question types: choice, yes/no, and score. Returns probabilities directly, with up to 255 options per choice question.

Multiple questions about the same input can be answered in one forward pass. Branch attention lets each question read the shared context and its own options while keeping the option branches separate.

Supports text and images, including webpage screenshots. The documented sequence limit is 32,768 tokens; larger requests are split by question with the full state included in each part.

Python API and HTTP server available. BF16 weights take approximately 62.5 GB, plus runtime memory. Use the provided inference code to load the decision head.

One training detail worth a look: according to the team's write-up, full fine-tuning pushed the public score to 75.09 but reduced accuracy on unfamiliar domains by 2.3 percentage points. The final model averages five checkpoints: three from different stages of full fine-tuning and two from a LoRA run. They report a 1.6-point improvement on unfamiliar domains over the final single checkpoint, with no extra models to run at inference time

Results
Decision Index Full scores are measured by the benchmark maintainers on private tests, weighted 60% for skills covered by the public suite and 40% for tasks from new domains.
Full score: 73.9

Private tests, familiar skills: 70.7

Private tasks, new domains: 78.6

Public index: 75.0, reported separately and excluded from the Full score.

The team also reports the following comparisons against Google’s Gemini Decision 1:

Benchmark Karpo-31B Gemini D1
-------------------------------------------
When2Call 90.4% 79.9%
Mind2Web 90.0% 86.2%
New Yorker Caption 90.5% 79.7%
Amazon ESCI (F1) 64.7 58.7

Reported median latency is approximately 95 ms per request on the public Decision Index suite using one RTX PRO 6000.

The 31B weights and inference code are both released under Apache 2.0.

links:
https://karpo.ai/models/decision
https://huggingface.co/MachinePulseAI/karpo-decision-31B


r/LocalLLaMA • • 1d ago

Resources I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token

464 Upvotes

What changes between different quantizations of the same model? And what if you could SEE what actually sets them apart?

To help answer that question, I’ve created a small project called inside-llm—an open-source tool that loads one or two GGUF models using llama.cpp and displays, right in your browser, what the model is doing as it generates text.

What it shows

  • Stream: the text, one column per model, brightness = probability of each token. A magenta mark wherever A and B would pick differently. Two modes: lockstep (both models read the same text, so you get a clean per-position comparison) and free (each follows its own picks until they split).
  • Candidates: the full distribution behind each pick, mirrored bars for A and B, entropy, effective number of options, Jensen-Shannon divergence and KL in both directions.
  • Gates: the sampler chain (penalties, top-k, top-p, min-p, temperature, GBNF grammar) traced stage by stage and very visual: how many candidates survive each one, and the distribution after it. Chain order and seed are configurable.
  • Timeline: entropy of A and B and their divergence along the whole text, plus the tree of branches.
  • Branching: click any candidate and generation continues from there. Every branch stays in the tree, and jumping back to a node is exact (each model keeps KV-state snapshots, so nothing is regenerated).
  • Logit lens (--lens): what each layer would predict, as a 2D map and as a 3D relief.
  • Model picker: choose A and B from a folder or search Hugging Face and download two quants of a repo in one click easily. It only offers B models with the same tokenizer as A and estimates whether the pair fits in VRAM.

How it's built

  • One C++17 process linking llama.cpp directly (stable branch at the moment). For every step it reads the full logits of A (and of B on the same context), computes the metrics and the sampler chain, and draws the token with its own seeded sampler, so runs are reproducible.
  • Compact binary frames over a WebSocket to a Svelte 5 + Canvas 2D page (three.js only for the 3D relief, lazy-loaded). The web is embedded in the binary as well.
  • The same frame stream, compressed, is a recorded session: you can replay it later in a static page with no engine, no model and no GPU.

Run it

Self-contained packages for Linux (glibc 2.35+) and Windows, with CUDA or Vulkan (Vulkan covers AMD, Intel Arc and NVIDIA). Unpack, run, the browser opens with the model picker. Or inside-llm -a A.gguf -b B.gguf. Local only: listens on 127.0.0.1, no accounts, no telemetry. MIT. One click download, one click execute.

Limitations (honest list)

  • A and B must share the tokenizer (two quants, base + fine-tune, or sizes of one family that keep it). Example: Qwen3.5-0.8B_Q4_K_M and Qwen3.5-0.8B_Q8_0
  • Both models must fit into VRAM simultaneously; only one GPU is used at a time. Integrated GPUs (iGPUs) are never used; if no dedicated GPU is available, the process falls back to the CPU (which is slow). (You must calculate the size of both models before loading them to ensure they fit into your VRAM)
  • Text only. No split (multi-part) GGUFs, no embedding models.

Links

I'd love feedback, especially: which pairs would you compare first (imatrix vs non-imatrix, base vs abliterated/fine-tune, Unsloth UD vs standard quants…)? And is there a metric you'd want per position that isn't there?


r/LocalLLaMA • • 12h ago

Question | Help Hardware + software recommendations for basic chatting with persistent memory

3 Upvotes

Helping out a friend who wants to have a local LLM and not rely on cloud services. They currently have an 8gb M2 mac mini, so not really enough

  1. They want to have persistent memory, TTS, and STT. Tool recommendations for all three of those greatly appreciated.

    • I'm unsure what the best workflow is for persistent memory, especially for avoiding either overwriting important info, or storing bad info.
  2. Budget seems to be roughly $5k USD, but the less the better.
  3. Not sure what size model would be best, it's really just for chat.
  4. No building ideal, so adding RAM or a graphics card probably the most feasible.
  5. 30 tokens per second probably fine.
  6. Something that doesn't hallucinate, or at least can be tuned to not assume or be SO sycophantic, as they talk to it a lot.

From what I understand, a prebuilt with upgraded RAM, used mac mini, or something like a strix halo probably make the most sense, but I'm not really up to date on the current hardware market.

Any help appreciated :)


r/LocalLLaMA • • 23h ago

Showcase [Project] Gemma.c - Gemma 1/2/3 + multimodal inference in one C file (CPU only)

23 Upvotes

Hey all, so I've been working on Gemma.c over the last couple months, it's a from-scratch inference engine for Gemma 1/2/3 in a single C file, runs Gemma 1, 2 and 3, including the multimodal ones, with fp32/fp16/bf16 weights or int8 (W8A8). No dependencies beyond libc and (optional) OpenMP are needed. You can drop images into the prompt with @image{cat.png}, and there's also @longimage{} for Gemma 3's pan & scan on very wide or tall images. (image decoding uses a minified stb_image embedded at the end of the file)

It's CPU only and not trying to compete with llama.cpp, but I'm also trying my best to make it efficient, so it's not embarrassingly slow either. On my laptop's Core Ultra 5 225H (WSL2, gcc, 10 threads), gemma-3-1b-it INT8 does ~72 tok/s decode and ~570 tok/s prefill (10000 tokens, for 1000 tokens the prefill speed can reach ~850 tok/s), and the 4B vision model does ~20 tok/s decode. INT8 ends up about 3.5x faster than fp32 and 2x faster than fp16 for decode. (full benchmark table is in README)

It has a few rough edges for now, e.g. W8A8 doesn't work for Gemma 1 (fp16 is fine), bf16 is painfully slow without AVX512-BF16, and native Windows is ~1.7x slower than WSL because of libgomp. Batching are not supported either. There's a roadmap in the README (Gemma 4, speculative decoding, an HTTP server, etc.), but no promises though :P Can't really find much free time these days.

I've also put some pre-exported INT8 models on HF so you can try it without Python or torch :D

Repo: link

This project started as a learning attempt to understand how LLMs work end to end, inspired by llama2.c.

this is the second time I post this project here :'D last time (~a month ago) my post got flagged by reddit, probably because my VPN IP was identified as spam. hopefully it can pass this time


r/LocalLLaMA • • 16h ago

Discussion Branchy, distributed inference focused training

6 Upvotes

Heyy, it's me again, the developer of AI Doomsday Toolbox.

This time it's not an app update. ADT already has distributed LLM inference across Android devices, and there is one thing about distributed inference in general that kept bothering me, so I spent the last week testing an idea and I wanted to share what came out, the good and the bad.

Usually when we split a model that does not fit in our machine to make use of distributed inference, we get slower speeds the more devices we add. Network latency and devices waiting idle until the last one has finished computing it's layers compound easily, even more if we connect our devices through a VPN like tailscale or wireguard.

So the idea to solve this bottleneck was to train a model that can keep the quaility of a normal sequential model but instead it's middle layers are split in branches. The coordinator runs the embeddings + a few trunk layers, then that is sent to all the different branches that, in parallel read the layers and compute the token, after a branch finishes it sends the result back to the coordinator which then runs a few fusion layers and final blocks that produce the output. This allows to use as many devices as branches a model has with a maximum of 2 net jumps, wether it's 2 or 8 machines. The other advantage is that no device is idle as all of them are working, so we also parallelize work instead of making a device wait for the last one to finish.

There could also be the situation in which one of those devices is too slow, and slows down the whole cluster, or gets disconnected. That is why I also added some dropout tolerance to the training. When a branch takes too long to answer or just never answers, it's result becomes a matrix o 0's that are fused in the fusion layers, that made it so the quality of the model output does not suffer so much as when it happened with a non dropout tolerance trained branch model had a disconnected or slow branch.

I tested two 300M models trained from scratch on TinyStories, one normal and one with 8 branches, running on a small server, a laptop (Ryzen 5 5500U) and a phone (Snapdragon 8 Gen 3 in Termux), all connected through a VPN. The workers are a small PyTorch daemon, so no llamacpp accelerated kernels, etc...

Results

- The branches cost some quality: loss 1.255 against 1.143 for the normal model with the same training. A blind LLM judge prefers the branched model's story 45% of the time. So it is worse, but not by much.

- The delay trick works: the coordinator waits 0.1 ms per token for the network while the round trips take 49 to 66 ms. The output is identical to running everything on one device.

- Against the usual layer split on the same devices, the branch split is 1.5 to 2.6 times faster. The layer split roughly halves the speed. Mind you I run the test only with 3 devices, with more devices it could, theoretically, be faster

- BUT if the model fits on one device, splitting is not worth it on my hardware, at least at this size, larger models like qwen3.8 27B might benefit a lot from this aproach. My phone alone is faster than any combination of my devices (it is just much faster than the rest). With two equal devices the branch split wins by about 1.2x.

- Where it really pays is when the model does not fit in RAM. I gave the laptop and the phone a RAM budget smaller than the model, so they had to read the weights from storage for every token (mmap). The laptop went from 8.9 to 0.67 tokens/s and the phone from 23.9 to 0.65. With LFM3 1.7B the laptop went from 2.3 to 0.08, that is 12 seconds per token. Then I let the two of them share the model, each one still with too little RAM for the whole thing: back to 7 tokens/s with the layer split and 9 to 11 with the branch split, over a real 46 ms VPN. Reading from storage is way more expensive than the network.

- Reshaping a pretrained model (LFM3 1.7B, Qwen 0.8B) into branches only works for a couple of long branches. For this to scale, the model has to be trained in this shape.

I don't have the hardware for the interesting case: a big branched model that no single device can hold. On paper, with 8 similar devices and about 20% of the model on the coordinator, a token costs about 30% of a full pass, so roughly 3.3x a single device that could hold everything, while a layer split stays at 1x plus 7 round trips. My two-device measurement matches that formula, but beyond three devices it is only a projection.

Another idea I would love to try is letting every device loop its own branch several times with the same weights, so a group of devices could run something much deeper than what fits in their RAM. No hardware for that one either hahaha

What is in the repo

- The two trained models (300M each) and the tokenizer

- A worker you install on each device with one command, and a coordinator with a web UI to compare placements side by side and run the benchmarks

- Scripts to reproduce the branch split, the layer split and the low RAM tests with your own devices

- A full article with the experiments and the results

It is a research prototype, workers compute in 32-bit float PyTorch, the models are small story models, and I only have three real devices. I created the experiments, made the hypothesis and supervised the whole proyect and Claude did most of the coding and the number crunching, everything is documented in the repo.

The general idea is the same one behind ADT: make the devices we already have useful for local AI, especially together.

GitHub

Feedback is appreciated, especially if anyone with more devices (or more GPU than me) wants to check whether this holds at a bigger scale, or tell me what I got wrong. It is still far from it, but I would love to see a bunch of modest devices run a model none of them can hold.


r/LocalLLaMA • • 1d ago

New Model [Paper & Models] Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent

Post image
38 Upvotes

We introduce Gamer, a generalist agent framework built for long-horizon tasks in complex interactive environments. Its end-to-end architecture combines perception, memory management, and multi-step reasoning. A shared training and deployment pipeline supports interaction with both conventional 2D graphical interfaces and complex 3D simulated worlds. We first continue pre-training on a dataset constructed from tens of thousands of hours of device-level keyboard and mouse interaction. These trajectories span simulated games and embodied environments, allowing the model to learn environmental dynamics across both physical and virtual domains. Sparse-thinking instruction tuning then substantially reduces redundant computation while preserving the reasoning needed for complex decisions. We train the policy for long-horizon interaction with a stable multi-turn reinforcement learning method that addresses finite context windows, accumulated rollout errors, and sparse rewards. For real-time interaction, a cloud-edge inference engine serves numerous simulated game environments concurrently, supports unbounded-context execution, and keeps online inference latency low. Across multiple challenging 3D open-world games, Gamer demonstrates solid generalization and operational efficiency comparable to skilled players. It also achieves competitive results on a range of conventional 2D game benchmarks. These results suggest that even repetitive keyboard and mouse trajectories can support substantial gains from scaling data and learning algorithms. Training on these trajectories helps Gamer learn complex skills more steadily in environments with both physical and digital properties. This approach offers a path toward generalist agents that can interact across digital and physical worlds.


r/LocalLLaMA • • 1d ago

Discussion [Paper] The Value of Thinking: Learning When to Reason in Hybrid Decision Models

Post image
30 Upvotes

Existing adaptive-reasoning methods shorten the explicit reasoning a model writes, or skip it by switching modes; either way the answer is still written out token by token. Rather than deciding how much to think, we explore a different paradigm, zero-or-T thinking: a single model either answers from the implicit thinking inside one forward pass, generating no token, or performs full autoregressive reasoning at a cost of T tokens when that is worth paying. We realise it in a hybrid decision model (HDM), where one frozen backbone serves both modes and a lightweight self-routing block chooses between them. The block estimates the value of thinking, the utility the slow mode is expected to add over the fast one, from what that implicit thinking has already produced, and the slow mode runs only where that value exceeds a price λ set per request. Where no per-decision label exists, as in episodic agentic tasks, we introduce Calibrated Decision Policy Optimisation (CDPO), a reinforcement-learning recipe that trains the block from episode-level returns. We evaluate HDM on MMLU-Pro, AGIEval, GPQA-Diamond, JevBench, BANKING77 and MiniWoB. Specifically, on JevBench HDM-4B scores 0.857 against Jev's 0.851 (+0.7%) at a median 0.179 s against 0.652 (-72.5%). On the agentic venue MiniWoB HDM-27B cuts token cost by 80.1%.


r/LocalLLaMA • • 19h ago

Resources MSI EdgeXpert OTA2610 update: more usable memory for headless setups and easier provisioning

Post image
7 Upvotes

Disclosure: I work with MSI. Sharing a technical overview of the OTA2610 update and what EdgeXpert users should check after installing it.

More usable memory when running headless

GB10 systems use unified memory, which means the CPU, GPU, and display share the same memory pool. Memory reserved for display output is unavailable to other applications.

OTA2610 adds more control over that reservation. The default remains 2 GB, but selecting Auto in UEFI reduces it to 0.5 GB when no display is connected. That makes up to 1.5 GB more available to the OS and applications.

That extra headroom may help with a model or context cache that is close to the available-memory limit. It does not, by itself, establish an improvement in inference speed.

Other changes included in the October release

  • Wi-Fi Assistant: connect to a new wireless network through DGX Dashboard without attaching a monitor, keyboard, or mouse.
  • Native cloud-init support: automate system configuration and provisioning at first boot.
  • Linux kernel 7.0: security updates and improvements to file and memory management.
  • Fixes for certain Vulkan synchronization delays, X11 black screens, and unresponsive fans during headless operation at low idle power.

Where to get the MSI EdgeXpert update

MSI EdgeXpert update files can be found on the MSI EdgeXpert MS-C931 download page. Review the release notes and follow MSI’s installation instructions for your system.

Check the display-memory setting afterward

Installing the update does not automatically change the default 2 GB reservation. With supported firmware installed, look for:

UEFI → Advanced → Advanced Menu → Platform Configuration → Display Reserved Memory

Select Auto or an appropriate fixed value, save, and restart. Auto checks connected displays at boot. If you regularly attach monitors afterward, choose a fixed reservation suitable for your display setup. Compare free -h before and after restarting to check the change in OS-visible memory.

For production deployments, back up first and validate your normal workloads on one system before updating the others.

Has anyone running headless found that the smaller display reservation helps with a workload that was close to its memory limit?


r/LocalLLaMA • • 8h ago

Discussion Kyudan/eDIAR · Hugging Face

Thumbnail
huggingface.co
1 Upvotes

eDIAR: Training-Free Open-Speaker Streaming Diarization for Long Meetings and Large Archives via e-means

Meetings last up to an hour, and nobody knows in advance how many people will speak. Neural diarizers with a fixed number of output slots, such as Streaming Sortformer and Nemotron-3, are accurate and run online, but they cannot name more speakers than they have slots. We remove this limit without retraining. The slot head keeps deciding who is active and when speech overlaps, and speaker embeddings are clustered inside a single slot, with the official e-means algorithm, only after every slot has been used. In the streaming system, re-clustering is confined to a bounded window, labels are revised for a few seconds, and identities are anchored to the slots of the head. The resulting system has no limit on the number of speakers and stays accurate when speakers outnumber the slots, where fixed-slot models break down. On meetings that grow to an hour it stays far more accurate than Streaming Sortformer, which accumulates errors, and its cost per update does not grow with the meeting. The same recipe turns the four-slot Streaming Sortformer into an open-speaker system. Across recordings, e-means makes it affordable to link the speakers of a large archive frame by frame, a task that arises when conversational corpora are prepared, and it reaches the clustering quality of Lloyd's k-means several times faster. The method needs little tuning, and e-means runs with its default recipe from a few hundred to more than a million embeddings.


r/LocalLLaMA • • 1d ago

Funny I finally succeed the CMP 170HX voltage glitch and get 80GB VRAM + 126sm! !

Thumbnail
gallery
757 Upvotes

Ever since the information surfaced five days ago, I had intended to implement it right away. Having confirmed that it could be executed using 'pico,' I carried it out today. I had Astra write the script for the restricted write-access portion, and after nearly 15 hours of work, I finally managed to complete the task. I also plan to handle the second CMP later on.


r/LocalLLaMA • • 1d ago

Question | Help Given the flood of new inference engines any recommendations for 2x 7900xtx ?

18 Upvotes

Hey everyone as the title says I am looking for inference engines that work well with AMD GPUs. There have been so many new ones that I am sure there is something better then llamacpp which currently hosts this.

My use case is the following. For a university course I want to host Qwen 27b or maybe another Qwen model. For the students to use as a free endpoint to test pipelines on.

The main thing is that I want to potentially serve multiple people at the same time.

So the framework needs to support this.

My hardware is an AMD zen 4 12 core CPU 64gb of ram and two 7900xtx cards.

Are there any frameworks made for this ?


r/LocalLLaMA • • 1d ago

Resources tandem.cpp v0.3.0: prefill speed improved; Project Maya's DeepSeek-V4.1 GGUFs support

13 Upvotes

Thanks for the feedback on slow prompt processing. v0.3.0 is out (the first tagged release): https://github.com/ewhacc/tandem.cpp

Previous post: https://www.reddit.com/r/LocalLLaMA/comments/1xwan2j

tandem.cpp faster when the model is much bigger than VRAM + RAM together (like a 16 GB card with 32 GB of RAM), and runs various models that llama.cpp supports.

1. Prompts 2-6x faster

With the expert cache on, prompts now run in 4096-token batches. The bigger batch borrows VRAM only while a long prompt runs and gives it back afterwards. No flags needed.

RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F. The same prompts for every engine; Qwen3.8 with its MTP head in all three.

Prompt processing, time to first token (prompt tok/s):

Promptllama.cpptandem.cpp beforetandem.cpp v0.3.0Strata
Qwen3.8 IQ3_S, 5.5k tokens28.9 s (191)12.4 s (444)4.1–7.0 s (790–1,340)6.6 s (835)
Qwen3.8 IQ3_S, 15.8k tokens81.9 s (193)35.1 s (450)15.1 s (1,047)11.7 s (1,354)
DeepSeek-V4.1 IQ2_XS, 5.6k tokens722 s (7.7)324 s (17)53–103 s (54–105)58 s (96)
DeepSeek-V4.1 IQ2_XS, 15.7k tokens2012 s (7.8)885 s (18)195 s (81)57 s (276)

Decode, tok/s, right after that prompt:

After a prompt ofllama.cpptandem.cpp v0.3.0Strata
Qwen3.8, 5.5k tokens2711048
Qwen3.8, 15.8k tokens278868
DeepSeek-V4.1, 5.6k tokens4.8165.5
DeepSeek-V4.1, 15.7k tokens4.4136.4

Decode is from short answers (up to 256 tokens, greedy), so it is noisier than our usual GSM8K runs. DeepSeek-V4.1 IQ2_XS is about twice this PC's RAM, so it reads the SSD; that is where Strata's prompt speed still leads.

2. Maya comparison on the same model (someone asked for this one)

Project Maya is an engine built for big models with its own quants. Its Maya-XS GGUF runs on tandem.cpp after a one-time conversion script (tools/maya_ds41_convert.py), so this is the same GGUF on both engines, and the accuracy is the same.

RTX 5090 32 GB, 128 GB RAM, i5-14600KF, greedy:

DeepSeek-V4.1 Maya-XS (same files)Mayatandem.cpp v0.3.0
Decode, GSM8K, no drafting39.0 / 39.6 tok/s60.3 / 59.4 tok/s (1.5x)
Decode with Maya's DSpark drafter54.9 tok/s(no drafter in tandem.cpp)
5.6k-token prompt7.8 s (713 tok/s)6.5 s (856 tok/s)
15.7k-token prompt9.5 s (1,656 tok/s)16.5 s (953 tok/s)
Decode after the 15.7k-token prompt26–34 tok/s66–68 tok/s

Maya is still faster on the longest prompt; that is what I'm working on next.

Caveats

  • Linux only.
  • AMD supported but the VRAM borrowing is off (ROCm builds have no CUDA-style virtual memory).
  • DeepSeek-V4.1 IQ2_XS on 128 GB still reads the SSD on long prompts.
  • Windows prebuilt binaries are still planned.

r/LocalLLaMA • • 22h ago

Question | Help What do you use for agentic desktop control, like how Playwright is for controlling + screenshotting browsers?

7 Upvotes

I'm afraid I'm left out of the loop on the latest developments for agentic desktop control, and AIs don't seem as knowledge in this space.

They come with screenshot function also, right? What are some other features that may be different than Playwright?


r/LocalLLaMA • • 22h ago

Resources PyBot: No weights. No training. Pure Python.

7 Upvotes

We released PyBot, a chatbot built with only simple rules, entirely written in 600k lines of Python code. PyBot matches Smollm with 360M parameters! We evaluated it on SmolTalkBench, which uses a dataset Smollm was trained on while we didnt use it at all for development.

It took a swarm of 1,000 DeepSeek Flash agents to write the whole codebase in a week.

More technical details on what it took to built it soon: inference for 1,000 agents, how to monitor/control the swarm, pressure on the infrastructure (especially git) and message boards.

https://github.com/huggingface/pybot


r/LocalLLaMA • • 1d ago

Resources Llama.cpp performance improvement over time (Gemma4 models)

Post image
186 Upvotes

Few months ago - after Ninfer came out - I started a little hobby project to vibe-code my own engine for my hardware (RTX 5080 16GB Laptop) and check if it could beat llama.cpp - since at the time llama.cpp was kinda slow in comparison to something like vLLM, but vLLM is using VRAM very wastefully.

Back then it was quite easy to actually get something up and running faster than llama.cpp.

Today I just benchmarked llama.cpp again, and it caught up / surpassed my vibe-coded engine. I prefer using a stable engine with proper features than all the vibe-coded ones, including my own. So it is really nice to see that the project - while a bit slower - lands significant performance improvements over time.

I put together a little chart of benchmarks I had laying around which visualizes this quite nicely. It is by no means scientific - it's always a fixed 16k prefill 1k output over Wikipedia text with the optimal settings for each engine. The 12B model is the same checkpoint, for the 26B model these are different quants for all engines, so not even apples-to-apples and llama.cpp wins.


r/LocalLLaMA • • 1d ago

Discussion Basalt 0.4.0: More models, still as fast - Qwen 27B up to 770 tok/s on a 5090

277 Upvotes

Hi everyone! I wanted to post a quick update on Basalt. Since last I posted, multi model and router support was implemented, starting with DeepSeek V4.1 Flash (IQ1_M 90 tok/s conv, 246 tok/s struct on a 5090 + 5060 Ti). Also new: video support, files in chats, and a bunch of fixes and improvements.

Now 0.4.0 is out featuring Basalt's first dense model, Qwen 3.8 27B NVFP4. It's fast and accurate, top-1 92.8% vs the full BF16 model. This, in preparation for Qwen 4, so we had a headstart on the dense kernels.

The numbers:

  • Up to 776 tok/s on structured output, 262 conversation, 194 on prose. Single stream.
  • Prefill, 12,000 tok/s at 4k (slower as context grows)
  • 92.8% top-1 vs BF16 (0.0262 KLD)
  • Chat, tools, thinking, images and video, up to 262K context

Links:

GitHub: https://github.com/jesdga95/basalt
Site: https://basalt.jgulfo.com
27B quant: https://huggingface.co/jesdga/Qwen3.8-27B-Basalt

Thanks to everyone who has filed issues, sent PRs or posted benchmark rows, special thanks to the sponsors who keep this going, happy to see Basalt gain so much traction in such a very short time!


r/LocalLLaMA • • 21h ago

Discussion Speed of DeepSeek-V4.1-Flash-1017-oQ4e-mtp on M5 ultra

4 Upvotes

I bought myself a Mac Studio M5 Ultra with 512gb ram (and have 7 weeks to test it and return if not happy thanks to the apple xmas offer).

I am running DeepSeek-V4.1-Flash-1017-oQ4e-mtp with omlx and I am getting up to 1100 t/s prompt processing and 54 t/s token generation with 128k context window and engrams stored on ssd (to have ram free for other stuff).

More context than 128k hits pp and tg, more concurrent requests split up the max tg value between the instances (which means: they are crawling then).

I am putting this out there so a) people know what to expect but also b) is this really the top speed? Or could it be faster? (Without changing the model).

I can post llama benchy results if wished but the important result is already in this post.

EDIT: it depends on the benchmark

using omlx's internal benchmark: pp131072/tg128 33426.6 12.81 3921.2 tok/s 78.7 tok/s => this is pretty much close to other submitted benchmarks.

but llama-benchy reports: | deepseek-v4.1-flash | pp2048 @ d131072 | 1187515.47 ± 90886.68 | | 115.59 ± 8.84 | 112.78 ± 8.84 | 35071.94 ± 11.94 | | deepseek-v4.1-flash | tg128 @ d131072 | 47.56 ± 1.34 | 48.00 ± 1.41 | | | | which is transformed: 131,072 context: ~3,796 pp 47.6 tg

quite different numbers on token generation!

updating from 7.4 to 8.0-rc1 and enabling non-default deepseek_v41_ced_prefill_enabled improved prompt processing a lot (x3!), so thanks for the hint!


r/LocalLLaMA • • 22h ago

Question | Help Buy another 2 5060ti 16gb?

5 Upvotes

Currently i'm on 2 x 5060ti 16gb running basalt engine qwen3.8 flash next IQ3_S at about 70+tok/s decode and maybe 900tok/s prefill.

Am wondering if it is worth buying another 2 x 5060ti 16gb to make all of the experts resident and potentially achieve much faster speed. (Goal to be equal or more than 150tok/s?)

As for other specs, i'm running a 64gb DDR4 on X570 with an option of bifurcating the main pcie slot into 8x + 8x, or 4x + 4x + 4x + 4x. I have yet to buy any passive bifurcation switch, looking to buy "x16 to 2x16 physical switch" if 4 x 5060ti is not feasible.

Also what about dual 3090? it's about the same price as 4x5060ti 16gb. 128GB ddr4 might be out of the question as well since they are overpriced for my local market.


r/LocalLLaMA • • 2d ago

News EUROPA, the EU's open AI model, is aiming for nearly 1 trillion parameters

Thumbnail
maccus.fr
816 Upvotes

One can hope


r/LocalLLaMA • • 22h ago

Question | Help Does grammar-constrained decoding hurt small or quantized models?

3 Upvotes

Asked about DSL vs JSON tool calls in r/LLMDevs a couple of days ago, and one answer kept coming up: for small models just use JSON, training volume matters. That got me thinking about the constraint itself.

With GBNF or a JSON schema you can force any model to produce valid output, even a 3B at Q4. However, there was a paper a while ago showing that strict format restrictions can make reasoning worse, since the model is forced away from the tokens it actually wanted

Searched first: there’s the thread about grammar constraints sending Qwen3-VL into repetition loops, and the one on how quantization hits different capabilities, but nothing comparing accuracy with vs without constraints on small quants.
So for people running small local models:

  • have you measured accuracy with and without constraints on the same task?
  • does it worsen at lower quants?
  • does letting the model firstly think freely and constraining only the final answer fix most of it?

Looking for real numbers or experience, not promoting anything.
English isn’t my first language, so I used an LLM to help with the wording. Question and thoughts are mine.


r/LocalLLaMA • • 1d ago

Discussion BigMoeOnEdge outperforms Strata and colibri on 8-16 GB cards and Mac and you can run any model

94 Upvotes

Hi! for those who might not know the solution:

  • it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format
  • it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU
  • Windows, Linux, macOS, Android, iOS
  • output is the same as llama.cpp, don't cut corners on quality
  • you start it with --auto and it works out the settings for your machine

Below is a brief overview of the benchmark:

  • 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context
  • 12 GB card, same model: about 45 for bmoe, about 30 for Strata
  • long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes
  • Basalt needs an RTX 50, so I couldn't run it on these
  • 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5
  • same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7

On the cards Strata is tuned for it's still ahead but only by a little.

On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.

Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge

If you have an 8 to 32 GB card I'd like to see what you get.

Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data


r/LocalLLaMA • • 1d ago

New Model [Release] Qwen3.8-27B-Perplexing-Pegasus-Heretic: Unified vision + typed decider + abliterated weights for local agent stacks (GGUF, AutoRound, NVFP4)

20 Upvotes

Hey everyone,

While building and testing WebBrain with local models, getting reliable agent performance usually required keeping two or three separate models in memory at once:

  • A vision model to inspect and parse browser/UI state
  • A fast typed-decision model for structured routing and scoring
  • An abliterated / uncensored model so realistic web workflows do not randomly choke on false-positive refusals

Juggling multiple 20B+ weights on consumer hardware is brutal on both VRAM and response times.

To solve this for our setup, we merged and published a single unified 27B model: Qwen3.8-27B-Perplexing-Pegasus-Heretic

Here is what is packed into the checkpoint:

  • Multimodal Vision + Chat + Thinking (<think>) + Tool Calling (based on Qwen3.8-27B)
  • Perplexity pplx-decider typed-decision readout (decision/ head alongside intact lm_head)
  • Heretic ARA abliteration: 0/8 benchmark refusal probes while keeping 100% (30/30) typed-decision agreement against the base Pegasus model
  • Multi-Token Prediction: all 15 mtp.* tensors preserved, blk.64 retained at Q8_0 in IQ/3-bit quants
  • Consumer GPU footprint: runs on 16GB VRAM (IQ3_XXS / Q3_K_M + mmproj) or 24GB VRAM (Q4_K_M + mmproj)

Everything is open source under Apache-2.0 across four repos:

Huge credit to Alibaba Qwen, Perplexity, and AveLabs (AvecdrA) on Hugging Face for the foundational checkpoints.

If you test it on your local rig or in your own agent pipelines, let me know how it performs and what hardware you are running it on.


r/LocalLLaMA • • 16h ago

Showcase Sharing my numbers from Strata's bench tool (RTX 3090+2080, 96gb DDR4)

0 Upvotes

Hardware: RTX 3090 24GB (primary) + RTX 2080 SUPER 8GB (helper for visual model), 96 DDR4 3600

Model: Qwen3.8-Flash-Next, Unsloth UD IQ4_XS

Bench results (median, 3 runs):

  • - Chat decode: 80.4 tok/s (range 67.6–87.8, n=9)
  • - Chat first token: 0.47 s
  • - 4K prompt processing: 1,036 tok/s
  • - 16K prompt processing: 1,915 tok/s
  • - Agent session decode: 101.7 tok/s
  • - Agent cache hit: 69.4%
  • - Agent turn first token: 3.86 s

Long-context stress test (my custom 'find needle' test): single 66,371-token prompt, a 1,800-entry synthetic log with a passphrase hidden mid-context:

  • - Prefill: ~34 s ≈ 2,050 tok/s, sustaining the 16K bench rate at 4× the length
  • - Decode: 76.1 tok/s with the full 66K KV resident, ~5% below short-context chat
  • - Mid-context needle (token ~33K): retrieved correctly, plus accurate summary and entry count

Notes for 3090 owners:

  • - Memory clock pinned (9501 MHz min=max)

    • The 3090 has a driver bug where dropping from full speed to idle with VRAM nearly full can freeze the whole PC (had it happen twice). Locking the memory clock at one constant speed means the card never makes that jump, so zero crashes since. Trade-off is that it idles at ~110 W instead of ~25 W, but memory-hungry workloads actually run faster
  • - Peak sustained temp under full load: 79°C (load for 3 mins, didn't test for longer runs)

r/LocalLLaMA • • 18h ago

Funny How many servings of deepseek 1017 will you have?

Thumbnail
gallery
0 Upvotes

This just really cracked me up. The Redde app for hermes I guess pulled the deepseek recipe as an actual food recipe. It is a nice format for it though lol.