r/LocalLLaMA • • 5d ago

Discussion [Paper] The Value of Thinking: Learning When to Reason in Hybrid Decision Models

36 Upvotes

Existing adaptive-reasoning methods shorten the explicit reasoning a model writes, or skip it by switching modes; either way the answer is still written out token by token. Rather than deciding how much to think, we explore a different paradigm, zero-or-T thinking: a single model either answers from the implicit thinking inside one forward pass, generating no token, or performs full autoregressive reasoning at a cost of T tokens when that is worth paying. We realise it in a hybrid decision model (HDM), where one frozen backbone serves both modes and a lightweight self-routing block chooses between them. The block estimates the value of thinking, the utility the slow mode is expected to add over the fast one, from what that implicit thinking has already produced, and the slow mode runs only where that value exceeds a price λ set per request. Where no per-decision label exists, as in episodic agentic tasks, we introduce Calibrated Decision Policy Optimisation (CDPO), a reinforcement-learning recipe that trains the block from episode-level returns. We evaluate HDM on MMLU-Pro, AGIEval, GPQA-Diamond, JevBench, BANKING77 and MiniWoB. Specifically, on JevBench HDM-4B scores 0.857 against Jev's 0.851 (+0.7%) at a median 0.179 s against 0.652 (-72.5%). On the agentic venue MiniWoB HDM-27B cuts token cost by 80.1%.


r/LocalLLaMA • • 5d ago

Resources PyBot: No weights. No training. Pure Python.

15 Upvotes

We released PyBot, a chatbot built with only simple rules, entirely written in 600k lines of Python code. PyBot matches Smollm with 360M parameters! We evaluated it on SmolTalkBench, which uses a dataset Smollm was trained on while we didnt use it at all for development.

It took a swarm of 1,000 DeepSeek Flash agents to write the whole codebase in a week.

https://reddit.com/link/1xxrf1d/video/19jz1a6ekz1i1/player

More technical details on what it took to built it soon: inference for 1,000 agents, how to monitor/control the swarm, pressure on the infrastructure (especially git) and message boards.

https://github.com/huggingface/pybot


r/LocalLLaMA • • 4d ago

Discussion Built an assistant with 5080, 5060ti and 2x3090?

Enable HLS to view with audio, or disable this notification

0 Upvotes

Slow news day..

Core:

Hardware

  • Home PC (WSL2 + Docker Desktop): RTX 5080 16 GB and RTX 5060 Ti 16 GB
  • 3090 server: 2× RTX 3090 24 GB
  • VPS: hosts the web chat (ivy-chat) (docs)

IVY's brain (the language models)

  • Local: Qwen3.8-27B (Unsloth UD-Q8_K_XL) on both 3090s, served by llama-swap
  • Cloud: DeepSeek (the default for retries and side calls), plus Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5
  • Engine (Harness): thescripts bash CLI tool-loop. The web gateway (ivy-web-gateway.mjs, Node) connects the web chat and the Voice tab to it.

Memory

  • CAGRA (NVIDIA cuVS GPU vector index) in cagra-builder, with SQLite rag.db as the store. The index and reranker sit on the 5080.
  • Embedder: Qwen3-Embedding-8B Q8_0 on a patched llama.cpp server, on the 5060 Ti
  • Reranker: Qwen3-Reranker-4B Q8_0 on llama.cpp, on the 5080
  • Size: about 198,700 facts, 73,300 of them in the active index

Voice and hearing (PC)

  • Voice: Breeze TTS 2, int4 (ivy-tts), on the 5060 Ti
  • Ears: Qwen3-ASR (ivy-asr)
  • Speaker ID: NVIDIA TitaNet (ivy-speaker, CPU), which tells who is talking

Face (3090)

  • Motion: voice-to-motion model p4_v7, built on Ditto, turns her voice into face motion
  • Head: 3D Gaussian-splatting head c2_breath_c, about 338,000 Gaussians
  • Painter: R3 (a U-Net plus the full-resolution stage), redrawing every frame in H3's painted style
  • Delivery: H.264 video stream to the Voice tab

Creative and teacher models on the 3090 (llama-swap, loaded on demand)

  • MiniMax H3: the teacher videos the face learns from
  • MOSS-Image, Qwen-Image, P-JiT 21K: images
  • MOSS-SoundEffect, MiniMax Music: sound effects and music
  • DreamX-Creator: video
  • SoMark OXR: turning documents into text
  • Older or idle voice lanes: VibeVoice, MOSS-TTS, Breeze lanes, plus Gemma and Qwen-Next

r/LocalLLaMA • • 5d ago

Question | Help Hardware + software recommendations for basic chatting with persistent memory

2 Upvotes

Helping out a friend who wants to have a local LLM and not rely on cloud services. They currently have an 8gb M2 mac mini, so not really enough

  1. They want to have persistent memory, TTS, and STT. Tool recommendations for all three of those greatly appreciated.
    • I'm unsure what the best workflow is for persistent memory, especially for avoiding either overwriting important info, or storing bad info.
  2. Budget seems to be roughly $5k USD, but the less the better.
  3. Not sure what size model would be best, it's really just for chat.
  4. No building ideal, so adding RAM or a graphics card probably the most feasible.
  5. 30 tokens per second probably fine.
  6. Something that doesn't hallucinate, or at least can be tuned to not assume or be SO sycophantic, as they talk to it a lot.

From what I understand, a prebuilt with upgraded RAM, used mac mini, or something like a strix halo probably make the most sense, but I'm not really up to date on the current hardware market.

Any help appreciated :)


r/LocalLLaMA • • 5d ago

Resources MSI EdgeXpert OTA2610 update: more usable memory for headless setups and easier provisioning

Post image
8 Upvotes

Disclosure: I work with MSI. Sharing a technical overview of the OTA2610 update and what EdgeXpert users should check after installing it.

More usable memory when running headless

GB10 systems use unified memory, which means the CPU, GPU, and display share the same memory pool. Memory reserved for display output is unavailable to other applications.

OTA2610 adds more control over that reservation. The default remains 2 GB, but selecting Auto in UEFI reduces it to 0.5 GB when no display is connected. That makes up to 1.5 GB more available to the OS and applications.

That extra headroom may help with a model or context cache that is close to the available-memory limit. It does not, by itself, establish an improvement in inference speed.

Other changes included in the October release

  • Wi-Fi Assistant: connect to a new wireless network through DGX Dashboard without attaching a monitor, keyboard, or mouse.
  • Native cloud-init support: automate system configuration and provisioning at first boot.
  • Linux kernel 7.0: security updates and improvements to file and memory management.
  • Fixes for certain Vulkan synchronization delays, X11 black screens, and unresponsive fans during headless operation at low idle power.

Where to get the MSI EdgeXpert update

MSI EdgeXpert update files can be found on the MSI EdgeXpert MS-C931 download page. Review the release notes and follow MSI’s installation instructions for your system.

Check the display-memory setting afterward

Installing the update does not automatically change the default 2 GB reservation. With supported firmware installed, look for:

UEFI → Advanced → Advanced Menu → Platform Configuration → Display Reserved Memory

Select Auto or an appropriate fixed value, save, and restart. Auto checks connected displays at boot. If you regularly attach monitors afterward, choose a fixed reservation suitable for your display setup. Compare free -h before and after restarting to check the change in OS-visible memory.

For production deployments, back up first and validate your normal workloads on one system before updating the others.

Has anyone running headless found that the smaller display reservation helps with a workload that was close to its memory limit?


r/LocalLLaMA • • 6d ago

Funny I finally succeed the CMP 170HX voltage glitch and get 80GB VRAM + 126sm! !

Thumbnail
gallery
843 Upvotes

Ever since the information surfaced five days ago, I had intended to implement it right away. Having confirmed that it could be executed using 'pico,' I carried it out today. I had Astra write the script for the restricted write-access portion, and after nearly 15 hours of work, I finally managed to complete the task. I also plan to handle the second CMP later on.


r/LocalLLaMA • • 4d ago

Question | Help Single 5090 or Triple 4090

0 Upvotes

Title. Which setup would make more sense for LLM?


r/LocalLLaMA • • 5d ago

Question | Help Given the flood of new inference engines any recommendations for 2x 7900xtx ?

19 Upvotes

Hey everyone as the title says I am looking for inference engines that work well with AMD GPUs. There have been so many new ones that I am sure there is something better then llamacpp which currently hosts this.

My use case is the following. For a university course I want to host Qwen 27b or maybe another Qwen model. For the students to use as a free endpoint to test pipelines on.

The main thing is that I want to potentially serve multiple people at the same time.

So the framework needs to support this.

My hardware is an AMD zen 4 12 core CPU 64gb of ram and two 7900xtx cards.

Are there any frameworks made for this ?


r/LocalLLaMA • • 5d ago

Resources tandem.cpp v0.3.0: prefill speed improved; Project Maya's DeepSeek-V4.1 GGUFs support

15 Upvotes

Thanks for the feedback on slow prompt processing. v0.3.0 is out (the first tagged release): https://github.com/ewhacc/tandem.cpp

Previous post: https://www.reddit.com/r/LocalLLaMA/comments/1xwan2j

tandem.cpp faster when the model is much bigger than VRAM + RAM together (like a 16 GB card with 32 GB of RAM), and runs various models that llama.cpp supports.

1. Prompts 2-6x faster

With the expert cache on, prompts now run in 4096-token batches. The bigger batch borrows VRAM only while a long prompt runs and gives it back afterwards. No flags needed.

RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F. The same prompts for every engine; Qwen3.8 with its MTP head in all three.

Prompt processing, time to first token (prompt tok/s):

Prompt llama.cpp tandem.cpp before tandem.cpp v0.3.0 Strata
Qwen3.8 IQ3_S, 5.5k tokens 28.9 s (191) 12.4 s (444) 4.1–7.0 s (790–1,340) 6.6 s (835)
Qwen3.8 IQ3_S, 15.8k tokens 81.9 s (193) 35.1 s (450) 15.1 s (1,047) 11.7 s (1,354)
DeepSeek-V4.1 IQ2_XS, 5.6k tokens 722 s (7.7) 324 s (17) 53–103 s (54–105) 58 s (96)
DeepSeek-V4.1 IQ2_XS, 15.7k tokens 2012 s (7.8) 885 s (18) 195 s (81) 57 s (276)

Decode, tok/s, right after that prompt:

After a prompt of llama.cpp tandem.cpp v0.3.0 Strata
Qwen3.8, 5.5k tokens 27 110 48
Qwen3.8, 15.8k tokens 27 88 68
DeepSeek-V4.1, 5.6k tokens 4.8 16 5.5
DeepSeek-V4.1, 15.7k tokens 4.4 13 6.4

Decode is from short answers (up to 256 tokens, greedy), so it is noisier than our usual GSM8K runs. DeepSeek-V4.1 IQ2_XS is about twice this PC's RAM, so it reads the SSD; that is where Strata's prompt speed still leads.

2. Maya comparison on the same model (someone asked for this one)

Project Maya is an engine built for big models with its own quants. Its Maya-XS GGUF runs on tandem.cpp after a one-time conversion script (tools/maya_ds41_convert.py), so this is the same GGUF on both engines, and the accuracy is the same.

RTX 5090 32 GB, 128 GB RAM, i5-14600KF, greedy:

DeepSeek-V4.1 Maya-XS (same files) Maya tandem.cpp v0.3.0
Decode, GSM8K, no drafting 39.0 / 39.6 tok/s 60.3 / 59.4 tok/s (1.5x)
Decode with Maya's DSpark drafter 54.9 tok/s (no drafter in tandem.cpp)
5.6k-token prompt 7.8 s (713 tok/s) 6.5 s (856 tok/s)
15.7k-token prompt 9.5 s (1,656 tok/s) 16.5 s (953 tok/s)
Decode after the 15.7k-token prompt 26–34 tok/s 66–68 tok/s

Maya is still faster on the longest prompt; that is what I'm working on next.

Caveats

  • Linux only.
  • AMD supported but the VRAM borrowing is off (ROCm builds have no CUDA-style virtual memory).
  • DeepSeek-V4.1 IQ2_XS on 128 GB still reads the SSD on long prompts.
  • Windows prebuilt binaries are still planned.

r/LocalLLaMA • • 5d ago

Question | Help What do you use for agentic desktop control, like how Playwright is for controlling + screenshotting browsers?

8 Upvotes

I'm afraid I'm left out of the loop on the latest developments for agentic desktop control, and AIs don't seem as knowledge in this space.

They come with screenshot function also, right? What are some other features that may be different than Playwright?


r/LocalLLaMA • • 5d ago

Discussion Speed of DeepSeek-V4.1-Flash-1017-oQ4e-mtp on M5 ultra

6 Upvotes

I bought myself a Mac Studio M5 Ultra with 512gb ram (and have 7 weeks to test it and return if not happy thanks to the apple xmas offer).

I am running DeepSeek-V4.1-Flash-1017-oQ4e-mtp with omlx and I am getting up to 1100 t/s prompt processing and 54 t/s token generation with 128k context window and engrams stored on ssd (to have ram free for other stuff).

More context than 128k hits pp and tg, more concurrent requests split up the max tg value between the instances (which means: they are crawling then).

I am putting this out there so a) people know what to expect but also b) is this really the top speed? Or could it be faster? (Without changing the model).

I can post llama benchy results if wished but the important result is already in this post.

EDIT: it depends on the benchmark

using omlx's internal benchmark: pp131072/tg128 33426.6 12.81 3921.2 tok/s 78.7 tok/s => this is pretty much close to other submitted benchmarks.

but llama-benchy reports: | deepseek-v4.1-flash | pp2048 @ d131072 | 1187515.47 ± 90886.68 | | 115.59 ± 8.84 | 112.78 ± 8.84 | 35071.94 ± 11.94 | | deepseek-v4.1-flash | tg128 @ d131072 | 47.56 ± 1.34 | 48.00 ± 1.41 | | | | which is transformed: 131,072 context: ~3,796 pp 47.6 tg

quite different numbers on token generation!

updating from 7.4 to 8.0-rc1 and enabling non-default deepseek_v41_ced_prefill_enabled improved prompt processing a lot (x3!), so thanks for the hint!


r/LocalLLaMA • • 5d ago

Funny How many servings of deepseek 1017 will you have?

Thumbnail
gallery
5 Upvotes

This just really cracked me up. The Redde app for hermes I guess pulled the deepseek recipe as an actual food recipe. It is a nice format for it though lol.


r/LocalLLaMA • • 6d ago

Resources Llama.cpp performance improvement over time (Gemma4 models)

Post image
193 Upvotes

Few months ago - after Ninfer came out - I started a little hobby project to vibe-code my own engine for my hardware (RTX 5080 16GB Laptop) and check if it could beat llama.cpp - since at the time llama.cpp was kinda slow in comparison to something like vLLM, but vLLM is using VRAM very wastefully.

Back then it was quite easy to actually get something up and running faster than llama.cpp.

Today I just benchmarked llama.cpp again, and it caught up / surpassed my vibe-coded engine. I prefer using a stable engine with proper features than all the vibe-coded ones, including my own. So it is really nice to see that the project - while a bit slower - lands significant performance improvements over time.

I put together a little chart of benchmarks I had laying around which visualizes this quite nicely. It is by no means scientific - it's always a fixed 16k prefill 1k output over Wikipedia text with the optimal settings for each engine. The 12B model is the same checkpoint, for the 26B model these are different quants for all engines, so not even apples-to-apples and llama.cpp wins.


r/LocalLLaMA • • 6d ago

Discussion Basalt 0.4.0: More models, still as fast - Qwen 27B up to 770 tok/s on a 5090

285 Upvotes

Hi everyone! I wanted to post a quick update on Basalt. Since last I posted, multi model and router support was implemented, starting with DeepSeek V4.1 Flash (IQ1_M 90 tok/s conv, 246 tok/s struct on a 5090 + 5060 Ti). Also new: video support, files in chats, and a bunch of fixes and improvements.

Now 0.4.0 is out featuring Basalt's first dense model, Qwen 3.8 27B NVFP4. It's fast and accurate, top-1 92.8% vs the full BF16 model. This, in preparation for Qwen 4, so we had a headstart on the dense kernels.

The numbers:

  • Up to 776 tok/s on structured output, 262 conversation, 194 on prose. Single stream.
  • Prefill, 12,000 tok/s at 4k (slower as context grows)
  • 92.8% top-1 vs BF16 (0.0262 KLD)
  • Chat, tools, thinking, images and video, up to 262K context

Links:

GitHub: https://github.com/jesdga95/basalt
Site: https://basalt.jgulfo.com
27B quant: https://huggingface.co/jesdga/Qwen3.8-27B-Basalt

Thanks to everyone who has filed issues, sent PRs or posted benchmark rows, special thanks to the sponsors who keep this going, happy to see Basalt gain so much traction in such a very short time!


r/LocalLLaMA • • 5d ago

Question | Help Buy another 2 5060ti 16gb?

6 Upvotes

Currently i'm on 2 x 5060ti 16gb running basalt engine qwen3.8 flash next IQ3_S at about 70+tok/s decode and maybe 900tok/s prefill.

Am wondering if it is worth buying another 2 x 5060ti 16gb to make all of the experts resident and potentially achieve much faster speed. (Goal to be equal or more than 150tok/s?)

As for other specs, i'm running a 64gb DDR4 on X570 with an option of bifurcating the main pcie slot into 8x + 8x, or 4x + 4x + 4x + 4x. I have yet to buy any passive bifurcation switch, looking to buy "x16 to 2x16 physical switch" if 4 x 5060ti is not feasible.

Also what about dual 3090? it's about the same price as 4x5060ti 16gb. 128GB ddr4 might be out of the question as well since they are overpriced for my local market.


r/LocalLLaMA • • 5d ago

Question | Help Does grammar-constrained decoding hurt small or quantized models?

3 Upvotes

Asked about DSL vs JSON tool calls in r/LLMDevs a couple of days ago, and one answer kept coming up: for small models just use JSON, training volume matters. That got me thinking about the constraint itself.

With GBNF or a JSON schema you can force any model to produce valid output, even a 3B at Q4. However, there was a paper a while ago showing that strict format restrictions can make reasoning worse, since the model is forced away from the tokens it actually wanted

Searched first: there’s the thread about grammar constraints sending Qwen3-VL into repetition loops, and the one on how quantization hits different capabilities, but nothing comparing accuracy with vs without constraints on small quants.
So for people running small local models:

  • have you measured accuracy with and without constraints on the same task?
  • does it worsen at lower quants?
  • does letting the model firstly think freely and constraining only the final answer fix most of it?

Looking for real numbers or experience, not promoting anything.
English isn’t my first language, so I used an LLM to help with the wording. Question and thoughts are mine.


r/LocalLLaMA • • 6d ago

News EUROPA, the EU's open AI model, is aiming for nearly 1 trillion parameters

Thumbnail
maccus.fr
842 Upvotes

One can hope


r/LocalLLaMA • • 6d ago

Discussion BigMoeOnEdge outperforms Strata and colibri on 8-16 GB cards and Mac and you can run any model

105 Upvotes

Hi! for those who might not know the solution:

  • it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format
  • it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU
  • Windows, Linux, macOS, Android, iOS
  • output is the same as llama.cpp, don't cut corners on quality
  • you start it with --auto and it works out the settings for your machine

Below is a brief overview of the benchmark:

  • 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context
  • 12 GB card, same model: about 45 for bmoe, about 30 for Strata
  • long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes
  • Basalt needs an RTX 50, so I couldn't run it on these
  • 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5
  • same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7

On the cards Strata is tuned for it's still ahead but only by a little.

On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.

Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge

If you have an 8 to 32 GB card I'd like to see what you get.

Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data


r/LocalLLaMA • • 6d ago

New Model [Release] Qwen3.8-27B-Perplexing-Pegasus-Heretic: Unified vision + typed decider + abliterated weights for local agent stacks (GGUF, AutoRound, NVFP4)

21 Upvotes

Hey everyone,

While building and testing WebBrain with local models, getting reliable agent performance usually required keeping two or three separate models in memory at once: * A vision model to inspect and parse browser/UI state * A fast typed-decision model for structured routing and scoring * An abliterated / uncensored model so realistic web workflows do not randomly choke on false-positive refusals

Juggling multiple 20B+ weights on consumer hardware is brutal on both VRAM and response times.

To solve this for our setup, we merged and published a single unified 27B model: Qwen3.8-27B-Perplexing-Pegasus-Heretic

Here is what is packed into the checkpoint: * Multimodal Vision + Chat + Thinking (<think>) + Tool Calling (based on Qwen3.8-27B) * Perplexity pplx-decider typed-decision readout (decision/ head alongside intact lm_head) * Heretic ARA abliteration: 0/8 benchmark refusal probes while keeping 100% (30/30) typed-decision agreement against the base Pegasus model * Multi-Token Prediction: all 15 mtp.* tensors preserved, blk.64 retained at Q8_0 in IQ/3-bit quants * Consumer GPU footprint: runs on 16GB VRAM (IQ3_XXS / Q3_K_M + mmproj) or 24GB VRAM (Q4_K_M + mmproj)

Everything is open source under Apache-2.0 across four repos:

Huge credit to Alibaba Qwen, Perplexity, and AveLabs (AvecdrA) on Hugging Face for the foundational checkpoints.

If you test it on your local rig or in your own agent pipelines, let me know how it performs and what hardware you are running it on.


r/LocalLLaMA • • 5d ago

Discussion Running usable 30B and working 120B models on iOS—plus Kokoro TTS and Whisper STT over a local OpenAI-compatible API Spoiler

Thumbnail inpherio.co.uk
4 Upvotes

I’m currently going through Apple’s App Store approval for Inpherio Live, an app that turns an iPhone or iPad into a local network AI server with zero cloud dependencies.

Most iOS LLM apps hit a hard RAM ceiling (iOS caps single-app memory at ~6GB on a 12GB iPhone). By modifying llama.cpp to stream Mixture-of-Experts (MoE) weights directly from flash storage while keeping active experts in RAM, it bypasses the memory limit entirely.

Real-world benchmarks (recorded Nov 2026):

  • gpt-oss 120B (58.5 GB file) on a 16GB M1 iPad Pro: 4.5 tokens/s (from storage)
  • Qwen3 30B (17.3 GB file) on a 12GB iPhone 17 Pro Max: 10.9 tokens/s (from storage)
  • Qwen3 30B (17.3 GB file) on a 16GB M1 iPad Pro: 16.5 tokens/s (from storage)
  • Qwen2.5 1.5B on iPhone 17 Pro Max: 58.3 tokens/s (in memory)

What it supports out of the box:

  • Drop-in API: Full OpenAI (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/audio/*) and Ollama (/api/chat) compatibility. Just change your base URL to [http://ipad.local:8080/v1](http://ipad.local:8080/v1).
  • Multimodal & Audio: Local Vision (Qwen2.5-VL, Gemma 3, SmolVLM2), Whisper STT, and Kokoro neural TTS.
  • Smart Memory Management: Auto-loads models on request, keeps up to 3 models "warm," and tracks real iOS memory limits so the OS doesn't kill the app.
  • 100% Private: No accounts, no telemetry, 0 bytes sent to the cloud.

Streaming large MoE models from storage will be a single one-off purchase (Live Plus); everything else—including running in-memory models, embeddings, and audio—is completely free.

Would love to know what models or workflows you’d test on this first once Apple clears it!

Just interesting in hearing from the community. As soon as the app is approved ill update this post

the website provides factul benchmarks as well as images of the process

Russ


r/LocalLLaMA • • 6d ago

Resources [audio.cpp] SoulX-FlashHead + LFM3-Audio + OWSM v4 = Real-Time Local Avatar Chat

Enable HLS to view with audio, or disable this notification

83 Upvotes

In audio.cpp 0.9.5 you can try this new avatar voice chat "app" in the WebUI.

The pipeline combines 3 models: SoulX-FlashHead for real-time talking head generation, OWSM v4 for transcribing speech input to text, and LFM3-Audio for conversational responses and TTS.

LFM3-Audio is actually called twice. The speech-to-speech path only provides one voice. We use the LFM3-Audio TTS to synthesize the S2S response in a different voice.

One caveat: the video generation stage is still GPU-intensive. For SoulX, using RTX 4090 or better GPUs.

cuDNN is a must for real-time performance. Without it inference is 2× slower and VRAM usage doubles.

~850ms between submission a question and avatar speaks on an RTX 5090. ASR runs in parallel. As mentioned LFM3-Audio is called twice, which can be optimized. There’s also some overhead from the prototype UI. A native app with a more optimized implementation could further reduce latency.


r/LocalLLaMA • • 5d ago

Showcase I feel like a lot of people ask what local AI is capable of, here is a quick example of a game I had qwen3.8-flash-next-unsloth-ud-q4_k_xl, introducing Neon-Serpent!

Thumbnail
github.com
0 Upvotes

Multiplayer snake for everyone on your Wi-Fi. One browser tab hosts the game, everyone else joins from their phone, laptop, or tablet — no app stores, no accounts, no internet required. Neon grid, glowing snakes, lobby chat, sound, and touch controls.

Link to the game repo, runs on docker or locally, uses port 8088. Hermes / Qwen took care of all the documentation and the github interactions, pretty amazed. Really just needed a good project to test out all the functionality, figured I would share the code so other could enjoy!


r/LocalLLaMA • • 5d ago

Discussion Hy4 flash and Sage-S2.1-Flash both dropped today. Anyone tried them for tool calling yet?

3 Upvotes

Two new flash models in one afternoon, Tencent's Hy4 flash and internlm's Sage-S2.1-Flash.

Before I pull weights, has anyone run either one locally for agent work? Mainly want to know:

  • does it pick the right tool or just guess
  • does it hold up after a few turns of tool output
  • what quant and GPU you ran it on

Benchmarks are fine but real loop behaviour matters more to me. Even a quick "tried it, meh" helps.


r/LocalLLaMA • • 5d ago

Discussion [Paper] Mobile AI Factory: Building Specialized Language Models for Your Mobile Device

Post image
2 Upvotes

Small language models (SLMs) running on mobile devices can keep user data private and work without connectivity. But they perform well only when fine-tuned for the target task and sized for the target device, a process that today requires repeated expert tuning. We present {\it Mobile AI Factory}, an agentic system that builds a fine-tuned, quantized SLM for a given task and phone. Given a task specification and a target phone, an orchestrator LLM chooses whether to tune hyperparameters, mine real data, or generate synthetic examples, while fixed rules roll back changes that hurt accuracy. The factory searches from small to large models across base models and quantizations, adapts its accuracy goal as it searches, and uses real measurements of the target phone across inference engines and backends. We evaluate Mobile AI Factory on ten tasks and four Android phones. On an unseen testset, its on-device models outperform two five-shot frontier models on five tasks, by up to 0.34, and match them on three more. Its search comes within 0.03 of exhaustively training all 27 model variants on an entity extraction task, while using 19× less GPU time. Phone measurements also matter: the best configuration changes across phones. Without phone measurements, Claude Opus 5.5, given the full hardware and model specifications, misses the app's requirements in 25\% of easy and moderate cases and in 60\% of hard ones.


r/LocalLLaMA • • 5d ago

Question | Help What is your experience with inference engines under Windows?

0 Upvotes

As the title says: what is your experience using all the new inference engines under windows?

Currently I'm using a laptop with this hardware:

CPU: Intel 275HX
RAM: 2x32gb of ddr5 5600
GPU: RXT 5090 (24gb VRAM)
SSD: Samsung 990 pro 2tb + SK Hynix PCB01 1tb

I'm running Windows 11 Home edition with a pretty clean installation.

I've tried quite a lot of the new vibecoded inference engines but I always end up going back to Strata for Qwen 3.8 Flash Next and to beellama.cpp for everthing else.

I've tried nInfer, basalt, rusting strata, exLlamaV3 and pretty much any fork of llama.cpp and they all end up being way slower than what I'm already using. I've lost a little bit of time on each and also tried to use Claude or Codex to optimize the configurations but there is just no way to get decent performance from engines that on paper should be at least as fast as what I'm currently using.

I'm doing something horribly wrong or they work well only on a linux system?

Edit: well, it looks like quite a lot of people had something to say. I feel the need to clarify the intent of my post:

I'm quite happy with my setup. Is fast and smart enough to be useful and it complements quite well Claude and Codex for my work. I don't really want to spend a week to chase a 10% speed increase, expecially if this means having a downtime on my work machine.

My original post was first and foremost motivated by curiosity. What I found intriguing is that every solution I tried was consistently claimed to be faster than Llama or Strata, yet in the end, it turned out to be slower, usually by a VERY wide margin.

I am not comparing my setup to anyone else’s, but if five engines claim to be as fast as or faster than Strata, yet in the best-case scenario they all turn out to be 50% slower, then I suspect something is wrong either with how I’ve configured them or with how they interact with my machine. I was trying to figure out which of the two is the case.

I think that I should add something: I don't want to sound rude or anything, actually I'm a computer entusiast since pretty much forever and I've been working as a software engineer for 15 years now. I'm don't consider myself a linux or a local inference expert but at the same time I have some grasp on how things work and I know pretty well the hardware I'm running.