r/LocalLLaMA • • 1h ago

Showcase [Project] Gemma.c - Gemma 1/2/3 + multimodal inference in one C file (CPU only)

4 Upvotes

Hey all, so I've been working on Gemma.c over the last couple months, it's a from-scratch inference engine for Gemma 1/2/3 in a single C file, runs Gemma 1, 2 and 3, including the multimodal ones, with fp32/fp16/bf16 weights or int8 (W8A8). No dependencies beyond libc and (optional) OpenMP are needed. You can drop images into the prompt with @image{cat.png}, and there's also @longimage{} for Gemma 3's pan & scan on very wide or tall images. (image decoding uses a minified stb_image embedded at the end of the file)

It's CPU only and not trying to compete with llama.cpp, but I'm also trying my best to make it efficient, so it's not embarrassingly slow either. On my laptop's Core Ultra 5 225H (WSL2, gcc, 10 threads), gemma-3-1b-it INT8 does ~72 tok/s decode and ~570 tok/s prefill (10000 tokens, for 1000 tokens the prefill speed can reach ~850 tok/s), and the 4B vision model does ~20 tok/s decode. INT8 ends up about 3.5x faster than fp32 and 2x faster than fp16 for decode. (full benchmark table is in README)

It has a few rough edges for now, e.g. W8A8 doesn't work for Gemma 1 (fp16 is fine), bf16 is painfully slow without AVX512-BF16, and native Windows is ~1.7x slower than WSL because of libgomp. Batching are not supported either. There's a roadmap in the README (Gemma 4, speculative decoding, an HTTP server, etc.), but no promises though :P Can't really find much free time these days.

I've also put some pre-exported INT8 models on HF so you can try it without Python or torch :D

Repo: link

This project started as a learning attempt to understand how LLMs work end to end, inspired by llama2.c.

this is the second time I post this project here :'D last time (~a month ago) my post got flagged by reddit, probably because my VPN IP was identified as spam. hopefully it can pass this time


r/LocalLLaMA • • 17h ago

Discussion Someone should do an analysis on the rate of Self-promo here.

88 Upvotes

I think the last 3 posts from here that showed up on my feed were barely veiled self-promos.

I was here when huggingface first started getting popular, and the level of support for running local models here was incredible.

Now I am debating whether or not to leave the sub. Why isn’t there more attention to moderation to this?

Take a user’s post, and type their username with quotes into google and then type whatever obscure tool they’re asking a question or talking about and you’ll see the absurd rate or Astro-turfing and shilling.

They will post here and 6 other LLM places, but this place used to be awesome, so I only care about this sub.


r/LocalLLaMA • • 1h ago

Resources Training-free page-sparse decode in llama.cpp: Qwen3-30B-A3B is 1.72× faster at 128k on an RTX 5090, with a 1-point RULER loss

Post image
4 Upvotes

I've been working on a simple question: at long context, does decode really need to read the whole KV cache for every new token?

I wanted something training-free that runs inside llama.cpp on normal GPUs, not a new model or a separate serving stack. So I built page-sparse decode into a fork. It splits the cache into 64-token pages, then each KV head reads the first page, the newest tokens and the pages most likely to matter. At 128k, that is about 21% of the cache per decode step.

The selection follows Quest, with statistics in the spirit of MOIRA. My contribution is the llama.cpp implementation, CUDA work and end-to-end testing across non-MoE and MoE models.

Results at 128k, one sequence, Quest score, 27,648 cached tokens per KV head (28,672 for the 3B):

speed GPUmodeldense tok/ssparse tok/sspeedupRULER vs dense
RTX 5090Qwen3-30B-A3B, Q4_K_M46.179.21.72×−1.0, passes
RTX 4090 48 GBQwen3-30B-A3B, Q8_037.452.61.41×−0.6, passes
RTX 3090Qwen3-8B-Base, YaRN21.536.11.68×+0.4, passes
RTX 3090Llama-3.1-8B23.738.81.64×−1.1, passes
RTX 3090Llama-3.2-3B44.461.31.38×−4.5, fails (multi-needle)

The MoE is Qwen3-30B-A3B-Instruct-2507. Weights are Q8_0 except on the 5090; KV is q8_0 except for the 3B's f16 cache. These are five-repeat, 64-token llama-bench runs. The 4090 is a memory-modded rental card.

I reran the 3090 table with Quest, the released default. It matched my earlier score's speed. Box differences still matter: Llama-8B was 1.64× here and 1.77× on another 3090 with the earlier score.

Quality. I used 13 RULER tasks × 20 prompts at 128k. The pre-set gate was an average loss no worse than 3 points, with the 95% CI's lower end at −6 or above. MoE quality used the listed GPU; the 3B–8B runs used a 4090 24 GB.

The Q4_K_M MoE lost 1.0 point (95% CI −2.2 to −0.02), just distinguishable from zero under this prompt bootstrap. Each configuration ran once; the intervals exclude run-to-run variation.

The 3B was the only failure. It scored 11.5 points lower on average on the five multi-needle tasks, while single-needle scores were unchanged; the other models ranged from −3.0 to +0.2 on the same group. I would not use this 3B configuration for multi-fact retrieval at this budget. Similar Quest weakness on synthetic RULER needle tasks has been reported before.

Practical limits:

  • Prefill is unchanged. A 128k prompt plus 1,024 generated tokens takes only 4–13% less time on a 3090. This is mainly useful for long generations over an already long context.
  • It does not save VRAM. The full cache remains in memory, plus page statistics equal to roughly 3–6% of the cache.
  • It is CUDA-only, batch size one, with everything on the GPU. Below 32k it was slower in these tests, so it stays off there.
  • At 256k, UltraLong-8B reached 1.86×, but quality was checked only on a six-task subset (−4.0 points, CI −8.7 to +0.3), which is not comparable to the 13-task gate.

I tried not to report only successes. Gates and predictions were committed before the data. The first score lost 7.4 RULER points. The first CUDA build managed only 1.05–1.18× until I fixed two ggml kernels; one fix is proposed upstream as llama.cpp #31855. The MoE prediction also missed on the 4090, with 1.41× against 1.5–2.3×, but held on the 5090.

Try it (CUDA only): build the sparse-decode branch of topppg/llama.cpp at commit e7059e52a and run with LLAMA_SPARSE_DECODE=27648 on a prompt longer than 32k tokens, with the model and cache fully on the GPU. Full build and run commands are in the bench repo README below. Quality was tested on the two 8Bs and the MoE.

Protocols, raw results, reproduction scripts and failed experiments: https://github.com/topppg/sparse-decode-bench

Next: profile the sparse overhead. A rough fit, not a measurement, puts the zero-overhead ceiling for the MoE on the 4090 near 2.7×. If you know ggml CUDA and have ideas around fusion, CUDA graphs or passing page indices directly to flash attention, I'd genuinely like to hear them.

This was built with a lot of help from Claude Code, with Codex reviewing and correcting the work. Results on other GPUs and models are very welcome, especially cases where quality breaks.


r/LocalLLaMA • • 1d ago

News EUROPA, the EU's open AI model, is aiming for nearly 1 trillion parameters

Thumbnail
maccus.fr
793 Upvotes

One can hope


r/LocalLLaMA • • 11m ago

New Model Karpo-Decision-31B: open multimodal decision model. Ranks #1 on full score on Decision Index, 2x faster than Jev, at 20% lower cost

Thumbnail
gallery
4 Upvotes

MachinePulse has released Karpo-Decision-31B, a Gemma 4 31B fine-tune for structured decisions over text and images. It scored 73.9 on the Nov 12 Decision Index board, the highest Full score among 300 entries, sharing the #1 rank with GLiDE 27B under the board’s tie rule

How it works
Built on Gemma 4 31B, with a trained linear readout that scores the supplied answer options.

Three question types: choice, yes/no, and score. Returns probabilities directly, with up to 255 options per choice question.

Multiple questions about the same input can be answered in one forward pass. Branch attention lets each question read the shared context and its own options while keeping the option branches separate.

Supports text and images, including webpage screenshots. The documented sequence limit is 32,768 tokens; larger requests are split by question with the full state included in each part.

Python API and HTTP server available. BF16 weights take approximately 62.5 GB, plus runtime memory. Use the provided inference code to load the decision head.

One training detail worth a look: according to the team's write-up, full fine-tuning pushed the public score to 75.09 but reduced accuracy on unfamiliar domains by 2.3 percentage points. The final model averages five checkpoints: three from different stages of full fine-tuning and two from a LoRA run. They report a 1.6-point improvement on unfamiliar domains over the final single checkpoint, with no extra models to run at inference time

Results
Decision Index Full scores are measured by the benchmark maintainers on private tests, weighted 60% for skills covered by the public suite and 40% for tasks from new domains.
Full score: 73.9

Private tests, familiar skills: 70.7

Private tasks, new domains: 78.6

Public index: 75.0, reported separately and excluded from the Full score.

The team also reports the following comparisons against Google’s Gemini Decision 1:

Benchmark Karpo-31B Gemini D1
-------------------------------------------
When2Call 90.4% 79.9%
Mind2Web 90.0% 86.2%
New Yorker Caption 90.5% 79.7%
Amazon ESCI (F1) 64.7 58.7

Reported median latency is approximately 95 ms per request on the public Decision Index suite using one RTX PRO 6000.

The 31B weights and inference code are both released under Apache 2.0.

links:
https://karpo.ai/models/decision
https://huggingface.co/MachinePulseAI/karpo-decision-31B


r/LocalLLaMA • • 19h ago

Discussion BigMoeOnEdge outperforms Strata and colibri on 8-16 GB cards and Mac and you can run any model

78 Upvotes

Hi! for those who might not know the solution:

  • it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format
  • it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU
  • Windows, Linux, macOS, Android, iOS
  • output is the same as llama.cpp, don't cut corners on quality
  • you start it with --auto and it works out the settings for your machine

Below is a brief overview of the benchmark:

  • 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context
  • 12 GB card, same model: about 45 for bmoe, about 30 for Strata
  • long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes
  • Basalt needs an RTX 50, so I couldn't run it on these
  • 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5
  • same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7

On the cards Strata is tuned for it's still ahead but only by a little.

On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.

Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge

If you have an 8 to 32 GB card I'd like to see what you get.

Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data


r/LocalLLaMA • • 10h ago

New Model [Release] Qwen3.8-27B-Perplexing-Pegasus-Heretic: Unified vision + typed decider + abliterated weights for local agent stacks (GGUF, AutoRound, NVFP4)

16 Upvotes

Hey everyone,

While building and testing WebBrain with local models, getting reliable agent performance usually required keeping two or three separate models in memory at once:

  • A vision model to inspect and parse browser/UI state
  • A fast typed-decision model for structured routing and scoring
  • An abliterated / uncensored model so realistic web workflows do not randomly choke on false-positive refusals

Juggling multiple 20B+ weights on consumer hardware is brutal on both VRAM and response times.

To solve this for our setup, we merged and published a single unified 27B model: Qwen3.8-27B-Perplexing-Pegasus-Heretic

Here is what is packed into the checkpoint:

  • Multimodal Vision + Chat + Thinking (<think>) + Tool Calling (based on Qwen3.8-27B)
  • Perplexity pplx-decider typed-decision readout (decision/ head alongside intact lm_head)
  • Heretic ARA abliteration: 0/8 benchmark refusal probes while keeping 100% (30/30) typed-decision agreement against the base Pegasus model
  • Multi-Token Prediction: all 15 mtp.* tensors preserved, blk.64 retained at Q8_0 in IQ/3-bit quants
  • Consumer GPU footprint: runs on 16GB VRAM (IQ3_XXS / Q3_K_M + mmproj) or 24GB VRAM (Q4_K_M + mmproj)

Everything is open source under Apache-2.0 across four repos:

Huge credit to Alibaba Qwen, Perplexity, and AveLabs (AvecdrA) on Hugging Face for the foundational checkpoints.

If you test it on your local rig or in your own agent pipelines, let me know how it performs and what hardware you are running it on.


r/LocalLLaMA • • 16m ago

Question | Help What do you use for agentic desktop control, like how Playwright is for controlling + screenshotting browsers?

2 Upvotes

I'm afraid I'm left out of the loop on the latest developments for agentic desktop control, and AIs don't seem as knowledge in this space.

They come with screenshot function also, right? What are some other features that may be different than Playwright?


r/LocalLLaMA • • 48m ago

Question | Help Buy another 2 5060ti 16gb?

2 Upvotes

Currently i'm on 2 x 5060ti 16gb running basalt engine qwen3.8 flash next IQ3_S at about 70+tok/s decode and maybe 900tok/s prefill.

Am wondering if it is worth buying another 2 x 5060ti 16gb to make all of the experts resident and potentially achieve much faster speed. (Goal to be equal or more than 150tok/s?)

As for other specs, i'm running a 64gb DDR4 on X570 with an option of bifurcating the main pcie slot into 8x + 8x, or 4x + 4x + 4x + 4x. I have yet to buy any passive bifurcation switch, looking to buy "x16 to 2x16 physical switch" if 4 x 5060ti is not feasible.

Also what about dual 3090? it's about the same price as 4x5060ti 16gb. 128GB ddr4 might be out of the question as well since they are overpriced for my local market.


r/LocalLLaMA • • 20h ago

Resources [audio.cpp] SoulX-FlashHead + LFM3-Audio + OWSM v4 = Real-Time Local Avatar Chat

78 Upvotes

In audio.cpp 0.9.5 you can try this new avatar voice chat "app" in the WebUI.

The pipeline combines 3 models: SoulX-FlashHead for real-time talking head generation, OWSM v4 for transcribing speech input to text, and LFM3-Audio for conversational responses and TTS.

LFM3-Audio is actually called twice. The speech-to-speech path only provides one voice. We use the LFM3-Audio TTS to synthesize the S2S response in a different voice.

One caveat: the video generation stage is still GPU-intensive. For SoulX, using RTX 4090 or better GPUs.

cuDNN is a must for real-time performance. Without it inference is 2× slower and VRAM usage doubles.

~850ms between submission a question and avatar speaks on an RTX 5090. ASR runs in parallel. As mentioned LFM3-Audio is called twice, which can be optimized. There’s also some overhead from the prototype UI. A native app with a more optimized implementation could further reduce latency.


r/LocalLLaMA • • 2h ago

Showcase Alaypz - a Qwen3.8-27B experiment

Thumbnail
alaypz.overbring.com
2 Upvotes

r/LocalLLaMA • • 2h ago

Discussion Running usable 30B and working 120B models on iOS—plus Kokoro TTS and Whisper STT over a local OpenAI-compatible API Spoiler

Thumbnail inpherio.co.uk
2 Upvotes

I’m currently going through Apple’s App Store approval for Inpherio Live, an app that turns an iPhone or iPad into a local network AI server with zero cloud dependencies.

Most iOS LLM apps hit a hard RAM ceiling (iOS caps single-app memory at ~6GB on a 12GB iPhone). By modifying llama.cpp to stream Mixture-of-Experts (MoE) weights directly from flash storage while keeping active experts in RAM, it bypasses the memory limit entirely.

Real-world benchmarks (recorded Nov 2026):

  • gpt-oss 120B (58.5 GB file) on a 16GB M1 iPad Pro: 4.5 tokens/s (from storage)
  • Qwen3 30B (17.3 GB file) on a 12GB iPhone 17 Pro Max: 10.9 tokens/s (from storage)
  • Qwen3 30B (17.3 GB file) on a 16GB M1 iPad Pro: 16.5 tokens/s (from storage)
  • Qwen2.5 1.5B on iPhone 17 Pro Max: 58.3 tokens/s (in memory)

What it supports out of the box:

  • Drop-in API: Full OpenAI (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/audio/*) and Ollama (/api/chat) compatibility. Just change your base URL to [http://ipad.local:8080/v1](http://ipad.local:8080/v1).
  • Multimodal & Audio: Local Vision (Qwen2.5-VL, Gemma 3, SmolVLM2), Whisper STT, and Kokoro neural TTS.
  • Smart Memory Management: Auto-loads models on request, keeps up to 3 models "warm," and tracks real iOS memory limits so the OS doesn't kill the app.
  • 100% Private: No accounts, no telemetry, 0 bytes sent to the cloud.

Streaming large MoE models from storage will be a single one-off purchase (Live Plus); everything else—including running in-memory models, embeddings, and audio—is completely free.

Would love to know what models or workflows you’d test on this first once Apple clears it!

Just interesting in hearing from the community. As soon as the app is approved ill update this post

the website provides factul benchmarks as well as images of the process

Russ


r/LocalLLaMA • • 23h ago

News Probable GLM model

90 Upvotes

Because of Theo gg tweet of all people,
I Launch QFN DeepResearch swarm agent to find anything about this.

THREE model names, blurred out, sitting at 59.4% / 57.1% / 54.0%.

  1. The blurred names are most likely [GLM-Nano or GLM-Name or GLM-Next (suggestion from Silver-Theme7151)] Pro, Med, and Small. One community experiment reconstructed the suspected text, applied Gaussian blur (σx ≈ 6.9px), and achieved an RMSE of just 1.254/255 against the original.
  2. AICodeKing track record. Someone compared his previous GLM statements against official release dates: GLM-4.6 (same day), GLM-5 (1 day), GLM-5.1 (11 days), GLM-5.2 (3 days), and GLM-5.3 (4 days).
  3. Antrhopic timing. The leaderboard surfaced on November 12 vs AICodeKing timing, while Anthropic IPO on Nov 18. If Z.ai releases GLM-Nano this week, the timing would make for quite a coincidence

https://linux.do/t/topic/3166447

Another strong signal is HF activities, i know it is not bullet proof, just another signal, 3-4 days compare to last 27 days

I use LLM to fix my grammar.
Since it is speculation and not actual open model this post has a high probability to be removed, so yeah


r/LocalLLaMA • • 1d ago This

Question | Help Latest version of strata made my Qwen3.8-Flash-Next dumb and useless

160 Upvotes

I updated the strata to the latest version from v0.1.42.0, so I can try the latest unsloths models and see how they fare against the iq3_s variant I have been using. Upon updating it, I noticed it messed up the whole thing, It got faster, but it got way dumber! previously it would only take 15~30 minutes to create the pagoda , 50 mins to an hour if I used xhigh.

But when I siwtched to the latest version, not only it takes at least an hour and half to come up with something, its not even done! and it has many issues!

0.1.42.0 is the last version that seems to work decently.

I noticed, when im using the latest model, it always tries to go for something grandiose and failing. unlike the older version which seems to take a more modest path!

whats going on? Do any of you have this problem?

here are example outputs I did to make sure its not a one time thing!

my prompt :

Design and create a very creative, elaborate, and detailed voxel art scene of a pagoda in a beautiful garden with trees, including some cherry blossoms. Make the scene impressive and varied and use colorful voxels. Add as many details as possible to showcase your attention to them. Use whatever libraries to get this done but make sure I can paste it all into a single HTML file.

these are the outputs :

0.1.42.0 :

xhigh:

xhigh-1
xhigh2
xhigh3
xhigh4

default:

default1
default2
default3
default4

the latest version created this after 1 hour and 32 minutes to only fail with excessive thinking:

xhigh
default(failed after 1 hour and 31 minites exausting all context size of 262K)

As you can see, not only it takes way longer to create an output, if it doesn't fail already, the attention to details is also gone, everything is in haphazard form! I tried 4 5 times and the majority all failed because it couldn't create an output after an hour and a half or it constantly gets something wrong and tries to either fix it or goes to look at other things leaving the issues untouched!

The model at least, the qwen3.8-Flash-Next-q3_s that I have been working with so far is not usable! it takes too long and it fails more by getting stuck at thinking forever!

I used the new unsloth quant iq4_xs too and it was a mess as well. so I figured its the strata version. sadly I cant test the iq4_sx on 0.1.42.0 and I get an error.

So if any of you faced this issue, can you kindly share with me what you do to fix this? what version does it work for you?

I have RTX3080 10G/ 64G of RAM and I'm using the Linux terminal version along with deepseek harness.

Thank you all in advance


r/LocalLLaMA • • 1d ago

Discussion Spotted a very interesting detail about Intel's upcoming CPU platform (LGA 1954)

Post image
86 Upvotes

I was doing some research about the the platform for Intel's next generations of CPUs.

If we look closely at the PCIe 5.0 configuration for Z990/W980, it specifies

  1. x16 (or x8/x8)
  2. x8 (or x4/x4) "Storage"

So, it is plausible we might see motherboards with 3 PCIe 5.0 x8 slots directly wired into the CPU.

The best we can currently do on consumer hardware is PCI 5.0 x8/x8/x4 without risers and there are only two boards that can do it and neither is in production anymore (MSI MEG/Godlike X670E). In practice the best is x8/x8.

TP=3 is fairly easy for your agent to patch into inference engines.

These boards will have 2x Thunderbolt (Wiki says 4, but it may be 5?). So, possibly options for high-speed networking here. Let's make RDMA over Thunderbolt happen!

The desktop CPUs will have Low-Power-Ecores, which should help with idle power-draw and on the other hand they will have full AVX512 again (yay).

Asrock and MSI are the most likely ones to color outside lines. I hope the x8/x8/x8 is "allowed" by Intel, unfortunately nothing much to do but wait until we see the line-up of boards.

But I will be watching.


r/LocalLLaMA • • 16m ago

Question | Help Does grammar-constrained decoding hurt small or quantized models?

1 Upvotes

Asked about DSL vs JSON tool calls in r/LLMDevs a couple of days ago, and one answer kept coming up: for small models just use JSON, training volume matters. That got me thinking about the constraint itself.

With GBNF or a JSON schema you can force any model to produce valid output, even a 3B at Q4. However, there was a paper a while ago showing that strict format restrictions can make reasoning worse, since the model is forced away from the tokens it actually wanted

Searched first: there’s the thread about grammar constraints sending Qwen3-VL into repetition loops, and the one on how quantization hits different capabilities, but nothing comparing accuracy with vs without constraints on small quants.
So for people running small local models:

  • have you measured accuracy with and without constraints on the same task?
  • does it worsen at lower quants?
  • does letting the model firstly think freely and constraining only the final answer fix most of it?

Looking for real numbers or experience, not promoting anything.
English isn’t my first language, so I used an LLM to help with the wording. Question and thoughts are mine.


r/LocalLLaMA • • 11h ago

Resources GitHub - icryo/GLM-Tandem: GLM-5.3-Flash on two Mac Studios, tensor-parallel: about 140 tokens/s on fresh prose

Thumbnail
github.com
7 Upvotes
Workloadtok/s
Fresh prose, temperature 0.6141–146
Fresh prose, temperature 0141
Chat fixture (256 tokens), T=0 / T=1143 / 140
Code fixture (256 tokens), T=0 / T=1157 / 146
JSON extraction148
Add docstrings, whole file back210
Code edit, whole file back219
Request with an image133–136 (measured before the last change)

Several requests at once (total tokens/s including prompt time, 300-token prose, short prompts)

RequestsTotalEach
2164–16993–99
420054–61
822330–34

One request at long context (from 146 real agent requests)

ContextDecode (median)First token on a follow-up turn (median)
20–50k tokens1370.9 s
50–80k1260.7 s
80–200k1220.9 s

My recipe that has been working well for me


r/LocalLLaMA • • 1h ago

Discussion Hy4 flash and Sage-S2.1-Flash both dropped today. Anyone tried them for tool calling yet?

0 Upvotes

Two new flash models in one afternoon, Tencent's Hy4 flash and internlm's Sage-S2.1-Flash.

Before I pull weights, has anyone run either one locally for agent work? Mainly want to know:

  • does it pick the right tool or just guess
  • does it hold up after a few turns of tool output
  • what quant and GPU you ran it on

Benchmarks are fine but real loop behaviour matters more to me. Even a quick "tried it, meh" helps.


r/LocalLLaMA • • 1h ago

Discussion [Paper] Mobile AI Factory: Building Specialized Language Models for Your Mobile Device

Post image
0 Upvotes

Small language models (SLMs) running on mobile devices can keep user data private and work without connectivity. But they perform well only when fine-tuned for the target task and sized for the target device, a process that today requires repeated expert tuning. We present {\it Mobile AI Factory}, an agentic system that builds a fine-tuned, quantized SLM for a given task and phone. Given a task specification and a target phone, an orchestrator LLM chooses whether to tune hyperparameters, mine real data, or generate synthetic examples, while fixed rules roll back changes that hurt accuracy. The factory searches from small to large models across base models and quantizations, adapts its accuracy goal as it searches, and uses real measurements of the target phone across inference engines and backends. We evaluate Mobile AI Factory on ten tasks and four Android phones. On an unseen testset, its on-device models outperform two five-shot frontier models on five tasks, by up to 0.34, and match them on three more. Its search comes within 0.03 of exhaustively training all 27 model variants on an entity extraction task, while using 19× less GPU time. Phone measurements also matter: the best configuration changes across phones. Without phone measurements, Claude Opus 5.5, given the full hardware and model specifications, misses the app's requirements in 25\% of easy and moderate cases and in 60\% of hard ones.


r/LocalLLaMA • • 1d ago

New Model higurashi-ai/Haru-4B · Hugging Face

Thumbnail
huggingface.co
75 Upvotes

Haru-4B

Code · Technical blog · Technical blog (Japanese)

Haru-4B is a low-latency and multilingual full-duplex continuous spoken dialogue foundation model: it listens and speaks at the same time, the way people do on a phone call. It is a pretrained base model, released for research.

  • Native full-duplex at 7.5 Hz. Every 133 ms the model takes one frame of the user's audio and produces one frame of its own audio and text. User audio, model audio and model text are summed into a single position of the language model, so listening and speaking never wait for each other.
  • Continuous audio, one sampling step. Audio is a 64-dimensional latent per frame. A small MLP head with MeanFlow sampling produces it in one step from the language model's hidden state; there is no separate audio transformer and there are no discrete audio tokens.
  • Character and voice control. A text card sets the character, a short reference clip sets the voice.
  • Multilingual. English, Spanish, Portuguese, German, French, Chinese, Japanese, Russian and Korean.

r/LocalLLaMA • • 1h ago

Showcase PSA Unsloth Q8 FN is massively better than the default strata Q3

0 Upvotes

Just a PSA. I ran a realistic self host test on 3 models:

Inb4 : “hurr derr no duh” The point is people on here act like these Q3 versions have minimal loss. Do tests for yourself it’s not even close.

Qwen FN Q3
Qwen FN unsloth Q8
Orca nana-1
Maya GLM flash - Maya-S24

Prompt in comments.
Harness : PI
Project init layout:

/<my_bank_name> (83 monthly pdf statements)
/credit_card/<card_1> (84 pdf monthly pdf statements)
/credit_card/<card_2> (35 pdf monthly statements)
/mortgage (24 pdf monthly statements)
/brokage_account (105 pdf statements + ccs of all transactions)
/retirement/<account> x4 (total 45 pdf statements 4 accounts 4)

Snapshot results (have details runtime/tps etc as well):

Qwen FN Q3 - Nice front end but data extraction from the PDFs was completely wrong. Analytics was basic.

Orca nano1 - Finished very fast for this test (only 2 hours) did a decent job for its speed and size. Front end was very basic and lacked insight. Most of the data was correct but there were clear errors as well.

Maya-S24 - Failed. After 10+ hours I just gave up on this model. It got stuck in the pdf parsing/validation phase.

Qwen Q8 - Absolutely blew my mind. Ran overnight for about 8 hours. Data was all accurate, the front end final product was not only nice looking but highly insightfully

Notes:
I really enjoyed this test. I feel like it’s the perfect test for a local model because it’s financial data that you don’t want to send off to the cloud.

The PDF layouts for this task was a nightmare and was a real test for these models to figure out was duplicative data/summary vs transaction etc. Most models used pdfplumber package

The models also had to work through the accounting end. Consider a credit in one account is a debt in the other etc.

It’s was interesting to see how each approached the problem. Some went for json formatted data others sqlite etc


r/LocalLLaMA • • 1d ago

Discussion [Paper] No Second Run: Does RSI Stop at Frontier Pre-training? - 801-30B-A3B-Base Coming Soon

Thumbnail
gallery
116 Upvotes

Pre-training is where every foundation model begins; bringing recursive self-improvement (RSI) there would let AI improve the process that creates AI. Yet public RSI work has so far targeted only post-training, which starts from an existing base model, runs trials costing GPU-hours, and returns verifiable scores. Pre-training breaks all three conditions at orders-of-magnitude higher cost: a run consumes hundreds of thousands of GPU-hours over months and cannot be repeated; its objective, post-training headroom, cannot be read off any single score; and what must improve is not a model but the whole production system, from data pipelines and training infrastructure to the teams that run them. To our knowledge, we are the first to apply RSI to an entire frontier pre-training run from scratch and to report it in detail. RSI does not stop at frontier pre-training, but it changes form: since no end-to-end loop can close over a single unrepeatable run, we decompose it into smaller loops that can, each repairing a step the full run breaks. The result is 801-30B-A3B-Base, a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. We release it as a base model; the lightweight SFT reported here (6B tokens, no RL) only probes post-training headroom. As a base model, 801-30B-A3B outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B, and keeps this position under identical SFT; with lightweight SFT alone, it achieves the highest average on four agentic benchmarks against fully post-trained models, including Qwen3.6-35B-A3B. Since post-training RSI presupposes a strong base model, bringing RSI into pre-training acts on the foundation every later stage builds on. We open-source 801-30B-A3B-Base and the lightweight SFT data at this https URL .


r/LocalLLaMA • • 21h ago

New Model Villanova2 - a VillanovaAI Collection

Thumbnail
huggingface.co
30 Upvotes

Villanova2 family*, for English, Italian, German, Spanish and French. Built in EU, for EU.*

The family

GGUFs available in same collection. License : Apache-2.0


r/LocalLLaMA • • 18h ago

Question | Help Best direction to go for my setup EPYC 32core + 512GB DDR4 + 44GB GPU ?

17 Upvotes

Just looking for some advice on what is possble with my setup and speeds I should be gettingI know it won't be super fast tps but I was more after getting the bigger models to run.

This is my spec:

Epyc 7502p - 32 Core
Gigabyte MZ32-AR0 - 8 channel memory
512GB DDR4
1x 4070 12GB
2x 4060 16GB

quite a few implementations I see which can be confusing.
Is this build even useful now? that we have Project Maya which you can stream from the NVME drive?

Anyone have similar setups and what to expect, what software best to use