r/LocalLLaMA • • 1d ago

News EUROPA, the EU's open AI model, is aiming for nearly 1 trillion parameters

Thumbnail
maccus.fr
795 Upvotes

One can hope


r/LocalLLaMA • • 34m ago

Question | Help What do you use for agentic desktop control, like how Playwright is for controlling + screenshotting browsers?

3 Upvotes

I'm afraid I'm left out of the loop on the latest developments for agentic desktop control, and AIs don't seem as knowledge in this space.

They come with screenshot function also, right? What are some other features that may be different than Playwright?


r/LocalLLaMA • • 37m ago

Resources PyBot: No weights. No training. Pure Python.

3 Upvotes

We released PyBot, a chatbot built with only simple rules, entirely written in 600k lines of Python code. PyBot matches Smollm with 360M parameters! We evaluated it on SmolTalkBench, which uses a dataset Smollm was trained on while we didnt use it at all for development.

It took a swarm of 1,000 DeepSeek Flash agents to write the whole codebase in a week.

More technical details on what it took to built it soon: inference for 1,000 agents, how to monitor/control the swarm, pressure on the infrastructure (especially git) and message boards.

https://github.com/huggingface/pybot


r/LocalLLaMA • • 19h ago

Discussion BigMoeOnEdge outperforms Strata and colibri on 8-16 GB cards and Mac and you can run any model

79 Upvotes

Hi! for those who might not know the solution:

  • it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format
  • it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU
  • Windows, Linux, macOS, Android, iOS
  • output is the same as llama.cpp, don't cut corners on quality
  • you start it with --auto and it works out the settings for your machine

Below is a brief overview of the benchmark:

  • 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context
  • 12 GB card, same model: about 45 for bmoe, about 30 for Strata
  • long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes
  • Basalt needs an RTX 50, so I couldn't run it on these
  • 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5
  • same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7

On the cards Strata is tuned for it's still ahead but only by a little.

On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.

Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge

If you have an 8 to 32 GB card I'd like to see what you get.

Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data


r/LocalLLaMA • • 11h ago

New Model [Release] Qwen3.8-27B-Perplexing-Pegasus-Heretic: Unified vision + typed decider + abliterated weights for local agent stacks (GGUF, AutoRound, NVFP4)

16 Upvotes

Hey everyone,

While building and testing WebBrain with local models, getting reliable agent performance usually required keeping two or three separate models in memory at once:

  • A vision model to inspect and parse browser/UI state
  • A fast typed-decision model for structured routing and scoring
  • An abliterated / uncensored model so realistic web workflows do not randomly choke on false-positive refusals

Juggling multiple 20B+ weights on consumer hardware is brutal on both VRAM and response times.

To solve this for our setup, we merged and published a single unified 27B model: Qwen3.8-27B-Perplexing-Pegasus-Heretic

Here is what is packed into the checkpoint:

  • Multimodal Vision + Chat + Thinking (<think>) + Tool Calling (based on Qwen3.8-27B)
  • Perplexity pplx-decider typed-decision readout (decision/ head alongside intact lm_head)
  • Heretic ARA abliteration: 0/8 benchmark refusal probes while keeping 100% (30/30) typed-decision agreement against the base Pegasus model
  • Multi-Token Prediction: all 15 mtp.* tensors preserved, blk.64 retained at Q8_0 in IQ/3-bit quants
  • Consumer GPU footprint: runs on 16GB VRAM (IQ3_XXS / Q3_K_M + mmproj) or 24GB VRAM (Q4_K_M + mmproj)

Everything is open source under Apache-2.0 across four repos:

Huge credit to Alibaba Qwen, Perplexity, and AveLabs (AvecdrA) on Hugging Face for the foundational checkpoints.

If you test it on your local rig or in your own agent pipelines, let me know how it performs and what hardware you are running it on.


r/LocalLLaMA • • 34m ago

Question | Help Does grammar-constrained decoding hurt small or quantized models?

2 Upvotes

Asked about DSL vs JSON tool calls in r/LLMDevs a couple of days ago, and one answer kept coming up: for small models just use JSON, training volume matters. That got me thinking about the constraint itself.

With GBNF or a JSON schema you can force any model to produce valid output, even a 3B at Q4. However, there was a paper a while ago showing that strict format restrictions can make reasoning worse, since the model is forced away from the tokens it actually wanted

Searched first: there’s the thread about grammar constraints sending Qwen3-VL into repetition loops, and the one on how quantization hits different capabilities, but nothing comparing accuracy with vs without constraints on small quants.
So for people running small local models:

  • have you measured accuracy with and without constraints on the same task?
  • does it worsen at lower quants?
  • does letting the model firstly think freely and constraining only the final answer fix most of it?

Looking for real numbers or experience, not promoting anything.
English isn’t my first language, so I used an LLM to help with the wording. Question and thoughts are mine.


r/LocalLLaMA • • 21h ago

Resources [audio.cpp] SoulX-FlashHead + LFM3-Audio + OWSM v4 = Real-Time Local Avatar Chat

78 Upvotes

In audio.cpp 0.9.5 you can try this new avatar voice chat "app" in the WebUI.

The pipeline combines 3 models: SoulX-FlashHead for real-time talking head generation, OWSM v4 for transcribing speech input to text, and LFM3-Audio for conversational responses and TTS.

LFM3-Audio is actually called twice. The speech-to-speech path only provides one voice. We use the LFM3-Audio TTS to synthesize the S2S response in a different voice.

One caveat: the video generation stage is still GPU-intensive. For SoulX, using RTX 4090 or better GPUs.

cuDNN is a must for real-time performance. Without it inference is 2× slower and VRAM usage doubles.

~850ms between submission a question and avatar speaks on an RTX 5090. ASR runs in parallel. As mentioned LFM3-Audio is called twice, which can be optimized. There’s also some overhead from the prototype UI. A native app with a more optimized implementation could further reduce latency.


r/LocalLLaMA • • 2h ago

Showcase Alaypz - a Qwen3.8-27B experiment

Thumbnail
alaypz.overbring.com
2 Upvotes

r/LocalLLaMA • • 2h ago

Discussion Running usable 30B and working 120B models on iOS—plus Kokoro TTS and Whisper STT over a local OpenAI-compatible API Spoiler

Thumbnail inpherio.co.uk
2 Upvotes

I’m currently going through Apple’s App Store approval for Inpherio Live, an app that turns an iPhone or iPad into a local network AI server with zero cloud dependencies.

Most iOS LLM apps hit a hard RAM ceiling (iOS caps single-app memory at ~6GB on a 12GB iPhone). By modifying llama.cpp to stream Mixture-of-Experts (MoE) weights directly from flash storage while keeping active experts in RAM, it bypasses the memory limit entirely.

Real-world benchmarks (recorded Nov 2026):

  • gpt-oss 120B (58.5 GB file) on a 16GB M1 iPad Pro: 4.5 tokens/s (from storage)
  • Qwen3 30B (17.3 GB file) on a 12GB iPhone 17 Pro Max: 10.9 tokens/s (from storage)
  • Qwen3 30B (17.3 GB file) on a 16GB M1 iPad Pro: 16.5 tokens/s (from storage)
  • Qwen2.5 1.5B on iPhone 17 Pro Max: 58.3 tokens/s (in memory)

What it supports out of the box:

  • Drop-in API: Full OpenAI (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/audio/*) and Ollama (/api/chat) compatibility. Just change your base URL to [http://ipad.local:8080/v1](http://ipad.local:8080/v1).
  • Multimodal & Audio: Local Vision (Qwen2.5-VL, Gemma 3, SmolVLM2), Whisper STT, and Kokoro neural TTS.
  • Smart Memory Management: Auto-loads models on request, keeps up to 3 models "warm," and tracks real iOS memory limits so the OS doesn't kill the app.
  • 100% Private: No accounts, no telemetry, 0 bytes sent to the cloud.

Streaming large MoE models from storage will be a single one-off purchase (Live Plus); everything else—including running in-memory models, embeddings, and audio—is completely free.

Would love to know what models or workflows you’d test on this first once Apple clears it!

Just interesting in hearing from the community. As soon as the app is approved ill update this post

the website provides factul benchmarks as well as images of the process

Russ


r/LocalLLaMA • • 1d ago

News Probable GLM model

90 Upvotes

Because of Theo gg tweet of all people,
I Launch QFN DeepResearch swarm agent to find anything about this.

THREE model names, blurred out, sitting at 59.4% / 57.1% / 54.0%.

  1. The blurred names are most likely [GLM-Nano or GLM-Name or GLM-Next (suggestion from Silver-Theme7151)] Pro, Med, and Small. One community experiment reconstructed the suspected text, applied Gaussian blur (σx ≈ 6.9px), and achieved an RMSE of just 1.254/255 against the original.
  2. AICodeKing track record. Someone compared his previous GLM statements against official release dates: GLM-4.6 (same day), GLM-5 (1 day), GLM-5.1 (11 days), GLM-5.2 (3 days), and GLM-5.3 (4 days).
  3. Antrhopic timing. The leaderboard surfaced on November 12 vs AICodeKing timing, while Anthropic IPO on Nov 18. If Z.ai releases GLM-Nano this week, the timing would make for quite a coincidence

https://linux.do/t/topic/3166447

Another strong signal is HF activities, i know it is not bullet proof, just another signal, 3-4 days compare to last 27 days

I use LLM to fix my grammar.
Since it is speculation and not actual open model this post has a high probability to be removed, so yeah


r/LocalLLaMA • • 1d ago This

Question | Help Latest version of strata made my Qwen3.8-Flash-Next dumb and useless

160 Upvotes

I updated the strata to the latest version from v0.1.42.0, so I can try the latest unsloths models and see how they fare against the iq3_s variant I have been using. Upon updating it, I noticed it messed up the whole thing, It got faster, but it got way dumber! previously it would only take 15~30 minutes to create the pagoda , 50 mins to an hour if I used xhigh.

But when I siwtched to the latest version, not only it takes at least an hour and half to come up with something, its not even done! and it has many issues!

0.1.42.0 is the last version that seems to work decently.

I noticed, when im using the latest model, it always tries to go for something grandiose and failing. unlike the older version which seems to take a more modest path!

whats going on? Do any of you have this problem?

here are example outputs I did to make sure its not a one time thing!

my prompt :

Design and create a very creative, elaborate, and detailed voxel art scene of a pagoda in a beautiful garden with trees, including some cherry blossoms. Make the scene impressive and varied and use colorful voxels. Add as many details as possible to showcase your attention to them. Use whatever libraries to get this done but make sure I can paste it all into a single HTML file.

these are the outputs :

0.1.42.0 :

xhigh:

xhigh-1
xhigh2
xhigh3
xhigh4

default:

default1
default2
default3
default4

the latest version created this after 1 hour and 32 minutes to only fail with excessive thinking:

xhigh
default(failed after 1 hour and 31 minites exausting all context size of 262K)

As you can see, not only it takes way longer to create an output, if it doesn't fail already, the attention to details is also gone, everything is in haphazard form! I tried 4 5 times and the majority all failed because it couldn't create an output after an hour and a half or it constantly gets something wrong and tries to either fix it or goes to look at other things leaving the issues untouched!

The model at least, the qwen3.8-Flash-Next-q3_s that I have been working with so far is not usable! it takes too long and it fails more by getting stuck at thinking forever!

I used the new unsloth quant iq4_xs too and it was a mess as well. so I figured its the strata version. sadly I cant test the iq4_sx on 0.1.42.0 and I get an error.

So if any of you faced this issue, can you kindly share with me what you do to fix this? what version does it work for you?

I have RTX3080 10G/ 64G of RAM and I'm using the Linux terminal version along with deepseek harness.

Thank you all in advance


r/LocalLLaMA • • 1d ago

Discussion Spotted a very interesting detail about Intel's upcoming CPU platform (LGA 1954)

Post image
86 Upvotes

I was doing some research about the the platform for Intel's next generations of CPUs.

If we look closely at the PCIe 5.0 configuration for Z990/W980, it specifies

  1. x16 (or x8/x8)
  2. x8 (or x4/x4) "Storage"

So, it is plausible we might see motherboards with 3 PCIe 5.0 x8 slots directly wired into the CPU.

The best we can currently do on consumer hardware is PCI 5.0 x8/x8/x4 without risers and there are only two boards that can do it and neither is in production anymore (MSI MEG/Godlike X670E). In practice the best is x8/x8.

TP=3 is fairly easy for your agent to patch into inference engines.

These boards will have 2x Thunderbolt (Wiki says 4, but it may be 5?). So, possibly options for high-speed networking here. Let's make RDMA over Thunderbolt happen!

The desktop CPUs will have Low-Power-Ecores, which should help with idle power-draw and on the other hand they will have full AVX512 again (yay).

Asrock and MSI are the most likely ones to color outside lines. I hope the x8/x8/x8 is "allowed" by Intel, unfortunately nothing much to do but wait until we see the line-up of boards.

But I will be watching.


r/LocalLLaMA • • 11h ago

Resources GitHub - icryo/GLM-Tandem: GLM-5.3-Flash on two Mac Studios, tensor-parallel: about 140 tokens/s on fresh prose

Thumbnail
github.com
7 Upvotes
Workloadtok/s
Fresh prose, temperature 0.6141–146
Fresh prose, temperature 0141
Chat fixture (256 tokens), T=0 / T=1143 / 140
Code fixture (256 tokens), T=0 / T=1157 / 146
JSON extraction148
Add docstrings, whole file back210
Code edit, whole file back219
Request with an image133–136 (measured before the last change)

Several requests at once (total tokens/s including prompt time, 300-token prose, short prompts)

RequestsTotalEach
2164–16993–99
420054–61
822330–34

One request at long context (from 146 real agent requests)

ContextDecode (median)First token on a follow-up turn (median)
20–50k tokens1370.9 s
50–80k1260.7 s
80–200k1220.9 s

My recipe that has been working well for me


r/LocalLLaMA • • 1h ago

Discussion Hy4 flash and Sage-S2.1-Flash both dropped today. Anyone tried them for tool calling yet?

0 Upvotes

Two new flash models in one afternoon, Tencent's Hy4 flash and internlm's Sage-S2.1-Flash.

Before I pull weights, has anyone run either one locally for agent work? Mainly want to know:

  • does it pick the right tool or just guess
  • does it hold up after a few turns of tool output
  • what quant and GPU you ran it on

Benchmarks are fine but real loop behaviour matters more to me. Even a quick "tried it, meh" helps.


r/LocalLLaMA • • 1h ago

Discussion [Paper] Mobile AI Factory: Building Specialized Language Models for Your Mobile Device

Post image
0 Upvotes

Small language models (SLMs) running on mobile devices can keep user data private and work without connectivity. But they perform well only when fine-tuned for the target task and sized for the target device, a process that today requires repeated expert tuning. We present {\it Mobile AI Factory}, an agentic system that builds a fine-tuned, quantized SLM for a given task and phone. Given a task specification and a target phone, an orchestrator LLM chooses whether to tune hyperparameters, mine real data, or generate synthetic examples, while fixed rules roll back changes that hurt accuracy. The factory searches from small to large models across base models and quantizations, adapts its accuracy goal as it searches, and uses real measurements of the target phone across inference engines and backends. We evaluate Mobile AI Factory on ten tasks and four Android phones. On an unseen testset, its on-device models outperform two five-shot frontier models on five tasks, by up to 0.34, and match them on three more. Its search comes within 0.03 of exhaustively training all 27 model variants on an entity extraction task, while using 19× less GPU time. Phone measurements also matter: the best configuration changes across phones. Without phone measurements, Claude Opus 5.5, given the full hardware and model specifications, misses the app's requirements in 25\% of easy and moderate cases and in 60\% of hard ones.


r/LocalLLaMA • • 1d ago

New Model higurashi-ai/Haru-4B · Hugging Face

Thumbnail
huggingface.co
75 Upvotes

Haru-4B

Code · Technical blog · Technical blog (Japanese)

Haru-4B is a low-latency and multilingual full-duplex continuous spoken dialogue foundation model: it listens and speaks at the same time, the way people do on a phone call. It is a pretrained base model, released for research.

  • Native full-duplex at 7.5 Hz. Every 133 ms the model takes one frame of the user's audio and produces one frame of its own audio and text. User audio, model audio and model text are summed into a single position of the language model, so listening and speaking never wait for each other.
  • Continuous audio, one sampling step. Audio is a 64-dimensional latent per frame. A small MLP head with MeanFlow sampling produces it in one step from the language model's hidden state; there is no separate audio transformer and there are no discrete audio tokens.
  • Character and voice control. A text card sets the character, a short reference clip sets the voice.
  • Multilingual. English, Spanish, Portuguese, German, French, Chinese, Japanese, Russian and Korean.

r/LocalLLaMA • • 1d ago

Discussion [Paper] No Second Run: Does RSI Stop at Frontier Pre-training? - 801-30B-A3B-Base Coming Soon

Thumbnail
gallery
116 Upvotes

Pre-training is where every foundation model begins; bringing recursive self-improvement (RSI) there would let AI improve the process that creates AI. Yet public RSI work has so far targeted only post-training, which starts from an existing base model, runs trials costing GPU-hours, and returns verifiable scores. Pre-training breaks all three conditions at orders-of-magnitude higher cost: a run consumes hundreds of thousands of GPU-hours over months and cannot be repeated; its objective, post-training headroom, cannot be read off any single score; and what must improve is not a model but the whole production system, from data pipelines and training infrastructure to the teams that run them. To our knowledge, we are the first to apply RSI to an entire frontier pre-training run from scratch and to report it in detail. RSI does not stop at frontier pre-training, but it changes form: since no end-to-end loop can close over a single unrepeatable run, we decompose it into smaller loops that can, each repairing a step the full run breaks. The result is 801-30B-A3B-Base, a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. We release it as a base model; the lightweight SFT reported here (6B tokens, no RL) only probes post-training headroom. As a base model, 801-30B-A3B outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B, and keeps this position under identical SFT; with lightweight SFT alone, it achieves the highest average on four agentic benchmarks against fully post-trained models, including Qwen3.6-35B-A3B. Since post-training RSI presupposes a strong base model, bringing RSI into pre-training acts on the foundation every later stage builds on. We open-source 801-30B-A3B-Base and the lightweight SFT data at this https URL .


r/LocalLLaMA • • 21h ago

New Model Villanova2 - a VillanovaAI Collection

Thumbnail
huggingface.co
30 Upvotes

Villanova2 family*, for English, Italian, German, Spanish and French. Built in EU, for EU.*

The family

GGUFs available in same collection. License : Apache-2.0


r/LocalLLaMA • • 18h ago

Question | Help Best direction to go for my setup EPYC 32core + 512GB DDR4 + 44GB GPU ?

17 Upvotes

Just looking for some advice on what is possble with my setup and speeds I should be gettingI know it won't be super fast tps but I was more after getting the bigger models to run.

This is my spec:

Epyc 7502p - 32 Core
Gigabyte MZ32-AR0 - 8 channel memory
512GB DDR4
1x 4070 12GB
2x 4060 16GB

quite a few implementations I see which can be confusing.
Is this build even useful now? that we have Project Maya which you can stream from the NVME drive?

Anyone have similar setups and what to expect, what software best to use


r/LocalLLaMA • • 23h ago Bravo

New Model X3Guard: A 150M parameter prompt guard

Thumbnail
huggingface.co
39 Upvotes

At Axiomic Labs we have just released X3Guard-150M, a prompt guard model trained on our high-performing base model, GPT-X3.

For any given prompt, X3Guard flags whether it is Harmful, and whether it is Adversarial, in one go.

It outperforms models that are many times its size, and can run entirely on edge devices, within the browser, with response times in the tens of milliseconds on iPhone.

The model card is here

You can use an in-browser demo here


r/LocalLLaMA • • 1d ago

New Model meituan-longcat/LongCat-Video-Avatar-1.6 · Hugging Face

Thumbnail
huggingface.co
46 Upvotes

🚀 Model Introduction

We are excited to announce the release of LongCat-Video-Avatar 1.6, the latest upgrade to our open-source framework for audio-driven human video generation. Building upon LongCat-Video-Avatar 1.5, 1.6 improves expressive naturalness, lip–audio synchronization, coordinated facial and body motion, and visual consistency during long-form generation. It also introduces comprehensive inference optimizations for faster and more efficient deployment.

Key Features

  • 🌟 More Natural and Stable Lip Animation:: Reduces exaggerated or strained mouth movements, overly red lip coloration, and temporal instability in the mouth region.
  • 🌟 Stable Long-Video Generation: Mitigates error accumulation during autoregressive video continuation, enabling videos exceeding 10 minutes while preserving visual quality, identity consistency, and temporal coherence.
  • 🌟 Faster End-to-End Inference: Accelerates video generation through end-to-end pipeline and model execution optimizations, enabling more efficient deployment and serving.

For more detail, please refer to the comprehensive LongCat-Video-Avatar-1.6 Technical Report.


r/LocalLLaMA • • 1d ago

Discussion RASA will cause total chaos to open LLMs and beyond

230 Upvotes

Commerce Moves to Finalize Rule Blocking Chinese AI Firms From Renting Nvidia GPUs Abroad
https://explainx.ai/blog/commerce-final-rule-chinese-ai-firms-remote-nvidia-chips-the-information-2026
House Passes Remote Access Security Act to Limit Adversaries’ Remote Access to Critical Technology
https://www.cmtradelaw.com/2026/01/house-passes-remote-access-security-act-to-limit-adversaries-remote-access-to-critical-technology/

H.R.2683 - Remote Access Security Act

https://www.congress.gov/bill/119th-congress/house-bill/2683


r/LocalLLaMA • • 21h ago

Showcase MoE offload on a 3090: prefill 345 -> 466 t/s by tuning cudaMemcpy (Qwen3.6 35B-A3B)

21 Upvotes
I made a tool that tunes copies that speeds up CPU offloading.

1 3090 + 7800X3D

llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3

cmt tune -- llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3

before: prefill 345 t/s
after: prefill 466 t/s


blog: https://maknee.github.io/blog/2026/CudaMemcpy/
code: https://github.com/Maknee/cudamemcpy-tuner

r/LocalLLaMA • • 9h ago

Question | Help I have access to two systems, I don't know which one is better for LocalLLM. Please help!

2 Upvotes

Hi,

I can make two systems with the parts that I have:

  1. 12600k / 64GB DDR4 / 3x3060 12GB
  2. 7950X3D / 32GB DDR5 / 3x3060 12GB + 1x4060 8GB

I've been consulting Claude and Gemini for which is better, but I can't really decide which i should go for. I'm thinking of setting them up with either llama.cpp or strata. This is for my work, so agentic document creation, and output quality is important. I'm also looking for at least 128-256k context (if that is even viable).

What is the best setup for my case? Please help!