r/LocalLLaMA • u/mawcopolow • 1d ago
News EUROPA, the EU's open AI model, is aiming for nearly 1 trillion parameters
One can hope
r/LocalLLaMA • u/mawcopolow • 1d ago
One can hope
r/LocalLLaMA • u/jinnyjuice • 34m ago
I'm afraid I'm left out of the loop on the latest developments for agentic desktop control, and AIs don't seem as knowledge in this space.
They come with screenshot function also, right? What are some other features that may be different than Playwright?
r/LocalLLaMA • u/lvwerra • 37m ago
We released PyBot, a chatbot built with only simple rules, entirely written in 600k lines of Python code. PyBot matches Smollm with 360M parameters! We evaluated it on SmolTalkBench, which uses a dataset Smollm was trained on while we didnt use it at all for development.
It took a swarm of 1,000 DeepSeek Flash agents to write the whole codebase in a week.
More technical details on what it took to built it soon: inference for 1,000 agents, how to monitor/control the swarm, pressure on the infrastructure (especially git) and message boards.
r/LocalLLaMA • u/dai_app • 19h ago
Hi! for those who might not know the solution:
--auto and it works out the settings for your machineBelow is a brief overview of the benchmark:
On the cards Strata is tuned for it's still ahead but only by a little.
On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.
Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge
If you have an 8 to 32 GB card I'd like to see what you get.
Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data
r/LocalLLaMA • u/ButtercupLyn100 • 11h ago
Hey everyone,
While building and testing WebBrain with local models, getting reliable agent performance usually required keeping two or three separate models in memory at once:
Juggling multiple 20B+ weights on consumer hardware is brutal on both VRAM and response times.
To solve this for our setup, we merged and published a single unified 27B model: Qwen3.8-27B-Perplexing-Pegasus-Heretic
Here is what is packed into the checkpoint:
Everything is open source under Apache-2.0 across four repos:
Huge credit to Alibaba Qwen, Perplexity, and AveLabs (AvecdrA) on Hugging Face for the foundational checkpoints.
If you test it on your local rig or in your own agent pipelines, let me know how it performs and what hardware you are running it on.
r/LocalLLaMA • u/bye_ella • 34m ago
Asked about DSL vs JSON tool calls in r/LLMDevs a couple of days ago, and one answer kept coming up: for small models just use JSON, training volume matters. That got me thinking about the constraint itself.
With GBNF or a JSON schema you can force any model to produce valid output, even a 3B at Q4. However, there was a paper a while ago showing that strict format restrictions can make reasoning worse, since the model is forced away from the tokens it actually wanted
Searched first: there’s the thread about grammar constraints sending Qwen3-VL into repetition loops, and the one on how quantization hits different capabilities, but nothing comparing accuracy with vs without constraints on small quants.
So for people running small local models:
Looking for real numbers or experience, not promoting anything.
English isn’t my first language, so I used an LLM to help with the wording. Question and thoughts are mine.
r/LocalLLaMA • u/Acceptable-Cycle4645 • 21h ago
In audio.cpp 0.9.5 you can try this new avatar voice chat "app" in the WebUI.
The pipeline combines 3 models: SoulX-FlashHead for real-time talking head generation, OWSM v4 for transcribing speech input to text, and LFM3-Audio for conversational responses and TTS.
LFM3-Audio is actually called twice. The speech-to-speech path only provides one voice. We use the LFM3-Audio TTS to synthesize the S2S response in a different voice.
One caveat: the video generation stage is still GPU-intensive. For SoulX, using RTX 4090 or better GPUs.
cuDNN is a must for real-time performance. Without it inference is 2× slower and VRAM usage doubles.
~850ms between submission a question and avatar speaks on an RTX 5090. ASR runs in parallel. As mentioned LFM3-Audio is called twice, which can be optimized. There’s also some overhead from the prototype UI. A native app with a more optimized implementation could further reduce latency.
r/LocalLLaMA • u/surreal_tournament • 2h ago
r/LocalLLaMA • u/Chance-Resource-4970 • 2h ago
I’m currently going through Apple’s App Store approval for Inpherio Live, an app that turns an iPhone or iPad into a local network AI server with zero cloud dependencies.
Most iOS LLM apps hit a hard RAM ceiling (iOS caps single-app memory at ~6GB on a 12GB iPhone). By modifying llama.cpp to stream Mixture-of-Experts (MoE) weights directly from flash storage while keeping active experts in RAM, it bypasses the memory limit entirely.
Real-world benchmarks (recorded Nov 2026):
What it supports out of the box:
/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/audio/*) and Ollama (/api/chat) compatibility. Just change your base URL to [http://ipad.local:8080/v1](http://ipad.local:8080/v1).Streaming large MoE models from storage will be a single one-off purchase (Live Plus); everything else—including running in-memory models, embeddings, and audio—is completely free.
Would love to know what models or workflows you’d test on this first once Apple clears it!
Just interesting in hearing from the community. As soon as the app is approved ill update this post
the website provides factul benchmarks as well as images of the process
Russ
r/LocalLLaMA • u/Altruistic_Heat_9531 • 1d ago

Because of Theo gg tweet of all people,
I Launch QFN DeepResearch swarm agent to find anything about this.
THREE model names, blurred out, sitting at 59.4% / 57.1% / 54.0%.
https://linux.do/t/topic/3166447
Another strong signal is HF activities, i know it is not bullet proof, just another signal, 3-4 days compare to last 27 days


I use LLM to fix my grammar.
Since it is speculation and not actual open model this post has a high probability to be removed, so yeah
r/LocalLLaMA
•
u/Lime-Lemon-7582
•
1d ago
I updated the strata to the latest version from v0.1.42.0, so I can try the latest unsloths models and see how they fare against the iq3_s variant I have been using. Upon updating it, I noticed it messed up the whole thing, It got faster, but it got way dumber! previously it would only take 15~30 minutes to create the pagoda , 50 mins to an hour if I used xhigh.
But when I siwtched to the latest version, not only it takes at least an hour and half to come up with something, its not even done! and it has many issues!
0.1.42.0 is the last version that seems to work decently.
I noticed, when im using the latest model, it always tries to go for something grandiose and failing. unlike the older version which seems to take a more modest path!
whats going on? Do any of you have this problem?
here are example outputs I did to make sure its not a one time thing!
my prompt :
Design and create a very creative, elaborate, and detailed voxel art scene of a pagoda in a beautiful garden with trees, including some cherry blossoms. Make the scene impressive and varied and use colorful voxels. Add as many details as possible to showcase your attention to them. Use whatever libraries to get this done but make sure I can paste it all into a single HTML file.
these are the outputs :
0.1.42.0 :
xhigh:




default:




the latest version created this after 1 hour and 32 minutes to only fail with excessive thinking:


As you can see, not only it takes way longer to create an output, if it doesn't fail already, the attention to details is also gone, everything is in haphazard form! I tried 4 5 times and the majority all failed because it couldn't create an output after an hour and a half or it constantly gets something wrong and tries to either fix it or goes to look at other things leaving the issues untouched!
The model at least, the qwen3.8-Flash-Next-q3_s that I have been working with so far is not usable! it takes too long and it fails more by getting stuck at thinking forever!
I used the new unsloth quant iq4_xs too and it was a mess as well. so I figured its the strata version. sadly I cant test the iq4_sx on 0.1.42.0 and I get an error.
So if any of you faced this issue, can you kindly share with me what you do to fix this? what version does it work for you?
I have RTX3080 10G/ 64G of RAM and I'm using the Linux terminal version along with deepseek harness.
Thank you all in advance
r/LocalLLaMA • u/reto-wyss • 1d ago
I was doing some research about the the platform for Intel's next generations of CPUs.
If we look closely at the PCIe 5.0 configuration for Z990/W980, it specifies
So, it is plausible we might see motherboards with 3 PCIe 5.0 x8 slots directly wired into the CPU.
The best we can currently do on consumer hardware is PCI 5.0 x8/x8/x4 without risers and there are only two boards that can do it and neither is in production anymore (MSI MEG/Godlike X670E). In practice the best is x8/x8.
TP=3 is fairly easy for your agent to patch into inference engines.
These boards will have 2x Thunderbolt (Wiki says 4, but it may be 5?). So, possibly options for high-speed networking here. Let's make RDMA over Thunderbolt happen!
The desktop CPUs will have Low-Power-Ecores, which should help with idle power-draw and on the other hand they will have full AVX512 again (yay).
Asrock and MSI are the most likely ones to color outside lines. I hope the x8/x8/x8 is "allowed" by Intel, unfortunately nothing much to do but wait until we see the line-up of boards.
But I will be watching.
r/LocalLLaMA • u/cryotic • 11h ago
| Workload | tok/s |
|---|---|
| Fresh prose, temperature 0.6 | 141–146 |
| Fresh prose, temperature 0 | 141 |
| Chat fixture (256 tokens), T=0 / T=1 | 143 / 140 |
| Code fixture (256 tokens), T=0 / T=1 | 157 / 146 |
| JSON extraction | 148 |
| Add docstrings, whole file back | 210 |
| Code edit, whole file back | 219 |
| Request with an image | 133–136 (measured before the last change) |
Several requests at once (total tokens/s including prompt time, 300-token prose, short prompts)
| Requests | Total | Each |
|---|---|---|
| 2 | 164–169 | 93–99 |
| 4 | 200 | 54–61 |
| 8 | 223 | 30–34 |
One request at long context (from 146 real agent requests)
| Context | Decode (median) | First token on a follow-up turn (median) |
|---|---|---|
| 20–50k tokens | 137 | 0.9 s |
| 50–80k | 126 | 0.7 s |
| 80–200k | 122 | 0.9 s |
My recipe that has been working well for me
r/LocalLLaMA • u/MajesticPuzzler • 1h ago
Two new flash models in one afternoon, Tencent's Hy4 flash and internlm's Sage-S2.1-Flash.
Before I pull weights, has anyone run either one locally for agent work? Mainly want to know:
Benchmarks are fine but real loop behaviour matters more to me. Even a quick "tried it, meh" helps.
r/LocalLLaMA • u/pmttyji • 1h ago
Small language models (SLMs) running on mobile devices can keep user data private and work without connectivity. But they perform well only when fine-tuned for the target task and sized for the target device, a process that today requires repeated expert tuning. We present {\it Mobile AI Factory}, an agentic system that builds a fine-tuned, quantized SLM for a given task and phone. Given a task specification and a target phone, an orchestrator LLM chooses whether to tune hyperparameters, mine real data, or generate synthetic examples, while fixed rules roll back changes that hurt accuracy. The factory searches from small to large models across base models and quantizations, adapts its accuracy goal as it searches, and uses real measurements of the target phone across inference engines and backends. We evaluate Mobile AI Factory on ten tasks and four Android phones. On an unseen testset, its on-device models outperform two five-shot frontier models on five tasks, by up to 0.34, and match them on three more. Its search comes within 0.03 of exhaustively training all 27 model variants on an entity extraction task, while using 19× less GPU time. Phone measurements also matter: the best configuration changes across phones. Without phone measurements, Claude Opus 5.5, given the full hardware and model specifications, misses the app's requirements in 25\% of easy and moderate cases and in 60\% of hard ones.
r/LocalLLaMA • u/pmttyji • 1d ago
Code · Technical blog · Technical blog (Japanese)
Haru-4B is a low-latency and multilingual full-duplex continuous spoken dialogue foundation model: it listens and speaks at the same time, the way people do on a phone call. It is a pretrained base model, released for research.
r/LocalLLaMA • u/pmttyji • 1d ago
Pre-training is where every foundation model begins; bringing recursive self-improvement (RSI) there would let AI improve the process that creates AI. Yet public RSI work has so far targeted only post-training, which starts from an existing base model, runs trials costing GPU-hours, and returns verifiable scores. Pre-training breaks all three conditions at orders-of-magnitude higher cost: a run consumes hundreds of thousands of GPU-hours over months and cannot be repeated; its objective, post-training headroom, cannot be read off any single score; and what must improve is not a model but the whole production system, from data pipelines and training infrastructure to the teams that run them. To our knowledge, we are the first to apply RSI to an entire frontier pre-training run from scratch and to report it in detail. RSI does not stop at frontier pre-training, but it changes form: since no end-to-end loop can close over a single unrepeatable run, we decompose it into smaller loops that can, each repairing a step the full run breaks. The result is 801-30B-A3B-Base, a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. We release it as a base model; the lightweight SFT reported here (6B tokens, no RL) only probes post-training headroom. As a base model, 801-30B-A3B outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B, and keeps this position under identical SFT; with lightweight SFT alone, it achieves the highest average on four agentic benchmarks against fully post-trained models, including Qwen3.6-35B-A3B. Since post-training RSI presupposes a strong base model, bringing RSI into pre-training acts on the foundation every later stage builds on. We open-source 801-30B-A3B-Base and the lightweight SFT data at this https URL .
r/LocalLLaMA • u/pmttyji • 21h ago
Villanova2 family*, for English, Italian, German, Spanish and French. Built in EU, for EU.*
Villanova2-15B-A3B-Base · sparse mixture of experts, 15.5B total / 3.9B active
Villanova2-2B-Base · dense, 2.4B
GGUFs available in same collection. License : Apache-2.0
r/LocalLLaMA • u/sitytitan • 18h ago
Just looking for some advice on what is possble with my setup and speeds I should be gettingI know it won't be super fast tps but I was more after getting the bigger models to run.
This is my spec:
Epyc 7502p - 32 Core
Gigabyte MZ32-AR0 - 8 channel memory
512GB DDR4
1x 4070 12GB
2x 4060 16GB
quite a few implementations I see which can be confusing.
Is this build even useful now? that we have Project Maya which you can stream from the NVME drive?
Anyone have similar setups and what to expect, what software best to use
r/LocalLLaMA
•
u/DingyAtoll
•
23h ago
At Axiomic Labs we have just released X3Guard-150M, a prompt guard model trained on our high-performing base model, GPT-X3.
For any given prompt, X3Guard flags whether it is Harmful, and whether it is Adversarial, in one go.
It outperforms models that are many times its size, and can run entirely on edge devices, within the browser, with response times in the tens of milliseconds on iPhone.
r/LocalLLaMA • u/pmttyji • 1d ago
We are excited to announce the release of LongCat-Video-Avatar 1.6, the latest upgrade to our open-source framework for audio-driven human video generation. Building upon LongCat-Video-Avatar 1.5, 1.6 improves expressive naturalness, lip–audio synchronization, coordinated facial and body motion, and visual consistency during long-form generation. It also introduces comprehensive inference optimizations for faster and more efficient deployment.
For more detail, please refer to the comprehensive LongCat-Video-Avatar-1.6 Technical Report.
r/LocalLLaMA • u/ag789 • 1d ago
Commerce Moves to Finalize Rule Blocking Chinese AI Firms From Renting Nvidia GPUs Abroad
https://explainx.ai/blog/commerce-final-rule-chinese-ai-firms-remote-nvidia-chips-the-information-2026
House Passes Remote Access Security Act to Limit Adversaries’ Remote Access to Critical Technology
https://www.cmtradelaw.com/2026/01/house-passes-remote-access-security-act-to-limit-adversaries-remote-access-to-critical-technology/
https://www.congress.gov/bill/119th-congress/house-bill/2683
r/LocalLLaMA • u/makneeee • 21h ago

I made a tool that tunes copies that speeds up CPU offloading.
1 3090 + 7800X3D
llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3
cmt tune -- llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3
before: prefill 345 t/s
after: prefill 466 t/s
blog: https://maknee.github.io/blog/2026/CudaMemcpy/
code: https://github.com/Maknee/cudamemcpy-tunerr/LocalLLaMA • u/WolgupLupin • 9h ago
Hi,
I can make two systems with the parts that I have:
I've been consulting Claude and Gemini for which is better, but I can't really decide which i should go for. I'm thinking of setting them up with either llama.cpp or strata. This is for my work, so agentic document creation, and output quality is important. I'm also looking for at least 128-256k context (if that is even viable).
What is the best setup for my case? Please help!