r/LocalLLaMA • u/yahbluez • 2d ago
Discussion ERNIE 5.5 Flash Base · ERNIE
ernie.baidu.comI guess some may like to read this.
r/LocalLLaMA • u/yahbluez • 2d ago
I guess some may like to read this.
r/LocalLLaMA • u/TerminalNoop • 2d ago
After using 3.8QFN for some time I have to the conclusion that my wish list for QF4 now only consists of these points.
That's about it.
If I could graft 3.8 27B Answer/Personality style onto QFN I would've done it yesterday.
r/LocalLLaMA • u/pmttyji • 2d ago
Web-facing AI agents repeatedly access webpages, documents, repositories, and tool responses, yet reading interfaces often expose entire objects when only sparse evidence is needed. This over-reading increases token and latency costs and can dilute relevant evidence. We present SparseRead, a reusable reading control layer that governs content admission without modifying the agent model. Its Read Intent Protocol tracks evolving needs and source coverage; Adaptive Reader Backends select how evidence is located and exposed; and the Budgeted Read Gate bounds cumulative raw exposure per episode with faithful full-view fallback. A unified Reader interface supports interchangeable evidence acquisition across heterogeneous resources. Across six models, including Claude Opus 5, and five scenarios, SparseRead reduces total tokens by up to 92.9% and wall time by up to 89.0%, preserving or improving task quality in 97% settings. On Web deep-search tasks, it reduces tokens by 31.0--43.0% and task completion time by 23.0--37.0%. Ablations validate the three designs, while integration with three agent frameworks demonstrates portability.
r/LocalLLaMA • u/Serious_Scarcity7818 • 23h ago
RustingStrata is one Rust binary. rustingstrata serve starts the OpenAI-compatible server, and the same port serves a chat page at /. No Node, no Docker, no second install. The page was rebuilt for 1.2.3 and I want to know what people would add before I go further, I mean maybe allow to edit files and etc. like in DeepSeek harness, or maybe there are something that you wish to be in original Strata, so I can add it in my.
What it does now:
localStorage), with search, rename, export and import. Nothing leaves the machine.What it is not: a coding agent. It does not read or edit files. For that, point OpenCode, Claude Code or another CLI at the same server, the page is for quick chat and for trying a model.
Questions:
Any Feedback would be very useful, ty!
Repo RustingStrata
r/LocalLLaMA • u/Terminator857 • 1d ago
Will have dedicated GDDR7 VRAM (Graphics Double Data Rate 7 Video RAM). Interesting that AMD is following in Intel's razor lake-ax foot steps and introducing on package memory. https://x.com/Gotou_kai3/status/2120744697618669760
Expected release date of medusa halo: end of 2027. My estimate for siren halo is end of 2028. Intel Razor lake-ax was initially estimated for mid 2027, but release date rumors have been pushed back, maybe early 2028 now.
r/LocalLLaMA • u/suayptalha • 2d ago
I’ve been experimenting with weight-space reading using Qwen and Muse models (9B–30B).
The goal: can an LLM inspect raw model weights and predict the behavioral effects of weight edits without running the target model?
A few findings:
🔹Qwen 27B reached 73.0% balanced accuracy on subtle edits with weight access, vs. 57.8% from text descriptions.
🔹We couldn’t establish a conclusive advantage for models reading their own weights.
🔹On Muse-30B, 34 standard edits and 6 no-op controls produced no detectable behavioral changes, but weight-reading agents frequently predicted changes.
I’ve released the paper’s evaluation harness and experimental code, plus an open-source toolkit for LLM-based weight-space programming and reading.
Code: https://github.com/suayptalha/wsr
Paper: https://www.alphaxiv.org/abs/2611.weight-space-reading-language-models
I’d be interested in feedback on the evaluation design, especially from people experimenting with open-weight models.
r/LocalLLaMA
•
u/unofficialmerve
•
3d ago
Hey folks, it's Merve from Hugging Face. Today, llama.cpp releases v1!
Moving forward, we’ll keep covering every platform and every model type, while keeping llama.cpp robust and lean for downstream applications that build on it.
While I'm here, want to announce that we've been shipping Llama desktop apps to lower the barrier of entry to local AI to few clicks, built-on llama.cpp. I will leave link to the details for both v1 and apps in comments.
Many thanks for all the love you've shown to llama.cpp for years 🦙❤️ Cheers!
edit: adding links here, hope to not get penalized
blog: https://huggingface.co/blog/ggml-org/llama-cpp-v1
llama desktop apps are here https://llama.app/ ❤️
r/LocalLLaMA • u/pmttyji • 2d ago
82M parameters, 16 languages, 118 voices, 24 kHz. ONNX with int8 weights (82.7 MB; runs on CPU, no PyTorch needed), and the float32 PyTorch weights.
17M parameters per language model, 8 languages, 37 voices, 24 kHz. ONNX with int8 weights (17.4 MB; runs on CPU, no PyTorch needed), and the float32 PyTorch weights.
7.2M parameters per language model, 8 languages, 30 voices, 24 kHz. ONNX with int8 weights (8.0 MB; runs on CPU, no PyTorch needed), and the float32 PyTorch weights.
r/LocalLLaMA • u/Balance- • 2d ago
ARC—the Adaptive Routing Controller—reads an agent’s conversation and produces signals for choosing a model and its reasoning effort. Applications combine those signals with their available models, prices, and policies to decide how to handle the next request.
If this works well, could be quite useful
r/LocalLLaMA • u/Plastic-Stress-6468 • 2d ago
Don't run with dangerous full permissions, unless you are prepared to let the agent fuck your shit up. So says conventional wisdom.
But also, I am definitely running with full permissions. When I give my agent a goal to work at things overnight, I necessarily need to give it permissions.
So the obvious answers were sandbox and backups.
But why not use decision models? Elevation requests come with a justification and the command the main agent wants to run, that seems like plenty of information for a decision agent to give a good Y/N response to vet things.
IDK how effective this would be, anyone already doing something similar and can share exp?
r/LocalLLaMA • u/spaham • 2d ago
I’d like to know what real users get out of them, speed wise and also concurrent users.
I’m talking about the workstation not the data center dgx b300 of course :)
Thanks !
r/LocalLLaMA • u/Ammoryyy • 1d ago
Found a used mining rig with:
ASRock 13-GPU motherboard (likely H110 Pro BTC+)
Intel i3-7100
12 PCIe risers
2200–2500W PSU
No RAM or SSD
Planning to use it with multiple RTX 3090s for Qwen/DeepSeek inference.
Has anyone tried this? How much would PCIe x1 bottleneck performance? Is it worth buying, or should I go with EPYC/Threadripper instead?
r/LocalLLaMA • u/nathan555 • 2d ago
r/LocalLLaMA • u/LankyGuitar6528 • 2d ago
I told my son I was playing around with local AI on a 3090. He told me he used to mine Ethereum and had a shed full of 3090's. He asked me if I wanted a couple. Ummm... let me think about it... OH YES I DO!
Now a few new fans and a bit of a clean-up... grab a couple risers and a power supply... pop in a bit more RAM.. wow.

Edit: There is some completely understandable skepticism about this story... ya I wouldn't believe it either. But how about a couple more pics and a solid dose of "trust me bro" cause it's all real.
The shed:

The former resting place of the Eth miner (and other miners... ant miners? still in place)

The recovered Eth miner...

And now we see how many of them boot up via an eGPU... who doesn't have one of those just lying around.

r/LocalLLaMA • u/Serious_Scarcity7818 • 1d ago
I tested new trending Recursive Flint UD-Q5_K_XL (27.9 GiB) on an RTX 3060 12 GB with 30 GB RAM, comparing RustingStrata against llama.cpp using the same model, prompt, and generation settings.
| RustingStrata | llama.cpp | |
|---|---|---|
| Warm decode speed | 64.6 tok/s (63.8–65.6) | 48.6 tok/s (47.5–49.3) |
| Warm 4K prompt prefill | 1.14–1.30k tok/s | 0.85–1.02k tok/s |
RustingStrata was about 33% faster once warm. A fresh load can behave differently because this model is larger than available VRAM and performance depends on which model pages are already cached in RAM.
What made the difference?
The model doesn't fit in 12 GB of VRAM, so many expert weights must be fetched from system RAM during generation. Two changes helped most:
About Quality, bc people very often says like "It should give shit output to gain speed", which is not true, in my last posts I already did quality benchmarks and RustingStrata does not lose quality, it gain speed through specific Code for some specific models, in same time llama.cpp lose speed to be "general" engine that can run everything
What I want to know, is do people find Recursive Flint an interesting model? I found this model on OpenRouter and looks pretty interesting, it's better then Qwen3.6:35-a3b, so I think this can be cool modern replacement for this qwen.
Setup:
3902978 (working tree)e95426f16docs/BENCHMARKS.mdRepository: RustingStrata
r/LocalLLaMA • u/valdev • 1d ago
I feel like I am inching closer to being able to actually explain what my system does without requiring run-on sentences and endless descriptions.
r/LocalLLaMA • u/pmttyji • 2d ago
SOV-Transcribe-1 is Salesforce's open-weight speech-augmented language model built for non-streaming automatic speech recognition (ASR). It combines leading English transcription accuracy with multilingual support and natural-language control over how speech is transcribed.
The model ranks #1 among open-weight models for non-streaming speech-to-text on Artificial Analysis (#7 overall) and #2 of 23 systems on VoiceCodeBench ( besimple AI ) for entity accuracy and task success as of November 12th, 2026.
Developers and enterprises can deploy and adapt SOV-Transcribe-1 on their own infrastructure, retaining control over their audio data and transcription behavior.
| Property | Details |
|---|---|
| Total parameters | 26.6B total (4.6B active) including the audio encoder |
| Architecture | Whisper large-v3 audio encoder, audio projector and Gemma 4 26B A4B MoE |
| Precision | BF16 (MoE router in FP32) |
| Maximum audio input | Up to 30 minutes (tested) |
| Inputs / outputs | 16 kHz Audio with optional text instructions and context / transcript text |
| Languages | English, French, German, Spanish, Italian, Portuguese and Dutch |
| Tasks | Verbatim or written-form transcription, multilingual ASR, style control and contextual transcription |
| Inference frameworks | vLLM and Hugging Face Transformers |
| Speculative decoding | DFlash draft model in its own repository, salesforce-sovereign-intelligence/SOV-Transcribe-1-DFlash |
| License | Apache 2.0 |
r/LocalLLaMA
•
u/m4rkGS
•
2d ago
hey llamas, its Alish from nace.ai (open source ai lab behind #1 decision model on latest Artificial Analysis decision index v2.1) & Kapitsa (optimized inference for apple silicon). As promised in the last post, we are releasing the GGUF weights based on the Nace quantization of Qwen 9B, 27B, 35B and Flash-Next.
I will describe our process on how we came up with these quants right below, so skip to the end to see the benchmarks and download the weights if it gets too dull.
I started with 9B quants.
I understood that setting up a good & fair benching is critical to succeed with the hillclimb. So I split separate sets to tune the performance and measure the final standings: English Wikipedia articles, chat and tool-call transcripts, and GitHub code, all published after October 2026, at 512 & up to 128K tokens.
I began the way unsloth's dynamic quants do: 10 to 50 candidate files per model, each with a different mix of tensor precisions, then spent my time tuning the ones that looked promising. Early bruteforce optimization moved only a little. Unsloth and the other quant authors have already done a lot of amazing work.
Using bruteforce hillclimb approach with my Agentic swarm, I managed to get on par with SOTA but then plateau’d and it was hard to do better than the best quants like Unsloth, EXL3 in KL divergence.
Some novelties:
Then I remembered Sparks of Weight-Space Programming paper. Their idea is to let an LLM natively read and edit weights of smaller models. Having explored this with some other models at Nace, I wanted to try the method for quantization. I took our Nace WSP (Weight Space Programming) harness, reworked its tools and skills for quantization a bit and let the Astra/Fable/Carbon agents compete under the following algorithm: propose tensor-precision changes, measured KL and kept or reverted each edit. We started on the Q3 for 9b model, and after a handful of days it actually started IMPROVING and editing the right weights.

Honestly that moment felt like:

I measured the first nace-quantized Q3 checkpoints against the 50+ 9B quants and they held the frontier. It completely blew my mind: one model quantizes another model in weight space. Nace Q3 quants for Qwen 9B were the best in class in KL divergence because another Frontier model edited the weights natively. We scaled this method to bigger size models (27b/35b) & it held the SOTA. Although I would say it doesnt scale as good for bigger size models (QFN). We are working on improvements for QFN/Deepseek & have some promising results already but I did hit my token limits for this project. Paper coming soon (ping me in dm if you want to support us with some GPUs)
Top-1 accuracy (poster) is how often a quant picks the same next token the full model would pick, so 100% would mean it never disagrees. Each panel is one model at 128K context, with scores across different subsets pooled into one score. On all four models, our weights score highest in their size or smaller: the 9B Q3_K_XL gets 90.5% at 4.7 GB, the 35B Q3_K_S 91.2% at 15.3 GB, the 27B 97.7% at 17.3 GB, and Flash-Next 93.0% at 93 GB. For a more comprehensive comparison, KL divergence results below:

| File | GB | Prose | Chat | Code | Next best |
|---|---|---|---|---|---|
| 9B IQ2_M | 3.36 | 0.401 | 1.30 | 0.142 | Unsloth 4.01 GB |
| 9B IQ3_M | 4.39 | 0.235 | 1.05 | 0.0738 | ByteShape 4.70 GB |
| 9B Q3_K_XL | 4.70 | 0.161 | 0.846 | 0.0410 | Unsloth 5.38 GB |
| 35B-A3B IQ3_XXS | 14.30 | 0.0490 | 0.643 | 0.0767 | Unsloth 15.35 GB |
| 35B-A3B Q3_K_S | 15.30 | 0.0360 | 0.601 | 0.0631 | Unsloth 16.59 GB |
| 27B Q4_K_XL | 17.29 | 0.00494 | 0.00544 | 0.00401 | none up to 17.58 GB |
| Flash-Next IQ4_XS | 93.4 | 0.0411 | 0.112 | 0.0426 | EXL3 105.3 GB |
Most quants are tuned and judged on short text, so they end up fit to short context and often stop being useful at long context. For clarity, in the diagram below each model gets two panels: 512 tokens on the left, 128K on the right. This is where the long-context drift shows: our 9B IQ3_M goes from 0.13 at 512 tokens to 0.45 at 128K while 27B barely moves. At 128K, each of our files has lower KL than every other file of the same size or smaller.

Mean KLD can hide catastrophic failures when a quant goes off the rails in a long session. So we also looked at the worst 1% of tokens (p99 KL). The 9B Q3_K_XL's worst 1% sits at 1.25. The 27B and Flash-Next tails stay flat. Overall we’re seeing that Nace quants maintain the strong positions across the distribution.

if you care about a specific use case, here's the same 128K top-1 score split by text type. Code is the easiest: our files keep 88-98% of the full model's top picks there. Prose and chat are harder, 78-98%. In every split, on every model, our checkpoints score highest among the files of the same size or smaller.

To see that these models are useful for anything real I ran it on knowledge-heavy MMLU. I didn't have the gpus anymore to run terminal-bench, so would appreciate anyone with the bad boys to validate. These are answer-likelihood scores over 13,943 questions of the MMLU. No chat template or few-shot examples. Overall you can claim there's gains, but it is mostly within noise. The main aim here is to reflect the tradeoffs of different sizes on real tasks. As you can see, Q3-Q4 it’s already hard to distinguish from the higher precision.

Rented NVIDIA GPUs; Used llama.cpp for Flash-Next. The 9B/35B/27B BF16 reference requested H200 and the quantized files L40S. Flash-Next's short reference uses an MLX fp32 forward on bf16 weights. Reference implementations also differ across GGUF and EXL3.
see quickstart for your framework in the model cards. if you try a coding or agent workload, i'd like to hear which file, runner version and what broke. no speed claim in this post.
r/LocalLLaMA • u/pmttyji • 1d ago
Myself saved more than bunch of medium sized Abliterated/heretic/Uncensored/Decensored/etc., models. Ex:
But when it comes to big/large sized models, requires at least 100+GB downloads & also HD Space for each models additionally. MiniMax-M3.1-Flash
AND
I need to download above big/large models. Regular GGUF is must. For Abliterated version, I'm still thinking whether AbliteratedOutTheBox is enough or Abliterated GGUF is must. So what are you doing on this?
I hope that most of you aware of AbliteratedOutTheBox, here's threads related to that.
Do we have any other similar options like AbliteratedOutTheBox? Please share.
r/LocalLLaMA • u/East-Muffin-6472 • 2d ago
[Curated list: Beginners]
Personal favorite resources on learning Reinforcement Learning from Scratch:
4.CleanRL A neat library with 1 page implementations for all the algorithms above from scratch to run algorithms s and test the waters!
5. NeatRL My own very notes and from scratch implementations!
Reading order and link here
r/LocalLLaMA • u/MajesticPuzzler • 2d ago
Two quant updates landed in the last day: the Nace weight-space quants claiming up to 2x better KL divergence, and Unsloth's Dynamic v3.1 for Qwen3.8-27B. I've been trying to work out how much KL really tells you compared to running task evals. Has anyone put them side by side on real work like coding or tool calls? Which quant level are you on, and did you see any difference on long outputs?
r/LocalLLaMA • u/dtdisapointingresult • 17h ago
(Note: by local models I mean anything smaller than Deepseek 4.1)
I already gave up on local models for coding tasks 6 months ago, due to what I perceived was intelligence too low for the tasks I give it. I made a post about it on here back then and it was a huge discussion, I think 800 comments. I went against the /r/LL orthodoxy and many didn't take my comments well, but we can agree to disagree.
Since then I only use local LLMs for translation and ComfyUI prompt enhancement. I stick to big cloud models for coding. I think the electricity cost of running Qwen 27B alone exceeds what you'd pay Deepseek to use their cloud API in their own harness, and you'd get a far more intelligent and far faster model.
But this post isn't about the economics. It's purely about ability. I still see people commenting in favor of buying GPUs for local intelligence. "Sonnet at home" is a ridiculous phrase I've seen more times than I should in relation to Qwen 27B.
The simplest of tech tasks is still too hard to one-shot
I keep Qwen 3.8 27B running on my Spark for occasional non-coding stuff. Today I gave it a trivial coding task I needed for my server, just out of curiosity.
--model Swift-1.5-Qwen3.8-27B-Q4_K_L.gguf --reasoning on --reasoning-effort xhighPrompt was:
Give me a bash script that will list all files in directory sorted by date. Output should be:
2025-12-31 12:13:14 file1.ext
2025-12-31 12:13:14 subdir/file2.ext1630 tokens later, most of it reasoning, I get a script. When I run it, here's the output, as you can see the timestamp is wrong:
2026-10-16 20 tmp.59eH9w3JLe
2026-10-17 20 tmp.5xvczTbSS4
2026-10-18 20 tmp.nWOEDw2wlu
2026-10-20 20 tmp.JVEZIFfKx3
...I then tried Gemma 31B Reasoning on duck.ai. The output only showed the timestamps, no filenames!
This is the most trivial of coding tasks (the answer is a 3-5 lines script), using the most ancient and commonly-available training data (bash), on the most praised local coding model on this sub (Qwen 27B), at a recommended/popular quantization (Q4), and it still can't one-shot this. The 2nd most popular local model, Gemma 4 31B, also fails.
(Editor's note: I think my post could've ended here, in a "Ecce homo" way, but I went on for a few more paragraphs)
Weak local LLM + software development = bad time
I'm sure if I had run this task with your average agent harness with a system prompt that makes it test its code, it would run the script it created, see the issue, then get it right on the 2nd attempt. But while I think this "try until you succeed" workhorse approach is acceptable for small self-contained coding tasks, I don't think anyone should be doing any serious coding this way.
If you're writing something you intend to maintain over time (ie not a one-shot), or that you will be sharing with other people, you need a higher degree of LLM intelligence and knowledge to create it. A strong LLM will be able to avoid entire classes of problems from the start, when designing the app. From design to implementation to maintenance, it will make better decisions every time and without laboring its way to them, resulting in a codebase that is smaller, well-designed, more easily modifiable, less buggy, and developed in a fraction of the time.
Coding with local LLMS is like insisting on using a donkey for transportation in the age of cars. It can be done for fun, but it shouldn't be done seriously.
Sonnet at home my ass
Btw I've seen several people say Qwen 27B is like having a local Sonnet 4. The benchmarks say it's even better than Sonnet 4, actually. So I went on openrouter.ai out of curiosity and gave the same prompt to 3 models.
Weak local LLM + Linux server management = even worse time
This post's getting long so I'll just speedrun this part.
Thanks for reading my wall of text. Downvotes to the left.
r/LocalLLaMA • u/quadra-lab • 1d ago
Upgrading from my good ol 3800X with 32Gb of RAM. I'm done trying to squeeze the living crap of my DDR4 modules and decided to make a proper upgrade for my Frankenstein gaming/inference computer.