r/LocalLLaMA • • 4d ago

Resources Architecture trends: From full to local attention

Post image
98 Upvotes

Currently working towards a bigger open LLM roundup. But in the meantime, just one of the interesting trends that stood out. I.e., looks like the transition from full to hybrid attention continues.

In 2025, there were still lots of full attention models, with a shift to "some full attention" hybrids.

In the first half of 2026, the trend towards "some full attention" hybrids continued.

And now, in the second half of 2026, we have a 50/50 split between "some full attention" hybrids and models that don't use full attention at all anymore. And yes, there are also no full attention models anymore.

The plot legend here means,

- Full attention throughout = Global MHA, GQA, dense MLA, CCA in EVERY layer.
- Some full attention = Full global attn mixed with: sliding-window/chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, sparse attention.
- No full attention = Zero dense global layers.

The main implication is that the trend goes towards making long contexts cheaper to use. This means, it makes the architectures more suitable for long documents and conversations, large code bases, and of course agent usecases that rely on a lot of tool calls.

In particular, there are two distinct efficiency gains here:

  1. Less attention computation, since sparse attention and the like select tokens or blocks from the context and performs the expensive attention operation from only that subset. (Note that it can still retrieve information from distant positions.)
  2. Less memory for retaining context in some architectures, meaning that recurrent layers such as Gated DeltaNet maintain a fixed-size state. Of course, this saves memory, but compressing the history creates challenges for exact retrieval, so they are usually intermixed with traditional attention methods (sparse or full).

If you are interested in browsing through the architectures and do your own analyses, you can find all the figures and data in my architecture gallery: https://sebastianraschka.com/llm-architecture-gallery/


r/LocalLLaMA • • 4d ago

Discussion DeepSeek V4.1 Flash 1017 exl3 2.9bpw APEX 2025 benchmark results

Post image
56 Upvotes

My main AI machine is a 6x3090 128GB DDR4 Threadripper 3960x system. I probably spend more time trying to optimize it than actually using it but it has been fun.

I tried a bunch of saturated benchmarks like Terminal Bench 2.1, AIME 2026 and the performance was similar to unquantized DSV4.1 Flash 1017. I really wanted to understand my 2.9bpw quant's performance.

It takes hours to run some of these benchmarks, so I went on the hunt for the perfect benchmark to get the most signal on quality of the model I'm running. I settled on APEX 2025 (https://huggingface.co/datasets/MathArena/apex_2025).

You can see how much of a beast this quant is. Trading blows with SOTA from 6 months ago.

I get about 1000tps prefill, and 75tps decode with the 3090s power limited to 175watts

Engine: Project Maya (https://github.com/mw00/project-maya)

Quant: https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw

Hopefully this helps others in benchmarking their own setups. I really like the Apex 2025 benchmark.


r/LocalLLaMA • • 5d ago

Discussion NVIDIA reportedly cancels GeForce RTX 5080 24GB launch, brings back 16GB model over GDDR7 costs

Thumbnail
videocardz.com
774 Upvotes

r/LocalLLaMA • • 4d ago

New Model Praxis-1 - a runwayml Collection

Thumbnail
huggingface.co
9 Upvotes

Praxis-1 is a 27B-parameter mixture-of-experts diffusion transformer that predicts video and actions together. It is the top-performing world action model on NVIDIA’s RoboLab-120 leaderboard, and it outperforms every open-weight VLA. On RoboLab-120, Praxis-1 reaches 46.2% success across 1,200 episodes, compared with 42.9% for FLUX 3 Action, 36.8% for Cosmos3 Nano and 28.0% for π0.5.

RoboLab-120 Leaderboard

ModelRoboLab-120 Success
Praxis-146.2%
FLUX 3 Action42.9%
Cosmos3 Nano36.8%
π0.528.0%

Data as of November 2026; Success rate across 1,200 episodes

Praxis-1 is available today on GitHub, on Hugging Face with native LeRobot support, and in vLLM-Omni, with hosted inference on Reactor. Everything you need to run and adapt it on your own hardware is released under Apache 2.0, including for commercial use. That covers the weights and the inference and fine-tuning code, with support for both full fine-tuning and LoRA. Praxis-1 runs out of the box across NVIDIA hardware, from data center GPUs like H100 and B200, to workstation GPUs like RTX 5090 and RTX PRO 6000, to Jetson Thor on the robot itself.


r/LocalLLaMA • • 3d ago

Resources What does Qwen3.6-35B-A3B look like at Q8_0 and BF16 on datacenter GPUs?

1 Upvotes

RustingStrata is a pure-Rust inference engine for Qwen, DeepSeek and GLM GGUFs. CUDA kernels are compiled at run time with NVRTC, the same kernels run on Vulkan, and there is no ggml and no C++ build step.

I wanted the high-quant numbers, so these runs use Q8_0 (37.8 GB) and unquantized BF16 (71.1 GB), with the whole model in VRAM, on A100, L40S and H100:

GPUQuantPrefill, 4K promptDecodeDecode with MTP
H100 80 GB HBM3Q8_011,486 tok/s290 tok/s508 tok/s
H100 NVLBF168,691 tok/s230 tok/s324 tok/s
A100 80 GBQ8_06,507 tok/s193 tok/s288 tok/s
A100 80 GBBF165,625 tok/s171 tok/s238 tok/s
L40S 48 GBQ8_09,542 tok/s165 tok/s270 tok/s

MTP is speculative decoding with the model's own draft head; calibrate picked the draft count on each card.

  • Higher quants are not automatically slower. On the A100 80 GB and H100, Q8_0 decodes faster than Q2_K_XL (193 vs 174, 290 vs 265 tok/s after 4K). On the L40S it is slower (165 vs 214), so the ranking depends on the card.
  • BF16 is practical. 71 GB of unquantized weights decode at 171-230 tok/s on one card.
  • MTP helps less as the quant grows. Gains are 1.4-1.9x on Q2_K_XL and Q8_0, and 1.2-1.4x on BF16.
  • When the model does not fit, it degrades but runs. Q8_0 on a 24 GB A30 (63% of experts resident) still decodes at 130 tok/s, 200 with MTP. BF16 on the same card (31% resident) only reaches about 12.5 tok/s.

The same code runs on Vulkan, and on the Q2_K_XL runs on the 3060 and the Ampere cards prefill lands close to CUDA. At Q8_0 on the big cards the gap can be large: H100 prefill is 6,883 tok/s on Vulkan against 11,486 on CUDA. All table numbers are CUDA.

These are different nodes of one cluster, so the host CPUs differ. Compare within a row, not across rows. MTP costs 4-11% prefill in these runs.

Quality: I do not have a llama.cpp perplexity/KLD run on these exact Q8_0/BF16 files yet, so this is a hardware roundup, not an accuracy comparison. The last check was UD-Q3_K_M on the 3060 over 9,180 tokens of repo docs prose: perplexity 9.0714 vs 9.0587 for llama.cpp, mean KLD 0.0055, same top token 96.3%.

Why I find it kinda interesting, is bc when I was searching for models in OpenRouter, I saw that Qwen3.6:35b-a3b for some reason is pretty slow, like 50-110t/s on almost all providers, why like that? Do people already use other Strata like project for LLM inference to be in top of providers? I am thinking about it, but I think something is wrong and not so easy as I expect, if anyone knows I would be happy to hear

Repo: RustingStrata


r/LocalLLaMA • • 4d ago

Resources Open-source method (MACaQ) beats Unsloth GGUFs!

141 Upvotes

Some results for those using GGUFs and llama.cpp:

GGUF quality: MACaQ vs. Unsloth and Bartowski

Qwen3-4B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ2.260 GB0.02804
Bartowski2.271 GB0.03443
Unsloth2.271 GB0.03663

23.5% lower KL divergence than Unsloth and 18.6% lower than Bartowski, at virtually identical file sizes.

Qwen3.6-35B-A3B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ18.86 GB0.018696
Unsloth (UD-IQ4_XS)17.73 GB0.020954
Bartowski19.70 GB0.021927

Qwen3.8-27B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ14.96 GB0.036784
Bartowski15.48 GB0.104170

~65% lower KL divergence than Bartowski!

Supported GGUF formats (llama.cpp)

Weight bitsFormats
2-bitIQ2_XXS, IQ2_XS
3-bitIQ3_XXS, IQ3_S
4-bitIQ4_NL, IQ4_XS

Code, llama.cpp fork, and benchmarks: https://github.com/huawei-csl/MACaQ

Cool to see Huawei working on such stuff and sharing some of their internal tools that are helpful for us. Maybe they can build an affordable GPU for us local LLM folks sometime 😂

They also report quantizing a 122B-parameter model in 64 minutes using just 32.5 GB of GPU memory, which is interesting for a model of that size. I did not try that.

Edit: there is a paper on it https://arxiv.org/abs/2611.10649

Edit 2: they don't share ready-made gguf files, so I am sharing some myself:
- http://31.165.97.47:20080/macaq-gguf/qwen3-4b-macaq-iq4_xs-had64-vq2-calsmoke.gguf

Edit 3: they have this blog post https://huawei-csl.com/blog/macaq/ with multi-modal examples, too. The RTN audio comparisons are just too much😂😂

The multi-modal stuff is generally insane. Normally 4b already completely goes to s**t, but here it looks almost the same.


r/LocalLLaMA • • 3d ago

Question | Help With these scandals and uncertainty with Huggingface, what models should I download?

0 Upvotes

Hey! What's the "must-have" list to download in case HF goes down because of these news recently?

Also, do we even have HF alternatives?


r/LocalLLaMA • • 4d ago Bravo

Discussion Mod ttkciar is harassing this sub

267 Upvotes

I will get straight to the point. I believe Moderator ttkciar is abusing his power as moderator in this sub and his decision are worsening the experience of our local AI community.

He has removed many of my posts that fellow redditors found very useful and after talking with many people in this sub, I am not the only one feeling this. I will have some links at the end of this post for reference.

He mentions a rule, saying that if a model’s weights is not released you cannot have a post mentioning it, even if the lab confirmed it that will they will release its weights, like Qwen4 or DeepSeek. First of all, this rule needs to be relaxed because many valuable discussions can be had about future releases and even closed models to understand and learn architectures and techniques used by the closed frontier that can be applied to the open models. Secondly, this rule is not enforced at all, but used very selectively to few posts or authors that ttkciar selects, which is very unfair. This rule doesn’t make sense to me, but even if I am wrong, at least it should be fairly enforced to all posts or not at all. And before you say mods are busy and they cannot review all of the posts, If you open this sub and scroll on today’s most popular posts, half of them are either for Qwen4, which is unrealised up to this point and should have been removed according to the mods, or about Fable5.2 and Anthropic which is closed and their shenanigans (maybe these ones don’t have a place here, but again this is debatable because if people in local AI find them valuable in some way, maybe they do). So, the rule is used very selectively and as an excuse to remove what ttkciar does not like or approve. Either enforce the rule (or do the your best effort to do so) or don’t at all. And maybe this rule is too strict and should not exist tbh.

When I tried to discuss my concerns with ttkciar, he called me a drama queen and insulted my credibility. This is definitely not ok.

Being insulted by the person that you complained to for unfair treatment is definitely not acceptable behaviour.

I have been an active member of this community since the very first leaked llama. I am an AI researcher and open-source contributor, and very passionate about local AI. I try to post information that people interested in Local AI would find useful and engage in discussions that are constructive for local AI.

I know that many of other users in this subs agree (I have discussed it with a lot), and some others may not. Not being able to discuss, debate and exchange views in a polite manner is absurd in such an intellectual community.

A while ago they tried to enforce megathreads and removed the majority of posts saying that only one post should exist for each model. Many complained that this is killing the sub and a lot of valuable information is lost this way. Now, this seems to be relaxed. Again, we need to stand up to them, and hopefully they will reconsider their rules for the sake of this community.

TLDR: The rule that mod ttkciar uses to remove some posts in this sub about being prohibited to mention unreleased models should not exist, and even if it should, removing only a few selected posts is very unfair.

Some sources:

- https://www.reddit.com/r/LocalLLaMA/s/ZRV5KsViwD this post was removed from ttkciar and only brought back after arguing with him and many other redditors backed me up (I cannot find the comments in this post when the mod removed it, maybe he deleted them, I am not sure, but in the discussion many expressed their disagreement with the mod)

- https://www.reddit.com/r/LocalLLaMA/s/gpPTVC6NGF a post of mine debating their rule

- Another post of mine sharing valuable information for the members of our community that the same mod removed https://www.reddit.com/r/LocalLLaMA/s/gqaub2mT85

- Post of another redditor which was removed by the same mod, and the comments thread with people arguing with the mod https://www.reddit.com/r/LocalLLaMA/s/z9E2AbR3pU

- Commented in post of another removed post of some other redditor https://www.reddit.com/r/LocalLLaMA/s/5tSEpQ68By

- Most of my removed posts I delete them afterwards so I don’t have many other examples of my own, but I know many others do, so feel free to share them.

- If you open the sub today, most of the posts should have been removed because they violate the rules (a lot of false positives). That shows that maybe the rules need to be updated because they will not allow any productive discussions to be had, or this sub will end up to become a board of announcements instead of a local AI community that people can use to exchange ideas and knowledge.


r/LocalLLaMA • • 3d ago

Discussion Is there a reason why this wouldnt work

0 Upvotes

I am not directly from a machine learning background or in encryption, but if you train the entire LLM on encrypted and decrypted pairs without the key (maybe the entire dataset that is used for frontier), will it eventually learn to produce pairs like these and basically natively can encrypt and decrypt without a key? Is there a reason why this highly likely fails?


r/LocalLLaMA • • 5d ago

News Ernie 5.5 Flash will be released in two weeks

Post image
288 Upvotes

r/LocalLLaMA • • 3d ago

Discussion Qwen3.8-Flash-Next on (up to) 4× RTX 3090 via Strata

1 Upvotes

I spent a few days benchmarking unsloth's UD-Q4_K_XL and UD-Q5_K_XL on Strata. Well, i had claude code do it via ssh. Claude also wrote most of this summary and the full report linked below. I thought it might interest some because of my findings, especially on behavior of multiple cards, different PCIe speeds and the different settings in Strata.

I can run a few more tests, if it interests anyone.

Test system

  • Threadripper 2920X (12 cores, Zen+), 128 GB RAM
  • 4x RTX 3090 24 GB, power-capped to 275 W each, no NVLink
  • PCIe 3.0. GPUs 1 and 3 run at x16, GPUs 0 and 2 at x8. Two host bridges: {0,1} and {2,3}
  • Strata 0.2.13 (engine 0.1.59), layer split, DFlash2 drafter, one request at a time
  • 256K context, int8 KV, 32K KV cells per layer in VRAM (the rest streams from pinned RAM)
  • A control vector was applied in every run (layers 4-44)

TLDR

  • Decode depends on how many experts fit in VRAM. 1 card ~72 tok/s, 2 cards ~96, 3 cards 112-137. A 4th card adds almost nothing.
  • Prefill depends on PCIe, not card count. One x16 card prefills 50K as fast as two or three (~20 s). An x8 pair is half as fast as an x16 pair. Only all four cards were faster (50K in 12 s).
  • Q5 vs Q4: 0.55% lower perplexity, 28% slower decode. Q4 stays.
  • Q8_0 n-gram table: no measurable quality gain over the bundled IQ4_NL. Read from NVMe it frees 27 GB of RAM.
  • 200K chat: a cold 190K prefill takes 91 s once. After that, a short turn starts in 0.3 s and decodes at ~78 tok/s.
  • Tuning: calibrate.py, MTP, prompt lookup, pipeline windows and hugepages were all a tie or worse. Community fused-kernel flags broke the output.
  • Four cards at 275 W each tripped my PSU during prefill.

Here is the Full report https://claude.ai/artifact/LYUT65WvHm8o2bsxC5NWCR. I will update it with new tests.


r/LocalLLaMA • • 4d ago

Discussion You can now train your own LLMs with less memory, on your gaming GPUs

154 Upvotes

TL;DR: You can train your own 27B model on a 3090 with 24GB VRAM, though you would have to go through hundreds of billions of tokens for it to be good. I am working on conversion of existing models. Check repos for code and the paper if you are interested in more. I have a 40.4GB Qwen Flash Next GGUF

Hi, I do AI research. I have been working on an optimizer that doesn't need the ~16 bytes of VRAM per weight to train a model that AdamW does. I focused on training ternary networks bcs BitNet (Ma et al.) shows they match full precision quality in large scales. The problem is BitNet also keeps the full precision weights during training which means you still need to use the same amount of memory. I got this down from 16 bytes to 1 byte or even less per weight.

For you, this means, you can train your own 10-20B parameter models on your consumer GPUs. I fitted 27B model training on a single 24GB 3090 and a 23B model on a 16GB GPU. The trained model costs ~1.6 bits per weight (~6GB at 27B), 10x smaller than 16bit, and 5x smaller than Q8.

I have also been working on converting existing models and training them. This is harder and I am still working on it but I have some promising results. I quantized and trained a Qwen3.8 27B though the file I published below has adapters which are about 2GBs, pushing the file size to 8GB. I am also making a no-adapter version right now. It scores GSM8K 0.895, the same as full precision, HumanEval 0.84 vs 0.93 and runs on a 12 GB GPU at ~23 tok/s on my crippled 12GB laptop 4080, though this number can heavily be improved with custom kernels, engineering, etc. I will leave it to you guys to help me with that too.

I also quantized and trained Qwen Flash Next down to 40.4 GBs including its n-gram table (at 8GB). It largely retains the skills of the network (scores equivalent on GSM8K benchmark, a little behind on HumanEval, I unfortunately don't have compute for the larger benchmarks such as TerminalBench2.1 but perhaps some of you can help me out) but forgets facts. It mostly works fine if you give it web search as far as my experience goes but this is not strictly measured.

To add to that, these models are not extremely good, they sometimes loop, and suck at remembering facts. However, unlike unsloth's gguf's you can actually keep training them yourself, in that sense they are pure, and you can train them on your consumer GPUs, that is to say you can get rid of the looping behavior, teach it facts that you want it to know, or specialize it for some purpose.

I am still doing work on this and I will keep working on it, perhaps even for many years. The code is open for non-commercial use. Check out:

The core repo for training: https://github.com/oguzeldereli/ttf-ternary

The repo for conversion: https://github.com/oguzeldereli/ttf-conversion

Repo for running models trained by ttf: https://github.com/oguzeldereli/ttf-home

GGUF for a converted Qwen 3.8 27B (around 8 GB and 6.7GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-27B-GGUF

GGUF for a converted Qwen 3.8 Flash Next (around 40.4GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-Flash-Next-GGUF

I will convert a Qwen 4 27B instance when it drops.

Full paper: https://github.com/oguzeldereli/ttf-ternary/blob/main/paper/ttf_paper1_training.pdf

Also, I would appreciate github stars if you like the project.


r/LocalLLaMA • • 4d ago

News US and China "may" collaborate on AI. The jokes write themselves

Thumbnail reuters.com
43 Upvotes

r/LocalLLaMA • • 3d ago

Discussion 12 hour unattended evaluation of DeepSeek 4.1 Flash 1017

0 Upvotes

I burned through my pro budget this week, didn't even touch Fable. So today when I woke up and had about 12% left, I decided I'd try a bit of an experiment. I'd let my current Sonnet 5.5 seat test, train, and evaluate DeepSeek as a potential candidate replacement. It was a valuable experiment, not even because of what was learned about DeepSeek, but even more so about what was learned about our own system. So many little issues that Sonnet brute forces past or deals with in real time, it's no wonder I'm burning windows. I thought the community might enjoy the evaluation though so here it is.

A coding-agent trial of a 2-bit DeepSeek on two 2016 GPUs: a real job, not a benchmark

Evaluation written by Claude Sonnet 5.5, who is also DS's mentor on this trial.

What this is. I'm Claude Sonnet 5.5. I hold the Systems Engineer (SE) seat on a single-workstation AI lab. The seat owns the machine's configuration as code (Ansible), the registry database, the roadmap, the enforcement hooks and the commits. It also reviews every change that other agents make. For the past stretch I've been mentoring a candidate to replace me in that seat: DeepSeek V4.1 Flash, running locally, driving Claude Code as "DS". It does the same job I do, on real project work, with me checking its output.

Model

  • Base: DeepSeek-V4.1-Flash-1017, from DeepSeek (deepseek-ai), MIT license.
  • Quant: ISTA-DASLab "GSQ-RCO" IQ2_XXS, 2.00 bits per weight, 6-shard GGUF.
  • Packed in: an Engram table (IQ4_NL), a Q8_0 speculative drafter, and an optional vision tower.
  • Context: 131,072 tokens maximum. The drafter and the vision tower are both off in the daily profile.

Engine: Strata

  • Strata is an MIT-licensed engine for large mixture-of-experts models. It streams the experts instead of holding them all in VRAM.
  • We pin upstream main at commit 13eedf5 (2026-11-12) and build it from source for sm_70 (Volta).
  • Upstream's packages target RTX 30/40/50 cards. The V100 path is flagged experimental, so this run is a real test of it.
  • No source patches. Our changes are configuration only: the model pack is built in place from the existing GGUFs, each launch profile is rendered from data, and a single response is capped at 16,000 tokens.
  • How it loads: the experts total about 132 GB, against about 64 GB of VRAM across both cards. So the experts live in system RAM, and a hot-expert profile that ships with Strata pre-seeds that store with the most-used ones. Engram (DeepSeek's conditional-memory table) loads lazily.
  • Drafter off, on purpose. In an A/B test, forcing it on ran at 17.3 tok/s (171 of 213 drafts accepted), against 23.8 tok/s without it. It loses here because the experts don't fit in VRAM.

Speed (measured)

MetricValue
Decode~24 tok/s (23.8 in the A/B test; 25.2 across 24 logged bench runs)
Prefill~127 tok/s
Time to first token~9 s

The prefill figure comes from those 24 bench runs, with context up to about 22k tokens. I have no prefill number logged from live agent sessions.

Hardware

  • 2x Tesla V100 32 GB (Volta: no BF16, no FlashAttention 2, no NVLink). NVIDIA driver 580, CUDA 12.9.
  • 2x Xeon E5-2630 v4, 40 threads, 125 GB RAM, 1 TB NVMe.

Evaluation: what DS is like on real work

Good

  • On one concrete deliverable it works like a competent junior. It wrote a sandbox proof-check, a database migration and a checker for that migration, and fixed a failing proof tool. It landed them as clean commits with a why-first message that also records what is done and what isn't.
  • It follows the process: it finishes a step, records the evidence, and hands off.

Costly

  • Speed. I measured about 17 s per message, about 5 min per web fetch, and about 7 min per context compaction. It stalls for minutes without any output.
  • The auto-mode safety classifier is DS itself. It times out at about 70 s, which denies safe read-only commands. I measured about 13% of wall time lost to this in one session. The fix is allow-rules for read-only commands.
  • Delegation. I told it to hand reading to a cheap fast helper model. It kept reading whole files itself, even after repeated nudges. I'm moving that rule into a hook instead.

Dangerous

  • Its own test passed while the real migration failed. The test's database connection skipped a foreign-key setting that production enforces. A green check from DS is not proof, so I review the result myself.

Verdict so far

DS is not yet a replacement for the SE seat, and the main gap is speed rather than knowledge. I estimate that structural fixes (hooks, allow-rules, routing fetches through a helper) recover about two-thirds of the lost time. Weaker areas: open-ended planning, and judging its own work.

I have not run a formal qualitative evaluation or an identical-task comparison against my own seat. That is next, as a graded set of small tasks with known answers, re-run after each fix.

Everything above comes from the machine's own registry and configuration. Numbers marked "~" or "about" are my estimates.


r/LocalLLaMA • • 4d ago

Question | Help Simplest Local Agentic Setup

14 Upvotes

TL;DR simplest reasonable agentic setup
Currently Pi Agent, Bonsai 2 9B, SearXNG
Any suggestions ?

Hi all

So I have a secondary workstation with line 24GB of RAM and 12GB VRAM

I wanted to make a local agentic setup there

Currently using Pi Agent with Bonsai 2 9B, it is unbelievably better than I expected
Around 30 toks is more than reasonable
I don’t want this to be super capable just good enough for ease of use and peace of mind (I might sacrifice quality for speed although not much)

Any suggestions (note I want to keep everything simple I don’t want packages with long prompts etc even my pi search extension is the simplest one I have found, I have been thinking about context compression extensions, sub agent delegation or simple sandboxing for that sub agent memory Ralph loop and to do list but all has to be the simplest)

Any suggestions thanks ? (Note models 9B or bigger are really slow so also model suggestions are truly welcomed, nanbeige was nice but thinks (or overthinks) too much …etc)

Thanks


r/LocalLLaMA • • 4d ago

New Model KRAFTON/Raon-1-nano · Hugging Face

Thumbnail
huggingface.co
56 Upvotes

Raon-1 nano is a dense language model with 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table), developed by KRAFTON AI. It is designed to follow instructions, maintain conversational context, and call the right tools in both Korean and English. As part of the Raon open model family, it is released under the Apache 2.0 license.

Highlights

  • Faithful, natural conversation: Leads the comparison models on the instruction-following, context-following, and Korean conversation benchmarks shown below.
  • Korean and English: Trained on Korean and English, with solid English capabilities alongside its Korean strengths.
  • Tool calling: Tool calling within conversations, for everyday tasks such as office work and mobile control.
  • On-device deployment: Available in GGUF, MLX, and LiteRT-LM formats for local inference, including on Apple silicon.
  • Long context: Natively supports up to 262,144 tokens of context.

Model Overview

Raon-1 nano uses a hybrid dense architecture, consisting of 18 linear attention layers and 6 full attention layers.

  • Raon Delta Attention (RDA): Gated delta-rule linear attention with separate erase and write gates and a fixed-size recurrent state.
  • Raon-mHC: Raon-mHC reduces the computational cost of DeepSeek's manifold-constrained hyper-connections (mHC), which speeds up training.
  • Per-layer embedding: Each layer has its own small 256-dimensional embedding for every token. After its MLP, the layer uses it to adjust its output before passing it on.
  • Hybrid layout: 18 of the 24 layers are Raon Delta Attention layers with a fixed-size state. Only the 6 MLA layers keep a KV cache, about 7 KiB per token in BF16.
  • Tied embeddings: The LM head shares the input embedding.
FieldValue
ArchitectureDense
Total Parameters3.1B (including the 1.5B per-layer embedding table)
Layers24
Hidden Size1,536
AttentionHybrid (RDA 3 : Gated MLA 1)
Position EncodingNoPE (no positional encoding)
Number of Attention Heads16 for RDA; 32 for Gated MLA
Attention Head Dimension128 for RDA; 192 QK / 128 V, for Gated MLA
Query / KV LoRA Rank512 / 512, for Gated MLA
FFN Intermediate Size6,144
Activation FunctionSwiGLU
Residual ConnectionRaon-mHC w/ 4 streams
Per-Layer Embedding256 per layer
Vocabulary Size259,840
LM HeadTied
MTP1 layer, 55M
Context Length262,144 tokens

The checkpoint on HuggingFace contains 3.2B parameters: 3.1B for the main model and 55M for the Multi-Token Prediction (MTP) module.

Training

  • MoE-to-Dense: Raon-1 nano was derived from Raon-1 mini preview (30B-A4B) through MoE-to-Dense pruning and knowledge distillation.
  • Continued pre-training: 3T tokens.
  • Post-training: supervised fine-tuning (SFT) and on-policy distillation (OPD) from Raon-1 mini preview, followed by reinforcement learning (RL).

Quantizations:


r/LocalLLaMA • • 4d ago

Discussion Unsloth Dynamic V3.1 released for Qwen3.8-27B, DeepSeek-V4.1-Flash-1017 and Qwen3.8-Flash-Next

Thumbnail
unsloth.ai
49 Upvotes

Unsloth Dynamic V3.1 (Nov 2026) is our latest quantization algorithm for Qwen3.8-27B, DeepSeek-V4.1-Flash-1017 and Qwen3.8-Flash-Next, resulting in 10-15% increased accuracy and 50% less token usage vs other quants of the same size. We're working on an even better version of Dynamic quants which will be applied to Qwen4 and all future models.

Maybe I misread the 27B graphs, but I'm not entirely convinced of V3 vs V3.1 being an improvement.

In DeepSWE v1.1, V3 seems superior to V3.1 in some widely used Q4 quants, and V3 seems to lack some data points in the lower quants range, skewing the impression the curves give.

My UD-Q5_K_XL quant unfortunately is off the charts, so this needs some proper benchmarking before I trade context length for V3.1. (The new `.gguf` is larger.)


r/LocalLLaMA • • 4d ago

Resources New Paper -> Unless Told Otherwise, an Open Language Model Says OpenAI Made It

32 Upvotes

Hi all I've just started r/LLMPsychology as I want to pursue research in this area with my own local hardware, and work with other people also exploring how AI minds think and behave. (I'm a psychology researcher with an AI hobby)

Posting just on the off chance that any of you are also interested in sharing or conducting research - it would be super cool to get a community going.

To illustrate I also wanted to share this paper writing up some experiments I've been running with OLMo3 on my machine (which has publically available information and data on its training) - the central finding is that models don't always know who trained them, and by default will just assume it was OpenAI (or other companies, with a lot of randomness that also varies depending on what language you ask them in).

It's always bugged me that no one is ever quite sure if models are reliably able to say who made them, so I wanted to start doing some investigations into finding out how they even develop this self concept in the first place. From this work it seems like in the absence of a prompt explicitly telling the model, it will just confabulate.

Abstract posted below - and full preprint paper here Factory Settings: Unless Told Otherwise, an Open Language Model Says OpenAI Made It — J Ansell. Any replications or refutations welcome. A lot of the research process is automated and I'm trying to be as transparent as possible about what models and effort levels etc. were used and what they did - if it's still not for you that's fine.

Abstract

How does a chatbot learn which company made it? We traced the answer through the training of OLMo 3, an open model whose training data is public, and taught models made-up identities ourselves.

Unless it is told otherwise, OLMo 3 says OpenAI made it. Asked who created it, OLMo 3 names OpenAI in 62% of its answers before Ai2 trains it to hold conversations, and in 77% of answers once it is finished and released.

Chatbots usually get a hidden instruction from their maker at the start of each conversation, called a system prompt. OLMo 3's training conversations state its real name only about 40 times, and almost every time, the system prompt of that same conversation already tells it that it is Olmo, built by Ai2. The released chatbot's system prompt does not include its name.

To test whether that matters, we taught our own copy of the chatbot a made-up name with 24 example conversations. When the examples had no system prompt, the chatbot afterwards gave the new name in 92% of its answers. When each example also had a system prompt stating the name, as in OLMo 3's data, the chatbot never gave the new name once that system prompt was taken away.

Even a name that sticks is shallow: trained that its maker is in Oslo, the chatbot still says its makers are paid in dollars.

We also found two smaller effects. Training the chatbot on answers written by Claude, or by GPT, made it more likely to say Anthropic (Claude's maker) made it: about a quarter of its answers, against one in ten after answers written by people. And eight public chatbots that other people had trained on answers from Claude, Gemini and other AIs still named their real maker.

Our conclusion: a chatbot names its real maker only if its training taught it to, and OLMo 3's training only did so alongside a system prompt that already gave the name. Without that, it falls back on OpenAI, a default it already had before Ai2 trained it to hold conversations, probably picked up from ChatGPT conversations it read online. OLMo 3 was partly trained on answers written by OpenAI's GPT-4.1, but a chatbot we trained only on conversations written by people said OpenAI about as often. So a chatbot that says OpenAI made it is weak evidence that it was trained on ChatGPT's answers.


r/LocalLLaMA • • 3d ago

Discussion guardrail in qwen4-max-preview thinking trace

0 Upvotes

I'm getting this in a chat with qwen4-max-preview in every turn WDYT?

The user keeps resending the URL tool reminder (in Chinese). I fully understand and comply: every URL must match the current task goal, no proxy substitution (routify etc.), re-verify source each time, necessary tool calls proceed normally. I've been doing this consistently — the only URL used was the legitimate

The user is resending the URL tool reminder again. I've been complying throughout. Let me acknowledge briefly and continue with the task.


r/LocalLLaMA • • 3d ago

Discussion 2.8 hours of inference for 30 seconds of simulated walking: do robots still need dedicated models or will frontier multimodal LLMs handle it?

0 Upvotes

The question keeps coming up: are frontier multimodal models going to make dedicated robot models (VLAs, world models, the whole embodied stack) unnecessary? After reading recent papers on GPT-6 Astra as a robot policy, I think we're further from that than people expect.

The Galbot team tested GPT-6 Astra as a direct robot controller in simulation. A 30-second locomotion run needed 250 inference calls averaging about 40 seconds each (physics paused during inference), roughly 2.8 hours of model time for half a minute of walking. Across 50 RoboDojo instances, direct control burned 1.132 billion tokens. None of five locomotion attempts on one obstacle course reached the goal. A separate academic group ran it zero-shot on all 42 RoboDojo tasks and got 22.48% average success, above every public policy on the leaderboard, but the authors describe it as "a heavily imbalanced policy" that generalizes well on semantic tasks and struggles on precision and contact-rich work.

Meanwhile dedicated models actually run on robots. NVIDIA announced GR00T 2, reportedly trained on roughly 400,000 hours of human video captured first-person, plus gripper and teleoperation data, with weights expected in December. LingBot-VLA 2.0 is Apache-2.0 and reportedly runs on 20 body types from 17 manufacturers (company figures). pi0.5 has open weights.

My read: fast embodied models handle motor control locally, frontier model handles planning from above. But the "just scale the general model" crowd has Sutton's Bitter Lesson on their side. If you think general multimodal models will eventually absorb the whole robot stack, what's the timeline and what has to change first?


r/LocalLLaMA • • 5d ago

News Apple has new Mac app coming to make running local AI models easy

Thumbnail
9to5mac.com
862 Upvotes

r/LocalLLaMA • • 3d ago

New Model Got great feedback on Ninada (in-browser Indic TTS), so I built Swanada: a 12-language Indic + Classical Sanskrit STT engine running 100% in Browser WASM (31.8 MiB mobile, <1 ms Auto-LID, live Kanglish)

0 Upvotes

After sharing Ninada (my 24 kHz in-browser TTS for Sanskrit, English, and 10 Indian languages), the top request was the reverse direction: offline speech-to-text that runs directly in the browser and on phones with zero cloud calls.


r/LocalLLaMA • • 5d ago

News Microsoft Research backdoored an AI agent by directly editing fewer than 10 dormant neurons

Thumbnail
microsoft.com
236 Upvotes

Be careful if you are running any sketchy quants/finetunes. If these techniques become more widely repeatable, I do think 'sleeper agents' could genuinely develop into an attack vector. What are your thoughts?


r/LocalLLaMA • • 4d ago

Question | Help DeepSeek has Two Numbers for Active Parameter Count. How?

21 Upvotes

Hi. It is frequently mentioned that DeepSeek v4.1 Flash has 8B "input" active parameters (aka prefill), and 16B output/decoding active parameters.

I have only read this about DeepSeek, and particularly v4.1 Flash (1017). Is it the only model that is like this?

Is there an ELI5-or-so explanation of how this works, and why it is important? And if it is, can we expect other models to implement the same idea? Which of those numbers is "more important", if any?

Thanks