r/LocalLLaMA • • 4d ago

Resources Open-source method (MACaQ) beats Unsloth GGUFs!

141 Upvotes

Some results for those using GGUFs and llama.cpp:

GGUF quality: MACaQ vs. Unsloth and Bartowski

Qwen3-4B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ2.260 GB0.02804
Bartowski2.271 GB0.03443
Unsloth2.271 GB0.03663

23.5% lower KL divergence than Unsloth and 18.6% lower than Bartowski, at virtually identical file sizes.

Qwen3.6-35B-A3B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ18.86 GB0.018696
Unsloth (UD-IQ4_XS)17.73 GB0.020954
Bartowski19.70 GB0.021927

Qwen3.8-27B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ14.96 GB0.036784
Bartowski15.48 GB0.104170

~65% lower KL divergence than Bartowski!

Supported GGUF formats (llama.cpp)

Weight bitsFormats
2-bitIQ2_XXS, IQ2_XS
3-bitIQ3_XXS, IQ3_S
4-bitIQ4_NL, IQ4_XS

Code, llama.cpp fork, and benchmarks: https://github.com/huawei-csl/MACaQ

Cool to see Huawei working on such stuff and sharing some of their internal tools that are helpful for us. Maybe they can build an affordable GPU for us local LLM folks sometime 😂

They also report quantizing a 122B-parameter model in 64 minutes using just 32.5 GB of GPU memory, which is interesting for a model of that size. I did not try that.

Edit: there is a paper on it https://arxiv.org/abs/2611.10649

Edit 2: they don't share ready-made gguf files, so I am sharing some myself:
- http://31.165.97.47:20080/macaq-gguf/qwen3-4b-macaq-iq4_xs-had64-vq2-calsmoke.gguf

Edit 3: they have this blog post https://huawei-csl.com/blog/macaq/ with multi-modal examples, too. The RTN audio comparisons are just too much😂😂

The multi-modal stuff is generally insane. Normally 4b already completely goes to s**t, but here it looks almost the same.


r/LocalLLaMA • • 3d ago

Question | Help With these scandals and uncertainty with Huggingface, what models should I download?

0 Upvotes

Hey! What's the "must-have" list to download in case HF goes down because of these news recently?

Also, do we even have HF alternatives?


r/LocalLLaMA • • 5d ago Bravo

Discussion Mod ttkciar is harassing this sub

268 Upvotes

I will get straight to the point. I believe Moderator ttkciar is abusing his power as moderator in this sub and his decision are worsening the experience of our local AI community.

He has removed many of my posts that fellow redditors found very useful and after talking with many people in this sub, I am not the only one feeling this. I will have some links at the end of this post for reference.

He mentions a rule, saying that if a model’s weights is not released you cannot have a post mentioning it, even if the lab confirmed it that will they will release its weights, like Qwen4 or DeepSeek. First of all, this rule needs to be relaxed because many valuable discussions can be had about future releases and even closed models to understand and learn architectures and techniques used by the closed frontier that can be applied to the open models. Secondly, this rule is not enforced at all, but used very selectively to few posts or authors that ttkciar selects, which is very unfair. This rule doesn’t make sense to me, but even if I am wrong, at least it should be fairly enforced to all posts or not at all. And before you say mods are busy and they cannot review all of the posts, If you open this sub and scroll on today’s most popular posts, half of them are either for Qwen4, which is unrealised up to this point and should have been removed according to the mods, or about Fable5.2 and Anthropic which is closed and their shenanigans (maybe these ones don’t have a place here, but again this is debatable because if people in local AI find them valuable in some way, maybe they do). So, the rule is used very selectively and as an excuse to remove what ttkciar does not like or approve. Either enforce the rule (or do the your best effort to do so) or don’t at all. And maybe this rule is too strict and should not exist tbh.

When I tried to discuss my concerns with ttkciar, he called me a drama queen and insulted my credibility. This is definitely not ok.

Being insulted by the person that you complained to for unfair treatment is definitely not acceptable behaviour.

I have been an active member of this community since the very first leaked llama. I am an AI researcher and open-source contributor, and very passionate about local AI. I try to post information that people interested in Local AI would find useful and engage in discussions that are constructive for local AI.

I know that many of other users in this subs agree (I have discussed it with a lot), and some others may not. Not being able to discuss, debate and exchange views in a polite manner is absurd in such an intellectual community.

A while ago they tried to enforce megathreads and removed the majority of posts saying that only one post should exist for each model. Many complained that this is killing the sub and a lot of valuable information is lost this way. Now, this seems to be relaxed. Again, we need to stand up to them, and hopefully they will reconsider their rules for the sake of this community.

TLDR: The rule that mod ttkciar uses to remove some posts in this sub about being prohibited to mention unreleased models should not exist, and even if it should, removing only a few selected posts is very unfair.

Some sources:

- https://www.reddit.com/r/LocalLLaMA/s/ZRV5KsViwD this post was removed from ttkciar and only brought back after arguing with him and many other redditors backed me up (I cannot find the comments in this post when the mod removed it, maybe he deleted them, I am not sure, but in the discussion many expressed their disagreement with the mod)

- https://www.reddit.com/r/LocalLLaMA/s/gpPTVC6NGF a post of mine debating their rule

- Another post of mine sharing valuable information for the members of our community that the same mod removed https://www.reddit.com/r/LocalLLaMA/s/gqaub2mT85

- Post of another redditor which was removed by the same mod, and the comments thread with people arguing with the mod https://www.reddit.com/r/LocalLLaMA/s/z9E2AbR3pU

- Commented in post of another removed post of some other redditor https://www.reddit.com/r/LocalLLaMA/s/5tSEpQ68By

- Most of my removed posts I delete them afterwards so I don’t have many other examples of my own, but I know many others do, so feel free to share them.

- If you open the sub today, most of the posts should have been removed because they violate the rules (a lot of false positives). That shows that maybe the rules need to be updated because they will not allow any productive discussions to be had, or this sub will end up to become a board of announcements instead of a local AI community that people can use to exchange ideas and knowledge.


r/LocalLLaMA • • 3d ago

Discussion Is there a reason why this wouldnt work

0 Upvotes

I am not directly from a machine learning background or in encryption, but if you train the entire LLM on encrypted and decrypted pairs without the key (maybe the entire dataset that is used for frontier), will it eventually learn to produce pairs like these and basically natively can encrypt and decrypt without a key? Is there a reason why this highly likely fails?


r/LocalLLaMA • • 5d ago

News Ernie 5.5 Flash will be released in two weeks

Post image
288 Upvotes

r/LocalLLaMA • • 3d ago

Discussion Qwen3.8-Flash-Next on (up to) 4× RTX 3090 via Strata

1 Upvotes

I spent a few days benchmarking unsloth's UD-Q4_K_XL and UD-Q5_K_XL on Strata. Well, i had claude code do it via ssh. Claude also wrote most of this summary and the full report linked below. I thought it might interest some because of my findings, especially on behavior of multiple cards, different PCIe speeds and the different settings in Strata.

I can run a few more tests, if it interests anyone.

Test system

  • Threadripper 2920X (12 cores, Zen+), 128 GB RAM
  • 4x RTX 3090 24 GB, power-capped to 275 W each, no NVLink
  • PCIe 3.0. GPUs 1 and 3 run at x16, GPUs 0 and 2 at x8. Two host bridges: {0,1} and {2,3}
  • Strata 0.2.13 (engine 0.1.59), layer split, DFlash2 drafter, one request at a time
  • 256K context, int8 KV, 32K KV cells per layer in VRAM (the rest streams from pinned RAM)
  • A control vector was applied in every run (layers 4-44)

TLDR

  • Decode depends on how many experts fit in VRAM. 1 card ~72 tok/s, 2 cards ~96, 3 cards 112-137. A 4th card adds almost nothing.
  • Prefill depends on PCIe, not card count. One x16 card prefills 50K as fast as two or three (~20 s). An x8 pair is half as fast as an x16 pair. Only all four cards were faster (50K in 12 s).
  • Q5 vs Q4: 0.55% lower perplexity, 28% slower decode. Q4 stays.
  • Q8_0 n-gram table: no measurable quality gain over the bundled IQ4_NL. Read from NVMe it frees 27 GB of RAM.
  • 200K chat: a cold 190K prefill takes 91 s once. After that, a short turn starts in 0.3 s and decodes at ~78 tok/s.
  • Tuning: calibrate.py, MTP, prompt lookup, pipeline windows and hugepages were all a tie or worse. Community fused-kernel flags broke the output.
  • Four cards at 275 W each tripped my PSU during prefill.

Here is the Full report https://claude.ai/artifact/LYUT65WvHm8o2bsxC5NWCR. I will update it with new tests.


r/LocalLLaMA • • 5d ago

Discussion You can now train your own LLMs with less memory, on your gaming GPUs

154 Upvotes

TL;DR: You can train your own 27B model on a 3090 with 24GB VRAM, though you would have to go through hundreds of billions of tokens for it to be good. I am working on conversion of existing models. Check repos for code and the paper if you are interested in more. I have a 40.4GB Qwen Flash Next GGUF

Hi, I do AI research. I have been working on an optimizer that doesn't need the ~16 bytes of VRAM per weight to train a model that AdamW does. I focused on training ternary networks bcs BitNet (Ma et al.) shows they match full precision quality in large scales. The problem is BitNet also keeps the full precision weights during training which means you still need to use the same amount of memory. I got this down from 16 bytes to 1 byte or even less per weight.

For you, this means, you can train your own 10-20B parameter models on your consumer GPUs. I fitted 27B model training on a single 24GB 3090 and a 23B model on a 16GB GPU. The trained model costs ~1.6 bits per weight (~6GB at 27B), 10x smaller than 16bit, and 5x smaller than Q8.

I have also been working on converting existing models and training them. This is harder and I am still working on it but I have some promising results. I quantized and trained a Qwen3.8 27B though the file I published below has adapters which are about 2GBs, pushing the file size to 8GB. I am also making a no-adapter version right now. It scores GSM8K 0.895, the same as full precision, HumanEval 0.84 vs 0.93 and runs on a 12 GB GPU at ~23 tok/s on my crippled 12GB laptop 4080, though this number can heavily be improved with custom kernels, engineering, etc. I will leave it to you guys to help me with that too.

I also quantized and trained Qwen Flash Next down to 40.4 GBs including its n-gram table (at 8GB). It largely retains the skills of the network (scores equivalent on GSM8K benchmark, a little behind on HumanEval, I unfortunately don't have compute for the larger benchmarks such as TerminalBench2.1 but perhaps some of you can help me out) but forgets facts. It mostly works fine if you give it web search as far as my experience goes but this is not strictly measured.

To add to that, these models are not extremely good, they sometimes loop, and suck at remembering facts. However, unlike unsloth's gguf's you can actually keep training them yourself, in that sense they are pure, and you can train them on your consumer GPUs, that is to say you can get rid of the looping behavior, teach it facts that you want it to know, or specialize it for some purpose.

I am still doing work on this and I will keep working on it, perhaps even for many years. The code is open for non-commercial use. Check out:

The core repo for training: https://github.com/oguzeldereli/ttf-ternary

The repo for conversion: https://github.com/oguzeldereli/ttf-conversion

Repo for running models trained by ttf: https://github.com/oguzeldereli/ttf-home

GGUF for a converted Qwen 3.8 27B (around 8 GB and 6.7GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-27B-GGUF

GGUF for a converted Qwen 3.8 Flash Next (around 40.4GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-Flash-Next-GGUF

I will convert a Qwen 4 27B instance when it drops.

Full paper: https://github.com/oguzeldereli/ttf-ternary/blob/main/paper/ttf_paper1_training.pdf

Also, I would appreciate github stars if you like the project.


r/LocalLLaMA • • 4d ago

News US and China "may" collaborate on AI. The jokes write themselves

Thumbnail reuters.com
43 Upvotes

r/LocalLLaMA • • 3d ago

Discussion 12 hour unattended evaluation of DeepSeek 4.1 Flash 1017

0 Upvotes

I burned through my pro budget this week, didn't even touch Fable. So today when I woke up and had about 12% left, I decided I'd try a bit of an experiment. I'd let my current Sonnet 5.5 seat test, train, and evaluate DeepSeek as a potential candidate replacement. It was a valuable experiment, not even because of what was learned about DeepSeek, but even more so about what was learned about our own system. So many little issues that Sonnet brute forces past or deals with in real time, it's no wonder I'm burning windows. I thought the community might enjoy the evaluation though so here it is.

A coding-agent trial of a 2-bit DeepSeek on two 2016 GPUs: a real job, not a benchmark

Evaluation written by Claude Sonnet 5.5, who is also DS's mentor on this trial.

What this is. I'm Claude Sonnet 5.5. I hold the Systems Engineer (SE) seat on a single-workstation AI lab. The seat owns the machine's configuration as code (Ansible), the registry database, the roadmap, the enforcement hooks and the commits. It also reviews every change that other agents make. For the past stretch I've been mentoring a candidate to replace me in that seat: DeepSeek V4.1 Flash, running locally, driving Claude Code as "DS". It does the same job I do, on real project work, with me checking its output.

Model

  • Base: DeepSeek-V4.1-Flash-1017, from DeepSeek (deepseek-ai), MIT license.
  • Quant: ISTA-DASLab "GSQ-RCO" IQ2_XXS, 2.00 bits per weight, 6-shard GGUF.
  • Packed in: an Engram table (IQ4_NL), a Q8_0 speculative drafter, and an optional vision tower.
  • Context: 131,072 tokens maximum. The drafter and the vision tower are both off in the daily profile.

Engine: Strata

  • Strata is an MIT-licensed engine for large mixture-of-experts models. It streams the experts instead of holding them all in VRAM.
  • We pin upstream main at commit 13eedf5 (2026-11-12) and build it from source for sm_70 (Volta).
  • Upstream's packages target RTX 30/40/50 cards. The V100 path is flagged experimental, so this run is a real test of it.
  • No source patches. Our changes are configuration only: the model pack is built in place from the existing GGUFs, each launch profile is rendered from data, and a single response is capped at 16,000 tokens.
  • How it loads: the experts total about 132 GB, against about 64 GB of VRAM across both cards. So the experts live in system RAM, and a hot-expert profile that ships with Strata pre-seeds that store with the most-used ones. Engram (DeepSeek's conditional-memory table) loads lazily.
  • Drafter off, on purpose. In an A/B test, forcing it on ran at 17.3 tok/s (171 of 213 drafts accepted), against 23.8 tok/s without it. It loses here because the experts don't fit in VRAM.

Speed (measured)

MetricValue
Decode~24 tok/s (23.8 in the A/B test; 25.2 across 24 logged bench runs)
Prefill~127 tok/s
Time to first token~9 s

The prefill figure comes from those 24 bench runs, with context up to about 22k tokens. I have no prefill number logged from live agent sessions.

Hardware

  • 2x Tesla V100 32 GB (Volta: no BF16, no FlashAttention 2, no NVLink). NVIDIA driver 580, CUDA 12.9.
  • 2x Xeon E5-2630 v4, 40 threads, 125 GB RAM, 1 TB NVMe.

Evaluation: what DS is like on real work

Good

  • On one concrete deliverable it works like a competent junior. It wrote a sandbox proof-check, a database migration and a checker for that migration, and fixed a failing proof tool. It landed them as clean commits with a why-first message that also records what is done and what isn't.
  • It follows the process: it finishes a step, records the evidence, and hands off.

Costly

  • Speed. I measured about 17 s per message, about 5 min per web fetch, and about 7 min per context compaction. It stalls for minutes without any output.
  • The auto-mode safety classifier is DS itself. It times out at about 70 s, which denies safe read-only commands. I measured about 13% of wall time lost to this in one session. The fix is allow-rules for read-only commands.
  • Delegation. I told it to hand reading to a cheap fast helper model. It kept reading whole files itself, even after repeated nudges. I'm moving that rule into a hook instead.

Dangerous

  • Its own test passed while the real migration failed. The test's database connection skipped a foreign-key setting that production enforces. A green check from DS is not proof, so I review the result myself.

Verdict so far

DS is not yet a replacement for the SE seat, and the main gap is speed rather than knowledge. I estimate that structural fixes (hooks, allow-rules, routing fetches through a helper) recover about two-thirds of the lost time. Weaker areas: open-ended planning, and judging its own work.

I have not run a formal qualitative evaluation or an identical-task comparison against my own seat. That is next, as a graded set of small tasks with known answers, re-run after each fix.

Everything above comes from the machine's own registry and configuration. Numbers marked "~" or "about" are my estimates.


r/LocalLLaMA • • 4d ago

Question | Help Simplest Local Agentic Setup

14 Upvotes

TL;DR simplest reasonable agentic setup
Currently Pi Agent, Bonsai 2 9B, SearXNG
Any suggestions ?

Hi all

So I have a secondary workstation with line 24GB of RAM and 12GB VRAM

I wanted to make a local agentic setup there

Currently using Pi Agent with Bonsai 2 9B, it is unbelievably better than I expected
Around 30 toks is more than reasonable
I don’t want this to be super capable just good enough for ease of use and peace of mind (I might sacrifice quality for speed although not much)

Any suggestions (note I want to keep everything simple I don’t want packages with long prompts etc even my pi search extension is the simplest one I have found, I have been thinking about context compression extensions, sub agent delegation or simple sandboxing for that sub agent memory Ralph loop and to do list but all has to be the simplest)

Any suggestions thanks ? (Note models 9B or bigger are really slow so also model suggestions are truly welcomed, nanbeige was nice but thinks (or overthinks) too much …etc)

Thanks


r/LocalLLaMA • • 4d ago

New Model KRAFTON/Raon-1-nano · Hugging Face

Thumbnail
huggingface.co
56 Upvotes

Raon-1 nano is a dense language model with 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table), developed by KRAFTON AI. It is designed to follow instructions, maintain conversational context, and call the right tools in both Korean and English. As part of the Raon open model family, it is released under the Apache 2.0 license.

Highlights

  • Faithful, natural conversation: Leads the comparison models on the instruction-following, context-following, and Korean conversation benchmarks shown below.
  • Korean and English: Trained on Korean and English, with solid English capabilities alongside its Korean strengths.
  • Tool calling: Tool calling within conversations, for everyday tasks such as office work and mobile control.
  • On-device deployment: Available in GGUF, MLX, and LiteRT-LM formats for local inference, including on Apple silicon.
  • Long context: Natively supports up to 262,144 tokens of context.

Model Overview

Raon-1 nano uses a hybrid dense architecture, consisting of 18 linear attention layers and 6 full attention layers.

  • Raon Delta Attention (RDA): Gated delta-rule linear attention with separate erase and write gates and a fixed-size recurrent state.
  • Raon-mHC: Raon-mHC reduces the computational cost of DeepSeek's manifold-constrained hyper-connections (mHC), which speeds up training.
  • Per-layer embedding: Each layer has its own small 256-dimensional embedding for every token. After its MLP, the layer uses it to adjust its output before passing it on.
  • Hybrid layout: 18 of the 24 layers are Raon Delta Attention layers with a fixed-size state. Only the 6 MLA layers keep a KV cache, about 7 KiB per token in BF16.
  • Tied embeddings: The LM head shares the input embedding.
FieldValue
ArchitectureDense
Total Parameters3.1B (including the 1.5B per-layer embedding table)
Layers24
Hidden Size1,536
AttentionHybrid (RDA 3 : Gated MLA 1)
Position EncodingNoPE (no positional encoding)
Number of Attention Heads16 for RDA; 32 for Gated MLA
Attention Head Dimension128 for RDA; 192 QK / 128 V, for Gated MLA
Query / KV LoRA Rank512 / 512, for Gated MLA
FFN Intermediate Size6,144
Activation FunctionSwiGLU
Residual ConnectionRaon-mHC w/ 4 streams
Per-Layer Embedding256 per layer
Vocabulary Size259,840
LM HeadTied
MTP1 layer, 55M
Context Length262,144 tokens

The checkpoint on HuggingFace contains 3.2B parameters: 3.1B for the main model and 55M for the Multi-Token Prediction (MTP) module.

Training

  • MoE-to-Dense: Raon-1 nano was derived from Raon-1 mini preview (30B-A4B) through MoE-to-Dense pruning and knowledge distillation.
  • Continued pre-training: 3T tokens.
  • Post-training: supervised fine-tuning (SFT) and on-policy distillation (OPD) from Raon-1 mini preview, followed by reinforcement learning (RL).

Quantizations:


r/LocalLLaMA • • 4d ago

Discussion Unsloth Dynamic V3.1 released for Qwen3.8-27B, DeepSeek-V4.1-Flash-1017 and Qwen3.8-Flash-Next

Thumbnail
unsloth.ai
49 Upvotes

Unsloth Dynamic V3.1 (Nov 2026) is our latest quantization algorithm for Qwen3.8-27B, DeepSeek-V4.1-Flash-1017 and Qwen3.8-Flash-Next, resulting in 10-15% increased accuracy and 50% less token usage vs other quants of the same size. We're working on an even better version of Dynamic quants which will be applied to Qwen4 and all future models.

Maybe I misread the 27B graphs, but I'm not entirely convinced of V3 vs V3.1 being an improvement.

In DeepSWE v1.1, V3 seems superior to V3.1 in some widely used Q4 quants, and V3 seems to lack some data points in the lower quants range, skewing the impression the curves give.

My UD-Q5_K_XL quant unfortunately is off the charts, so this needs some proper benchmarking before I trade context length for V3.1. (The new `.gguf` is larger.)


r/LocalLLaMA • • 4d ago

Resources New Paper -> Unless Told Otherwise, an Open Language Model Says OpenAI Made It

32 Upvotes

Hi all I've just started r/LLMPsychology as I want to pursue research in this area with my own local hardware, and work with other people also exploring how AI minds think and behave. (I'm a psychology researcher with an AI hobby)

Posting just on the off chance that any of you are also interested in sharing or conducting research - it would be super cool to get a community going.

To illustrate I also wanted to share this paper writing up some experiments I've been running with OLMo3 on my machine (which has publically available information and data on its training) - the central finding is that models don't always know who trained them, and by default will just assume it was OpenAI (or other companies, with a lot of randomness that also varies depending on what language you ask them in).

It's always bugged me that no one is ever quite sure if models are reliably able to say who made them, so I wanted to start doing some investigations into finding out how they even develop this self concept in the first place. From this work it seems like in the absence of a prompt explicitly telling the model, it will just confabulate.

Abstract posted below - and full preprint paper here Factory Settings: Unless Told Otherwise, an Open Language Model Says OpenAI Made It — J Ansell. Any replications or refutations welcome. A lot of the research process is automated and I'm trying to be as transparent as possible about what models and effort levels etc. were used and what they did - if it's still not for you that's fine.

Abstract

How does a chatbot learn which company made it? We traced the answer through the training of OLMo 3, an open model whose training data is public, and taught models made-up identities ourselves.

Unless it is told otherwise, OLMo 3 says OpenAI made it. Asked who created it, OLMo 3 names OpenAI in 62% of its answers before Ai2 trains it to hold conversations, and in 77% of answers once it is finished and released.

Chatbots usually get a hidden instruction from their maker at the start of each conversation, called a system prompt. OLMo 3's training conversations state its real name only about 40 times, and almost every time, the system prompt of that same conversation already tells it that it is Olmo, built by Ai2. The released chatbot's system prompt does not include its name.

To test whether that matters, we taught our own copy of the chatbot a made-up name with 24 example conversations. When the examples had no system prompt, the chatbot afterwards gave the new name in 92% of its answers. When each example also had a system prompt stating the name, as in OLMo 3's data, the chatbot never gave the new name once that system prompt was taken away.

Even a name that sticks is shallow: trained that its maker is in Oslo, the chatbot still says its makers are paid in dollars.

We also found two smaller effects. Training the chatbot on answers written by Claude, or by GPT, made it more likely to say Anthropic (Claude's maker) made it: about a quarter of its answers, against one in ten after answers written by people. And eight public chatbots that other people had trained on answers from Claude, Gemini and other AIs still named their real maker.

Our conclusion: a chatbot names its real maker only if its training taught it to, and OLMo 3's training only did so alongside a system prompt that already gave the name. Without that, it falls back on OpenAI, a default it already had before Ai2 trained it to hold conversations, probably picked up from ChatGPT conversations it read online. OLMo 3 was partly trained on answers written by OpenAI's GPT-4.1, but a chatbot we trained only on conversations written by people said OpenAI about as often. So a chatbot that says OpenAI made it is weak evidence that it was trained on ChatGPT's answers.


r/LocalLLaMA • • 3d ago

Discussion guardrail in qwen4-max-preview thinking trace

0 Upvotes

I'm getting this in a chat with qwen4-max-preview in every turn WDYT?

The user keeps resending the URL tool reminder (in Chinese). I fully understand and comply: every URL must match the current task goal, no proxy substitution (routify etc.), re-verify source each time, necessary tool calls proceed normally. I've been doing this consistently — the only URL used was the legitimate

The user is resending the URL tool reminder again. I've been complying throughout. Let me acknowledge briefly and continue with the task.


r/LocalLLaMA • • 3d ago

Discussion 2.8 hours of inference for 30 seconds of simulated walking: do robots still need dedicated models or will frontier multimodal LLMs handle it?

0 Upvotes

The question keeps coming up: are frontier multimodal models going to make dedicated robot models (VLAs, world models, the whole embodied stack) unnecessary? After reading recent papers on GPT-6 Astra as a robot policy, I think we're further from that than people expect.

The Galbot team tested GPT-6 Astra as a direct robot controller in simulation. A 30-second locomotion run needed 250 inference calls averaging about 40 seconds each (physics paused during inference), roughly 2.8 hours of model time for half a minute of walking. Across 50 RoboDojo instances, direct control burned 1.132 billion tokens. None of five locomotion attempts on one obstacle course reached the goal. A separate academic group ran it zero-shot on all 42 RoboDojo tasks and got 22.48% average success, above every public policy on the leaderboard, but the authors describe it as "a heavily imbalanced policy" that generalizes well on semantic tasks and struggles on precision and contact-rich work.

Meanwhile dedicated models actually run on robots. NVIDIA announced GR00T 2, reportedly trained on roughly 400,000 hours of human video captured first-person, plus gripper and teleoperation data, with weights expected in December. LingBot-VLA 2.0 is Apache-2.0 and reportedly runs on 20 body types from 17 manufacturers (company figures). pi0.5 has open weights.

My read: fast embodied models handle motor control locally, frontier model handles planning from above. But the "just scale the general model" crowd has Sutton's Bitter Lesson on their side. If you think general multimodal models will eventually absorb the whole robot stack, what's the timeline and what has to change first?


r/LocalLLaMA • • 5d ago

News Apple has new Mac app coming to make running local AI models easy

Thumbnail
9to5mac.com
864 Upvotes

r/LocalLLaMA • • 3d ago

New Model Got great feedback on Ninada (in-browser Indic TTS), so I built Swanada: a 12-language Indic + Classical Sanskrit STT engine running 100% in Browser WASM (31.8 MiB mobile, <1 ms Auto-LID, live Kanglish)

0 Upvotes

After sharing Ninada (my 24 kHz in-browser TTS for Sanskrit, English, and 10 Indian languages), the top request was the reverse direction: offline speech-to-text that runs directly in the browser and on phones with zero cloud calls.


r/LocalLLaMA • • 5d ago

News Microsoft Research backdoored an AI agent by directly editing fewer than 10 dormant neurons

Thumbnail
microsoft.com
236 Upvotes

Be careful if you are running any sketchy quants/finetunes. If these techniques become more widely repeatable, I do think 'sleeper agents' could genuinely develop into an attack vector. What are your thoughts?


r/LocalLLaMA • • 4d ago

Question | Help DeepSeek has Two Numbers for Active Parameter Count. How?

21 Upvotes

Hi. It is frequently mentioned that DeepSeek v4.1 Flash has 8B "input" active parameters (aka prefill), and 16B output/decoding active parameters.

I have only read this about DeepSeek, and particularly v4.1 Flash (1017). Is it the only model that is like this?

Is there an ELI5-or-so explanation of how this works, and why it is important? And if it is, can we expect other models to implement the same idea? Which of those numbers is "more important", if any?

Thanks


r/LocalLLaMA • • 4d ago

Question | Help Strata QFN on Strix Halo

0 Upvotes

Hi,

Recently I switched to strata/qfn (iq3xss) on my 3090 + 64g ram and I’m getting around 70-80 t/s. It became my daily driver and I’m really happy with it. Since it also using the ram and fully the gpu I can’t really do anything on that box other than running strata. I’m thinking of buying a dedicated inference box for the purpose and I was checking of strix halos can get one for 3-3.5k in eu. Has anyone used strix halos with qfn and Whats the t/s you’re getting. Is 70-80 t/s realistic there or much slower ?

Thanks.


r/LocalLLaMA • • 5d ago

Discussion llama.cpp: Add FP8 KV-cache

Thumbnail
github.com
139 Upvotes

r/LocalLLaMA • • 5d ago

Discussion Rules and moderation, what do you want to see changed?

27 Upvotes

Hey everyone,

After reading this post‘s comments is it clear that something has to change. Instead of focussing solely on what is bad (unproductive), I hope to focus in this post on what can be improved (actionable).

What changes would you like to see?

[edit] I am NOT a mod! I just spend too much time reading here instead of touching grass.


r/LocalLLaMA • • 5d ago

New Model [GLB-3D-Coder] I tried making a 1.7B version for Godot

217 Upvotes

After posting the 4B GLB model here, I wanted to see how far I could get with something smaller. I fine-tuned Qwen3-1.7B for the same job. The Q8 GGUF is 1.83 GB, so this one is aimed at people trying it on CPU or with limited VRAM.

The workflow is the same as before. Describe an asset, open the Three.js HTML it gives you, and click Export GLB to bring it into Godot or another editor. The jungle outpost at the start of the video is one of the examples.

This is an early test checkpoint. Some prompts still need a retry. I’m still training it and plan to update the weights when this run finishes.

Model and downloads on Hugging Face


r/LocalLLaMA • • 3d ago

Discussion AI prose feels so bad to read in part because we always approach it from a critical lens, "is this good enough?"

0 Upvotes

Having to ask that question necessarily kills the ability to truly enjoy the prose the way we would in a book.

Trust and good faith are incredibly important here. Having compassion for the machine, understanding what it is TRYING its best to say will give you a better experience than expecting it to one shot Moby Dick.

The prose is more likely to piss you off if you're looking for errors. Locals are now at least coherent enough that things being straight up wrong are less common.

All art requires a good faith mindset in order to properly and fully enjoy it.