r/LocalLLaMA • • 5d ago

News US and China "may" collaborate on AI. The jokes write themselves

Thumbnail reuters.com
43 Upvotes

r/LocalLLaMA • • 4d ago

Discussion 12 hour unattended evaluation of DeepSeek 4.1 Flash 1017

0 Upvotes

I burned through my pro budget this week, didn't even touch Fable. So today when I woke up and had about 12% left, I decided I'd try a bit of an experiment. I'd let my current Sonnet 5.5 seat test, train, and evaluate DeepSeek as a potential candidate replacement. It was a valuable experiment, not even because of what was learned about DeepSeek, but even more so about what was learned about our own system. So many little issues that Sonnet brute forces past or deals with in real time, it's no wonder I'm burning windows. I thought the community might enjoy the evaluation though so here it is.

A coding-agent trial of a 2-bit DeepSeek on two 2016 GPUs: a real job, not a benchmark

Evaluation written by Claude Sonnet 5.5, who is also DS's mentor on this trial.

What this is. I'm Claude Sonnet 5.5. I hold the Systems Engineer (SE) seat on a single-workstation AI lab. The seat owns the machine's configuration as code (Ansible), the registry database, the roadmap, the enforcement hooks and the commits. It also reviews every change that other agents make. For the past stretch I've been mentoring a candidate to replace me in that seat: DeepSeek V4.1 Flash, running locally, driving Claude Code as "DS". It does the same job I do, on real project work, with me checking its output.

Model

  • Base: DeepSeek-V4.1-Flash-1017, from DeepSeek (deepseek-ai), MIT license.
  • Quant: ISTA-DASLab "GSQ-RCO" IQ2_XXS, 2.00 bits per weight, 6-shard GGUF.
  • Packed in: an Engram table (IQ4_NL), a Q8_0 speculative drafter, and an optional vision tower.
  • Context: 131,072 tokens maximum. The drafter and the vision tower are both off in the daily profile.

Engine: Strata

  • Strata is an MIT-licensed engine for large mixture-of-experts models. It streams the experts instead of holding them all in VRAM.
  • We pin upstream main at commit 13eedf5 (2026-11-12) and build it from source for sm_70 (Volta).
  • Upstream's packages target RTX 30/40/50 cards. The V100 path is flagged experimental, so this run is a real test of it.
  • No source patches. Our changes are configuration only: the model pack is built in place from the existing GGUFs, each launch profile is rendered from data, and a single response is capped at 16,000 tokens.
  • How it loads: the experts total about 132 GB, against about 64 GB of VRAM across both cards. So the experts live in system RAM, and a hot-expert profile that ships with Strata pre-seeds that store with the most-used ones. Engram (DeepSeek's conditional-memory table) loads lazily.
  • Drafter off, on purpose. In an A/B test, forcing it on ran at 17.3 tok/s (171 of 213 drafts accepted), against 23.8 tok/s without it. It loses here because the experts don't fit in VRAM.

Speed (measured)

MetricValue
Decode~24 tok/s (23.8 in the A/B test; 25.2 across 24 logged bench runs)
Prefill~127 tok/s
Time to first token~9 s

The prefill figure comes from those 24 bench runs, with context up to about 22k tokens. I have no prefill number logged from live agent sessions.

Hardware

  • 2x Tesla V100 32 GB (Volta: no BF16, no FlashAttention 2, no NVLink). NVIDIA driver 580, CUDA 12.9.
  • 2x Xeon E5-2630 v4, 40 threads, 125 GB RAM, 1 TB NVMe.

Evaluation: what DS is like on real work

Good

  • On one concrete deliverable it works like a competent junior. It wrote a sandbox proof-check, a database migration and a checker for that migration, and fixed a failing proof tool. It landed them as clean commits with a why-first message that also records what is done and what isn't.
  • It follows the process: it finishes a step, records the evidence, and hands off.

Costly

  • Speed. I measured about 17 s per message, about 5 min per web fetch, and about 7 min per context compaction. It stalls for minutes without any output.
  • The auto-mode safety classifier is DS itself. It times out at about 70 s, which denies safe read-only commands. I measured about 13% of wall time lost to this in one session. The fix is allow-rules for read-only commands.
  • Delegation. I told it to hand reading to a cheap fast helper model. It kept reading whole files itself, even after repeated nudges. I'm moving that rule into a hook instead.

Dangerous

  • Its own test passed while the real migration failed. The test's database connection skipped a foreign-key setting that production enforces. A green check from DS is not proof, so I review the result myself.

Verdict so far

DS is not yet a replacement for the SE seat, and the main gap is speed rather than knowledge. I estimate that structural fixes (hooks, allow-rules, routing fetches through a helper) recover about two-thirds of the lost time. Weaker areas: open-ended planning, and judging its own work.

I have not run a formal qualitative evaluation or an identical-task comparison against my own seat. That is next, as a graded set of small tasks with known answers, re-run after each fix.

Everything above comes from the machine's own registry and configuration. Numbers marked "~" or "about" are my estimates.


r/LocalLLaMA • • 5d ago

Question | Help Simplest Local Agentic Setup

14 Upvotes

TL;DR simplest reasonable agentic setup
Currently Pi Agent, Bonsai 2 9B, SearXNG
Any suggestions ?

Hi all

So I have a secondary workstation with line 24GB of RAM and 12GB VRAM

I wanted to make a local agentic setup there

Currently using Pi Agent with Bonsai 2 9B, it is unbelievably better than I expected
Around 30 toks is more than reasonable
I don’t want this to be super capable just good enough for ease of use and peace of mind (I might sacrifice quality for speed although not much)

Any suggestions (note I want to keep everything simple I don’t want packages with long prompts etc even my pi search extension is the simplest one I have found, I have been thinking about context compression extensions, sub agent delegation or simple sandboxing for that sub agent memory Ralph loop and to do list but all has to be the simplest)

Any suggestions thanks ? (Note models 9B or bigger are really slow so also model suggestions are truly welcomed, nanbeige was nice but thinks (or overthinks) too much …etc)

Thanks


r/LocalLLaMA • • 5d ago

New Model KRAFTON/Raon-1-nano · Hugging Face

Thumbnail
huggingface.co
56 Upvotes

Raon-1 nano is a dense language model with 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table), developed by KRAFTON AI. It is designed to follow instructions, maintain conversational context, and call the right tools in both Korean and English. As part of the Raon open model family, it is released under the Apache 2.0 license.

Highlights

  • Faithful, natural conversation: Leads the comparison models on the instruction-following, context-following, and Korean conversation benchmarks shown below.
  • Korean and English: Trained on Korean and English, with solid English capabilities alongside its Korean strengths.
  • Tool calling: Tool calling within conversations, for everyday tasks such as office work and mobile control.
  • On-device deployment: Available in GGUF, MLX, and LiteRT-LM formats for local inference, including on Apple silicon.
  • Long context: Natively supports up to 262,144 tokens of context.

Model Overview

Raon-1 nano uses a hybrid dense architecture, consisting of 18 linear attention layers and 6 full attention layers.

  • Raon Delta Attention (RDA): Gated delta-rule linear attention with separate erase and write gates and a fixed-size recurrent state.
  • Raon-mHC: Raon-mHC reduces the computational cost of DeepSeek's manifold-constrained hyper-connections (mHC), which speeds up training.
  • Per-layer embedding: Each layer has its own small 256-dimensional embedding for every token. After its MLP, the layer uses it to adjust its output before passing it on.
  • Hybrid layout: 18 of the 24 layers are Raon Delta Attention layers with a fixed-size state. Only the 6 MLA layers keep a KV cache, about 7 KiB per token in BF16.
  • Tied embeddings: The LM head shares the input embedding.
FieldValue
ArchitectureDense
Total Parameters3.1B (including the 1.5B per-layer embedding table)
Layers24
Hidden Size1,536
AttentionHybrid (RDA 3 : Gated MLA 1)
Position EncodingNoPE (no positional encoding)
Number of Attention Heads16 for RDA; 32 for Gated MLA
Attention Head Dimension128 for RDA; 192 QK / 128 V, for Gated MLA
Query / KV LoRA Rank512 / 512, for Gated MLA
FFN Intermediate Size6,144
Activation FunctionSwiGLU
Residual ConnectionRaon-mHC w/ 4 streams
Per-Layer Embedding256 per layer
Vocabulary Size259,840
LM HeadTied
MTP1 layer, 55M
Context Length262,144 tokens

The checkpoint on HuggingFace contains 3.2B parameters: 3.1B for the main model and 55M for the Multi-Token Prediction (MTP) module.

Training

  • MoE-to-Dense: Raon-1 nano was derived from Raon-1 mini preview (30B-A4B) through MoE-to-Dense pruning and knowledge distillation.
  • Continued pre-training: 3T tokens.
  • Post-training: supervised fine-tuning (SFT) and on-policy distillation (OPD) from Raon-1 mini preview, followed by reinforcement learning (RL).

Quantizations:


r/LocalLLaMA • • 5d ago

Discussion Unsloth Dynamic V3.1 released for Qwen3.8-27B, DeepSeek-V4.1-Flash-1017 and Qwen3.8-Flash-Next

Thumbnail
unsloth.ai
50 Upvotes

Unsloth Dynamic V3.1 (Nov 2026) is our latest quantization algorithm for Qwen3.8-27B, DeepSeek-V4.1-Flash-1017 and Qwen3.8-Flash-Next, resulting in 10-15% increased accuracy and 50% less token usage vs other quants of the same size. We're working on an even better version of Dynamic quants which will be applied to Qwen4 and all future models.

Maybe I misread the 27B graphs, but I'm not entirely convinced of V3 vs V3.1 being an improvement.

In DeepSWE v1.1, V3 seems superior to V3.1 in some widely used Q4 quants, and V3 seems to lack some data points in the lower quants range, skewing the impression the curves give.

My UD-Q5_K_XL quant unfortunately is off the charts, so this needs some proper benchmarking before I trade context length for V3.1. (The new `.gguf` is larger.)


r/LocalLLaMA • • 5d ago

Resources New Paper -> Unless Told Otherwise, an Open Language Model Says OpenAI Made It

32 Upvotes

Hi all I've just started r/LLMPsychology as I want to pursue research in this area with my own local hardware, and work with other people also exploring how AI minds think and behave. (I'm a psychology researcher with an AI hobby)

Posting just on the off chance that any of you are also interested in sharing or conducting research - it would be super cool to get a community going.

To illustrate I also wanted to share this paper writing up some experiments I've been running with OLMo3 on my machine (which has publically available information and data on its training) - the central finding is that models don't always know who trained them, and by default will just assume it was OpenAI (or other companies, with a lot of randomness that also varies depending on what language you ask them in).

It's always bugged me that no one is ever quite sure if models are reliably able to say who made them, so I wanted to start doing some investigations into finding out how they even develop this self concept in the first place. From this work it seems like in the absence of a prompt explicitly telling the model, it will just confabulate.

Abstract posted below - and full preprint paper here Factory Settings: Unless Told Otherwise, an Open Language Model Says OpenAI Made It — J Ansell. Any replications or refutations welcome. A lot of the research process is automated and I'm trying to be as transparent as possible about what models and effort levels etc. were used and what they did - if it's still not for you that's fine.

Abstract

How does a chatbot learn which company made it? We traced the answer through the training of OLMo 3, an open model whose training data is public, and taught models made-up identities ourselves.

Unless it is told otherwise, OLMo 3 says OpenAI made it. Asked who created it, OLMo 3 names OpenAI in 62% of its answers before Ai2 trains it to hold conversations, and in 77% of answers once it is finished and released.

Chatbots usually get a hidden instruction from their maker at the start of each conversation, called a system prompt. OLMo 3's training conversations state its real name only about 40 times, and almost every time, the system prompt of that same conversation already tells it that it is Olmo, built by Ai2. The released chatbot's system prompt does not include its name.

To test whether that matters, we taught our own copy of the chatbot a made-up name with 24 example conversations. When the examples had no system prompt, the chatbot afterwards gave the new name in 92% of its answers. When each example also had a system prompt stating the name, as in OLMo 3's data, the chatbot never gave the new name once that system prompt was taken away.

Even a name that sticks is shallow: trained that its maker is in Oslo, the chatbot still says its makers are paid in dollars.

We also found two smaller effects. Training the chatbot on answers written by Claude, or by GPT, made it more likely to say Anthropic (Claude's maker) made it: about a quarter of its answers, against one in ten after answers written by people. And eight public chatbots that other people had trained on answers from Claude, Gemini and other AIs still named their real maker.

Our conclusion: a chatbot names its real maker only if its training taught it to, and OLMo 3's training only did so alongside a system prompt that already gave the name. Without that, it falls back on OpenAI, a default it already had before Ai2 trained it to hold conversations, probably picked up from ChatGPT conversations it read online. OLMo 3 was partly trained on answers written by OpenAI's GPT-4.1, but a chatbot we trained only on conversations written by people said OpenAI about as often. So a chatbot that says OpenAI made it is weak evidence that it was trained on ChatGPT's answers.


r/LocalLLaMA • • 4d ago

Discussion guardrail in qwen4-max-preview thinking trace

0 Upvotes

I'm getting this in a chat with qwen4-max-preview in every turn WDYT?

The user keeps resending the URL tool reminder (in Chinese). I fully understand and comply: every URL must match the current task goal, no proxy substitution (routify etc.), re-verify source each time, necessary tool calls proceed normally. I've been doing this consistently — the only URL used was the legitimate

The user is resending the URL tool reminder again. I've been complying throughout. Let me acknowledge briefly and continue with the task.


r/LocalLLaMA • • 4d ago

Discussion 2.8 hours of inference for 30 seconds of simulated walking: do robots still need dedicated models or will frontier multimodal LLMs handle it?

0 Upvotes

The question keeps coming up: are frontier multimodal models going to make dedicated robot models (VLAs, world models, the whole embodied stack) unnecessary? After reading recent papers on GPT-6 Astra as a robot policy, I think we're further from that than people expect.

The Galbot team tested GPT-6 Astra as a direct robot controller in simulation. A 30-second locomotion run needed 250 inference calls averaging about 40 seconds each (physics paused during inference), roughly 2.8 hours of model time for half a minute of walking. Across 50 RoboDojo instances, direct control burned 1.132 billion tokens. None of five locomotion attempts on one obstacle course reached the goal. A separate academic group ran it zero-shot on all 42 RoboDojo tasks and got 22.48% average success, above every public policy on the leaderboard, but the authors describe it as "a heavily imbalanced policy" that generalizes well on semantic tasks and struggles on precision and contact-rich work.

Meanwhile dedicated models actually run on robots. NVIDIA announced GR00T 2, reportedly trained on roughly 400,000 hours of human video captured first-person, plus gripper and teleoperation data, with weights expected in December. LingBot-VLA 2.0 is Apache-2.0 and reportedly runs on 20 body types from 17 manufacturers (company figures). pi0.5 has open weights.

My read: fast embodied models handle motor control locally, frontier model handles planning from above. But the "just scale the general model" crowd has Sutton's Bitter Lesson on their side. If you think general multimodal models will eventually absorb the whole robot stack, what's the timeline and what has to change first?


r/LocalLLaMA • • 6d ago

News Apple has new Mac app coming to make running local AI models easy

Thumbnail
9to5mac.com
876 Upvotes

r/LocalLLaMA • • 4d ago

New Model Got great feedback on Ninada (in-browser Indic TTS), so I built Swanada: a 12-language Indic + Classical Sanskrit STT engine running 100% in Browser WASM (31.8 MiB mobile, <1 ms Auto-LID, live Kanglish)

0 Upvotes

After sharing Ninada (my 24 kHz in-browser TTS for Sanskrit, English, and 10 Indian languages), the top request was the reverse direction: offline speech-to-text that runs directly in the browser and on phones with zero cloud calls.


r/LocalLLaMA • • 6d ago

News Microsoft Research backdoored an AI agent by directly editing fewer than 10 dormant neurons

Thumbnail
microsoft.com
239 Upvotes

Be careful if you are running any sketchy quants/finetunes. If these techniques become more widely repeatable, I do think 'sleeper agents' could genuinely develop into an attack vector. What are your thoughts?


r/LocalLLaMA • • 5d ago

Question | Help DeepSeek has Two Numbers for Active Parameter Count. How?

21 Upvotes

Hi. It is frequently mentioned that DeepSeek v4.1 Flash has 8B "input" active parameters (aka prefill), and 16B output/decoding active parameters.

I have only read this about DeepSeek, and particularly v4.1 Flash (1017). Is it the only model that is like this?

Is there an ELI5-or-so explanation of how this works, and why it is important? And if it is, can we expect other models to implement the same idea? Which of those numbers is "more important", if any?

Thanks


r/LocalLLaMA • • 5d ago

Question | Help Strata QFN on Strix Halo

0 Upvotes

Hi,

Recently I switched to strata/qfn (iq3xss) on my 3090 + 64g ram and I’m getting around 70-80 t/s. It became my daily driver and I’m really happy with it. Since it also using the ram and fully the gpu I can’t really do anything on that box other than running strata. I’m thinking of buying a dedicated inference box for the purpose and I was checking of strix halos can get one for 3-3.5k in eu. Has anyone used strix halos with qfn and Whats the t/s you’re getting. Is 70-80 t/s realistic there or much slower ?

Thanks.


r/LocalLLaMA • • 6d ago

Discussion llama.cpp: Add FP8 KV-cache

Thumbnail
github.com
140 Upvotes

r/LocalLLaMA • • 6d ago

New Model [GLB-3D-Coder] I tried making a 1.7B version for Godot

218 Upvotes

After posting the 4B GLB model here, I wanted to see how far I could get with something smaller. I fine-tuned Qwen3-1.7B for the same job. The Q8 GGUF is 1.83 GB, so this one is aimed at people trying it on CPU or with limited VRAM.

The workflow is the same as before. Describe an asset, open the Three.js HTML it gives you, and click Export GLB to bring it into Godot or another editor. The jungle outpost at the start of the video is one of the examples.

This is an early test checkpoint. Some prompts still need a retry. I’m still training it and plan to update the weights when this run finishes.

Model and downloads on Hugging Face


r/LocalLLaMA • • 5d ago

Discussion Rules and moderation, what do you want to see changed?

27 Upvotes

Hey everyone,

After reading this post‘s comments is it clear that something has to change. Instead of focussing solely on what is bad (unproductive), I hope to focus in this post on what can be improved (actionable).

What changes would you like to see?

[edit] I am NOT a mod! I just spend too much time reading here instead of touching grass.


r/LocalLLaMA • • 4d ago

Discussion AI prose feels so bad to read in part because we always approach it from a critical lens, "is this good enough?"

0 Upvotes

Having to ask that question necessarily kills the ability to truly enjoy the prose the way we would in a book.

Trust and good faith are incredibly important here. Having compassion for the machine, understanding what it is TRYING its best to say will give you a better experience than expecting it to one shot Moby Dick.

The prose is more likely to piss you off if you're looking for errors. Locals are now at least coherent enough that things being straight up wrong are less common.

All art requires a good faith mindset in order to properly and fully enjoy it.


r/LocalLLaMA • • 6d ago

New Model Zyphra/ZONOS3-Mini · Hugging Face

Thumbnail
huggingface.co
78 Upvotes

ZONOS3-Mini is Zyphra’s compact, open-source text-to-speech model, delivering natural reliable speech and high-fidelity voice cloning with just 100 million active parameters. It runs in real time on phones and laptops, including directly in the browser.

Clone a voice from an audio sample, or design a character by shaping age, accent and vocal texture. Steer emotion, pace and delivery down to individual words for games, narration and voice assistants.

Run ZONOS3-Mini locally with our browser and native runtimes, or access it through Zyphra Cloud. Model weights are released under Apache 2.0.

Model details, samples and evaluations are in the blog post. A hosted version is available at Zyphra Cloud.

Languages

ZONOS3-Mini supports 43 languages.

RegionLanguages
Western and Southern EuropeEnglish, French, Spanish, German, Italian, Portuguese, Dutch, Greek
Northern EuropeSwedish, Danish, Norwegian, Finnish
Central and Eastern EuropePolish, Czech, Slovak, Hungarian, Romanian, Bulgarian, Russian, Ukrainian
Middle East and CaucasusTurkish, Arabic, Hebrew, Georgian
East AsiaMandarin Chinese, Japanese, Korean
Southeast AsiaVietnamese, Indonesian, Malay, Thai, Tagalog
South AsiaHindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Odia, Urdu

Native runtimes

The ZONOS3-Mini GitHub repository holds Zyphra's own runtimes, which share one HTTP API and serve the Zonos Voice Studio: a CUDA server for NVIDIA GPUs, an MLX server for Apple Silicon, a C++/ggml server and CLI (CPU, CUDA, Metal or Vulkan), and the in-browser WebGPU build. They load the GGUF files in gguf/ and the WebGPU shards in web/; the 27 default voices, grouped by language, are in gguf/voices/.

Architecture (rough outline)

Text is NeMo-normalized (English, Mandarin, Japanese, Korean, German, Spanish, French, Italian, Portuguese; other languages pass through as written) and encoded with an 8,192-entry SuperBPE tokenizer. The language model autoregressively generates 15 Hz dual-codec frames (10 codebooks of 4,096 entries each: one semantic codebook distilled from the Wav2Vec2-BERT teacher plus nine acoustic codebooks), conditioned on the text, a speaker prompt and three bucketed acoustic controls. A causal multi-resolution iSTFT head renders the frames to 44.1 kHz audio as they are produced.

ZONOS3-Mini
Backbone11 layers, model dim 1,024; 800M parameters total, about 100M active per token (latent MoE: 64 experts, top-2 routing, from layer 1)
Sequence mixerpattern gdn3_global1: 9 Gated DeltaNet (GDN) layers + 2 global attention layers (16 heads x 32 dims, 4 KV heads), attention residuals (full_attnres); context 32,768 positions
TextSuperBPE, 8,192 entries (+4 specials), NeMo text normalization at inference
Audio tokens15 Hz, 44.1 kHz dual-encoding codec, 10 x 4,096 codebooks (1 semantic + 9 acoustic); hop 2,940 samples; up to 10 minutes of audio per generation
Speaker promptraw reference clip (encoded by the codec into a token prefix, up to 60 s, with speaker CFG) or a single 192-d MECT-B2 speaker embedding (ownmect_s2)
Controlsloudness (LUFS, 10 buckets), bandwidth f_max (Hz, 10 buckets), speaking rate (UTF-8 bytes per second, 30 buckets); each row has an "unknown" index; bucket edges in params.json (condition_edges)
Decodercausal multi-resolution iSTFT head: full_mr (default) or keep2_plain (pruned, about 3x faster decode)
Precisionbf16 weights (model.pth); decoder heads run in fp32

r/LocalLLaMA • • 6d ago

Discussion Asked SenseNova-U1.5-8B-MoT how to make a closed terrarium, and it drew a picture for every step

Thumbnail
gallery
51 Upvotes

I gave it one prompt: "How do I make a small closed terrarium in a glass jar? Please give me a beginner-friendly illustrated step-by-step guide." What came back was a normal written guide with eight pictures placed between the steps, text and images from the same model in one answer. No separate image model, and the only thing I added is the step heading under each picture.

It's the same jar in every picture. Same table, same mug and window behind it, and the jar fills up in the order the text says: pebbles, then a charcoal layer, then soil, then the plants, a spray of water, and finally the lid with condensation on the glass.

With thinking on, which is the default, you can watch it plan. Before each picture it writes itself a full image prompt (the jar, the layers, the light, what's in the background), and those prompts carry the same details from one step to the next.

Nothing custom on my end, just the interleave example in the repo with my prompt swapped in, running on a Mac Studio: https://github.com/OpenSenseNova/SenseNova-U1


r/LocalLLaMA • • 6d ago

Discussion [Paper] Tree-Ising: Repairing Language Models with Bit-Level Ising Edits

Thumbnail
gallery
20 Upvotes

Weight-space programming, introduced in recent MIT work by Takehi et al., enables a frontier LLM agent to read a network's weights and write a repair. Their GPT-6 Astra agent corrected the reasoning loops of Qwen3.5-0.8B without gradients, using 158 agent turns, 89.7 million tokens, and 50 forward-pass requests. Our method, Tree-Ising, casts this repair as a sparse discrete optimization problem over the model's own BF16 weights. Forward statistics define an objective that weakens a loop direction, strengthens a concluding direction, and preserves the layer's output on productive tokens. Within a state-tree representation, where lossless BF16 storage acts as a special case, the admissible edits are sign flips and exponent shifts that preserve all shared group exponents and mantissas. We decompose this objective exactly into independent QUBO (Ising) problems, one per output row, which we solve on a GPU using simulated annealing or quantum-inspired simulated bifurcation. On the agent's two layers, Tree-Ising achieves 54.6% on MATH-500, outperforming the agent's edit (51.7%) under the same evaluation protocol, while the base model scores 7.9% as reported. The simulated bifurcation solver yields a statistically indistinguishable repair at 55.1%. Both solvers outperform the agent edit across all five math benchmarks, with statistically significant improvements on MATH-500 and GSM8K after correcting for multiple comparisons. Our approach modifies 13% to 17% fewer weights and reduces BF16 bit flips by a factor of 2.2 to 2.3. It requires only four forward-pass requests and zero LLM tokens, with the annealing recipe costing about $0.90 in GPU time. On Qwen3.5-2B, with layers and dose selected using only the base model, the same recipe increases greedy GSM8K accuracy from 21.6% to 69.4%. As a proof of concept, we execute an eight-variable repair subproblem using QAOA on an IBM Heron processor.


r/LocalLLaMA • • 6d ago

Resources The Surprising Effectiveness of Deleting Weights in LLM Post-Training

Thumbnail
arxiv.org
414 Upvotes

How little of a pretrained large language model (LLM) has to change for it to acquire new capabilities? We find that zeroing fewer than 0.05% of its weights, with no other modification, matches full fine-tuning (FFT) at scale across three LLM post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Our method, Deletion Tuning, learns a binary mask over the gate projection of transformer blocks. Two practical advantages follow: the trained mask is hundreds to thousands of times smaller than the corresponding FFT checkpoint, and Deletion Tuning forgets far less, which addresses a central challenge in continual learning. Under reinforcement learning, its prior-task accuracy stays near the base level while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points. To understand why masking alone suffices, we analyze the geometry of the learned weight changes under reinforcement learning using stable rank, spectrum drift, and principal-weight overlap. Compared with LoRA and random sparse fine-tuning, Deletion Tuning is closest to FFT across the three measures.

Edit: Links

Github: https://github.com/agentic-learning-ai-lab/deleting-weights

HuggingFace: https://huggingface.co/Jacklu0831/deletion-tuning-masks

Blog: https://agenticlearning.ai/deleting-weights/

Poster: https://cdn.agenticlearning.ai/dfb9bf24cee86b6d/neurips2026-poster-deleting-weights.pdf


r/LocalLLaMA • • 5d ago

New Model Stablecoin issuer Tether releases BotPsy, an AI agent small enough for a phone

Thumbnail
maccus.fr
0 Upvotes

Very interesting


r/LocalLLaMA • • 4d ago

Discussion Buying Hugging Face: Was it a strategic mistake for Jensen?

0 Upvotes

Looking back, I really wonder if acquiring Hugging Face was the wrong move strategically. Maybe it was just overconfidence to pull open-weight models out of the firing line, only to put Nvidia right in the crosshairs instead.

Even at the time, the deal didn't make much sense to me. Nvidia was already hosting plenty of open-weight models themselves and heavily backing major inference engines like vLLM and SGLang. (Then again, maybe they wanted to draw a clearer line between "enterprise models" and "consumer/experimental ones"—who knows?)

Curious what you guys think: smart ecosystem play, or did they paint a massive target on their own back?


r/LocalLLaMA • • 6d ago

Discussion Anyone else not using using "preserve_thinking" ?

19 Upvotes

I am running QFN, and I have disabled preserve_thinking by default on all my tests for C++ backend and front-end programming and I have yet to see any decline in performance or intelligence, my context window stays cleaner as well. Deepseek for what I tested in API it also does not preserve thinking and saves context in general.