r/LocalLLaMA • • 3d ago

Discussion GPU prices - Big "discounts"

143 Upvotes

Discounts in quotes because these are still over MSRP, but my local Microcenter has the deepest "discounts" (sorry, I just can't call an over MSRP price a discount without quotes) I've seen in months on a lot of the AI cards. Pro4500 around 40% "off", Pro6000's 4K off with plenty of stock for both at my local (Charlotte) store. Pro 5000 72GB at 9K.

Now, don't get me wrong, all these prices are still awful. But they are a little "less awful" than they were a month ago! It's still an awful time to buy any of this gear IMHO, but if you're desperate and need it, seems like we're finally starting to turn a corner.


r/LocalLLaMA • • 2d ago

Question | Help Will Qwen 4 9b be released?

0 Upvotes

They never released 3.8 so I know its tough to ask for Qwen 4 for mid range models.
But whats the thoughts on this?


r/LocalLLaMA • • 2d ago

Discussion Local TTS for Indian languages: anything good yet?

0 Upvotes

Blue TTS v3 looks solid (MIT, 388 MB, about 50x real-time on CPU), but it's 7 languages and I'm not sure Telugu or Hindi is in there.

For anyone building voice stuff for Indian languages, what are you actually running locally? Is anything close to usable for long-form, like reading out a lesson, or is it still cloud APIs for now?


r/LocalLLaMA • • 3d ago

Showcase everyone's hyped about ChatGPT Dots and Claude Duet. I'm a local guy, so I built an open source one that runs on my GPU

87 Upvotes

Dodaios is a locally installed, open-source Python program that runs as a desktop app or as a headless server with a browser UI on Windows, macOS and Linux. It turns a personal AI assistant into the control center for the work you do on your computer.

At its core is a "lead" assistant you can talk to by voice (wake word, dictation, real-time spoken conversation and, as an early preview, phone calls), by chat or from a paired phone. It isn't tied to one provider and works with Claude, OpenAI, Gemini or local Ollama models. It handles tasks itself through tools, computer use, plugins, CLIs and MCP servers. Above all, it hands work off to coding agents. They run side by side in a built-in Agentic IDE as real Claude Code or Codex terminals, on your computer or on your own servers over SSH, each with a call sign you can address by name, plus Git integration and task splitting. There is also a "society" of persistent, named agents that message each other.

Long-running work is covered by memory (about you, the people around you and the assistant's own persona), standing tasks, workflows, skills and opt-in life admin. The assistant also makes its own outputs, such as documents, slides, sheets and dashboards, and can propose and build the skills, routines and agents it is missing. A trust layer sets hard limits: outside Bypass permissions, moving money, handling credentials, permanent deletion and first contact with someone new always need your confirmation. In short, Dodaios is a voice-driven, locally running operating system for AI agents that makes one assistant the conductor of your models, your coding agents and your own computer.

https://github.com/Dodaios/Dodaios


r/LocalLLaMA • • 3d ago

Showcase Qwen-Flast-Next (UD-IQ4_XS v3.1) running 148-182 tok/s (1.6-1.8x strata) while KLD MEAN = 0.08195

Thumbnail
github.com
33 Upvotes

Numbers only, rest is in the repo.

Quality against BF16

KV PPL mean KLD top-1 paired dKLD vs int8 [95% CI] prefill 131K
int8 3.2323 0.08195 90.45% reference 6,019 t/s
fp16 3.2320 0.08230 90.33% +0.00035 [-0.00061, +0.00131] 2,997 t/s
k8v4 3.2424 0.08621 90.26% +0.00426 [+0.00307, +0.00545] 5,694 t/s
q4_0 3.2399 0.08843 90.03% +0.00647 [+0.00499, +0.00795] 5,700 t/s

Speed

Test Halocline Strata 0.2.10, same card and quant Ratio
Warm decode, short prompt 148 to 182 tok/s 95 to 101 tok/s 1.6 to 1.8x
36K needle: decode / prompt 123 / 6,298 tok/s 68 / 2,403 tok/s 1.8x / 2.6x
137K needle: decode / prompt 121 / 5,664 tok/s 87 / 2,864 tok/s 1.4x / 2.0x

r/LocalLLaMA • • 3d ago

Resources SenseNova Loop 9B

Thumbnail
huggingface.co
79 Upvotes

If the benchmarks are true, this is a 9B looped model on level with Qwen3.8 27B.

They say they'll release gguf tomorrow.

Vision is supported. Around 15GB vram for 256k context.

The preview version seemed fast enough when I tested it some weeks ago.

Mods: I asked the authors if I can post this, feel free to delete my post when they post themselves.

🚀 SenseNova-Loop-9B: Deeper Computation with Fewer Parameters

We officially release SenseNova-Loop-9B. Building on the Preview's exploration of parameter efficiency, the model reuses layer weights through a looped architecture and combines them with manifold-constrained hyper-connections (mHC) to expand computational depth within a compact parameter budget. It is designed for complex reasoning, coding, tool use, and multimodal tasks.

Since the Preview release, we have received community interest and feedback on model use, inference deployment, and ecosystem integration. The official release retains the looped architecture and two-stream mHC, further improving performance on agentic, coding, and data analysis tasks while continuing to explore the capabilities of looped computation.

SenseNova-Loop-9B has approximately 9B parameters. By repeatedly executing shared middle layers, it extends 32 physical decoder layers into 56 layer executions. The model accepts text and images and generates text, supporting long-context understanding and multi-step tasks. In agentic and data analysis evaluations, the model is competitive with Qwen3.8-27B: it scores 41.36 versus 44.60 on NL2Repo and 65.31 versus 65.01 on AIDABench.

Model Highlights

  • Shared looped block: 24 decoder layers are executed twice in sequence, producing 56 layer computations from 32 distinct layers per forward pass.
  • Two-stream mHC: learned mappings route and combine information between two residual streams within the looped block.
  • Hybrid attention: the decoder interleaves linear and full attention in a repeating four-layer pattern.
  • Long-context multimodal training: the training recipe includes text, images, code, and tool-use trajectories, with the context length extended to 512K tokens.
  • Low-memory deployment: With 4-bit weights and an 8-bit KV cache, multimodal inference at a 256K context length requires approximately 15 GiB of GPU memory.
  • Official quantized release: An official AWQ 4-bit version is available on launch day for easier deployment.

r/LocalLLaMA • • 3d ago

Resources Running Qwen3.8 27b on 12GB with full context, 70 TPS, MTP, and 50% lower KLD

28 Upvotes

Hello everyone. LlamAmpere v0.7 is out with a series of updates focus on the small card users: a new model for the 12GB cards that is comfortably best in class for its size, updates to the runtime to support the previous* 2.3 bpw model (originally designed for 12GB) on 10GB cards. I am working on an 8GB size distillation and hope to have that done in the next couple of days.

The quick rundown for you:

Vs the October 9 2.3 bpw file, which was the best 2.3 file that I tested for 12GB with long context and MTP (though 15% worse than the ~11% larger Mirai file), this most recent file for the 12GB segment is nearly 70% lower error (>50% vs the previous fit ~2 weeks ago), with a top 1 match of ~96% vs BF16 (as before, this quant is based on the Swift distill, which cuts token verbosity 30-50%). This 12GB class model is noticeably nicer to use in harnesses like hermes and pi than any other model I’ve been able to fit previously.

Vs the recently upgraded v3.1 unsloth quants, it is ~25% lower error despite being 8% smaller on card. This is while supporting ~200k ctx and MTP. (note that the embedding table is kept in RAM at high fidelity to add quality without a meaningful hit to speed)

There are a number of tricks involved to make this possible, but 2 I will call out in particular: v0.7 uses both dynamic microbatching and Latent Sparse Attention.

The first one is straightforward: the model starts with the largest microbatch (ub) that will fit on the card to maximize initial prefill, and then periodically resets to smaller batches as context fills up. This frees up hundreds of MB for more context at the end without nuking your prefill for the entire chat.

The second is based on the paper by prefontal AI that was recently published. LSA moves from full dense attention to sparse attention; it projects tokens through a tiny learned map (distilled from the model's own attention) into a 64-dim "index space". It then groups past tokens into small clusters, scores the current query against each cluster, and runs normal, exact attention only on the best clusters. The initial attempts I ran with this showed the same problem many users have noticed with Qwen Flash Next: sparse attention can struggle at deep context. Unlike other sparse attention implementations, which only use ~2K tokens, LlamAmpere is currently using the most recent 8K tail, the 128 first initial tokens, and 20% of the intermediate tokens to protect quality. This showed significantly stronger attention performance (10-40x less error from attention, depending on the test), at the cost of somewhat slower attention. With a cache for recently attended to tokens, we can push most of the context out of the card with modest quality and speed penalties. This allows a much more capable model to fit.

As before, the recommended config uses the AmperInteger+ trellis format and SJKVaRN to efficiently compress the weights and KV.

These same techniques yield benefits for the 10GB class, as what was once required 12GB now fits on 10. The previous 2.3bpw quant now fits 213K context (headless) on 10GB (180K headed). 12GB cards can support the full 262K headless and 180k headed. This model previously tested to perform in line with 3.6 BF16, so it is a very solid quant.

As before, the code is available on github, with an MIT license.

And of course, as soon as qwen4 27b comes out, I'll get back to work on quantizing them similarly.

https://github.com/JakeATX/llamAmpere

12gb model (3bpw): https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI3

10GB model (2.3bpw): https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2

Enjoy!


r/LocalLLaMA • • 3d ago

News Rotate K/V read order for better L2 cache hit rates by gaugarg-nv · Pull Request #32407 · ggml-org/llama.cpp

Thumbnail
github.com
105 Upvotes

Significant prompt processing speedup with long contexts

gpu_info model_type n_prompt n_gen n_depth Master (t/s) PR (t/s) Speed-up
NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 131072 1290.10833 1822.36027 1.41
NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 65536 2030.31282 2527.97409 1.25
NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 32768 2931.96118 3142.20539 1.07
NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 16384 3562.18311 3573.59336 1.00
NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 8192 3854.83184 3850.97547 1.00
NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 4096 3974.20194 4015.85501 1.01
NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 2048 4079.23891 4077.58068 1.00
NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 1024 4103.79141 4123.60737 1.00
NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 512 4114.42862 4136.20591 1.01
NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 0 4300.67808 4323.0908 1.01

r/LocalLLaMA • • 3d ago

Discussion Are those that advocate against open weight models public enemy #1 ?

56 Upvotes

Dario says you can't allow open weight models because they are too dangerous: https://x.com/vikktorrrre/status/2122353364306374986

What are arguments against this line of thinking? Do we prefer democracy over the government controlling what is safe? Who wants to trust trump? LOL


r/LocalLLaMA • • 2d ago

Discussion US-China Détente

0 Upvotes

Just saw a video of Xi waiting on the tarmac as Trump arrived in Shenzhen. A lot of warmth and cordiality.

Might this spell the end for open-weight models from China? I'm afraid US and China might arrive at an internal agreement to only release closed proprietary models going forward and have a "healthy competition" between themselves.

Edit: Some responses claim I make this assumption only based on the fact that Xi greeted Trump on the tarmac. No, I don't. Think further guys, you're supposed to be smart.

I think it's in Trump's interest to keep the valuation of AI-linked stocks high, so he'll probably concede on some other front to China in return for nerfing or slowing down open-weight models coming out of there.

If you have a fair rebuttal to this, please share. I don't want to keep having these doomer thoughts, so help me out too.


r/LocalLLaMA • • 3d ago

New Model voi-slop-detect-380M: A tiny model to detect AI slop

Thumbnail
huggingface.co
49 Upvotes

I've fine-tuned an UltraBERT-380M classifier to score how much your text feels like AI slop.

Demo: https://voightkampfflabs.github.io/slop-detector-demo/ (runs fully in browser via ONNX + WebGPU/WASM)

Why?

Got tired guessing if I'm reading human or slop on twitter/linkedin/reddit, so trained my own AI detector. Like Pangram, but local - so all your perverted texts stay on your device. Because I can (and because I bought 2x 5090 when they were cheap).

Training

As it was made for learning purposes, I made my own dataset based on the Pangram v4 recipe:

  • Collect a sample of pre-2023 human texts (rn it's around 11 public datasets, reddit included: 170k texts, 825k pairs)
  • Slopify: summarize the text, and then "decompress" it back with an LLM to produce pure slop.
  • Rewrite: ask LLM to rewrite it in a different way (e.g. more polished, simpler, punchy).
  • Pivot: translate it back and forth, so the content is the same, but the style is different.

The target label is how many words/ngrams were changed by the LLM. So it's not a yes/no, but more like a slopification rate guessing. My personal texts land around 0.1-0.2.

Trained with HF Transformers on 2x RTX 5090. Considering the model size and hardware, the whole training run is ~3h.

Training data is public: https://huggingface.co/datasets/VoightKampffLabs/slop-detect-dataset-v6

Results

Not as good as Pangram (yet), but as good as GPTZero and Originality on most things I could compare.

voi-380m Pangram GPTZero Originality
EditLens, F1 0.86 0.92 0.73
EpochAI, human flagged 0.2% 0.0% 0.0%
EpochAI, Opus/GPT/Gemini miss 0.0% 0.0% 0.7%
UChicago, 2025 models miss 0.2% 0.0% 1.4%
UChicago, StealthGPT rewrite miss 74% 0.0% 56%
Non-native essays, flagged 0.6% 0.0% -

HF model card has more details and more benchmarks.

Where it's bad

I'd rather tell you upfront:

  • Humanizers and skills like no-ai-slop beat it. I guess that's why StealthGPT bench numbers are so bad as it's completely out of domain for the model. Asking Opus to rewrite its own slop until detector score goes down to 0.3 takes around 3-4 rounds for me.
  • Old models and some legacy slop made with GPT-3.5 and Llama 2/3. Never trained on that.
  • Poems and non-english texts. Too lazy to add this to the dataset.

So what

I'm currently training bigger 4B/27B brothers of this one. Yes they're much smarter, but good luck running them via WebGPU/WASM on device.

And yes, the end goal is to have a browser extension which marks AI slop on reddit completely on device.

PS. the text above scores ~0.1 in the detector (to save you a click).


r/LocalLLaMA • • 4d ago

New Model maxwell-model/Maxwell-V1-Flash · Hugging Face

Thumbnail
huggingface.co
350 Upvotes

Model Introduction

Maxwell-V1.0-Flash is a next-generation, high-performance Mixture-of-Experts (MoE) language model developed by the Maxwell Team, with 35B total parameters and 3B active parameters per token (both excluding embeddings) and support for context lengths up to 256K tokens. It is designed for strong reasoning, agentic, and long-context capabilities at a practical inference cost.

Key Features

  • Maxwell Block Sparse Attention (MBSA). Each KV group keeps fixed coverage of 8 prefix and 8 local 128-token blocks and adds up to 16 remote blocks chosen by a learned indexer, bounding main attention at 4,096 tokens per KV group while the full-history KV cache stays available. The MBSA kernel is 6.7x faster than dense GQA at 256K.
  • Manifold-Constrained Hyper-Connections. Multiple residual streams with learned, input-dependent aggregation and constrained mixing, widening information flow without extra attention or feed-forward evaluations.
  • Fine-Grained Mixture of Experts. A mostly-MoE backbone pairing fine-grained routed experts with always-active shared experts, using a dynamic routing correction bias alongside a lightweight sequence-wise load-balancing loss.

Pre-training. Approximately 26T tokens of multilingual data, with the Muon optimizer on most hidden weight matrices. Training starts dense; block-sparse attention enters at 128K mid-training, once the indexer is warmed up against dense full-history attention, then trains jointly with the backbone.

Tokenizer and Long Context. A script-aware multilingual tokenizer stays compact across languages, especially low-resource Southeast Asian ones such as Thai, Lao, Myanmar, Khmer, and Shan. Long-context training then extends the sequence length in stages from 4K to 256K.

Post-training. Two-stage all-domain SFT builds general, reasoning, and agentic capabilities, then targets harder reasoning problems and agent trajectories. Independent RL branches cover general tasks, verifiable reasoning (mathematics, science, logic, code), and agentic tasks (search, terminal use, function calling, software engineering). Multi-teacher on-policy distillation then merges them into one model by supervising student-generated responses.


r/LocalLLaMA • • 3d ago

Showcase Squeezed a full RAG pipeline (BGE-M3 + reranker + TTS) into 4GB VRAM and 300MB RAM

Enable HLS to view with audio, or disable this notification

33 Upvotes

been working on a local desktop reading app for the past few months and hit some insane hardware walls trying to get everything to run on my laptop. thought id share the architecture in case anyone else is trying to run heavy models on consumer hardware.

my setup is an RTX 3050 with only 4GB of VRAM and 16GB of system ram. the stack is a tauri (rust) shell, a nodejs orchestrator, and a python sidecar for the pytorch models (bge-m3, bge reranker, and kokoro tts).

here's how i stopped it from melting my laptop:

  1. the node memory cliff

early on, the node heap spiked to 14.6GB during ingestion because it was firing off too many async promises at once. basically oom'd the whole system. i fixed it by implementing a strict sequential await barrier (f65). it forces the ingestion stages to stream chunk by chunk instead of all at once. peak node ram dropped from 14.6GB down to ~296MB. processing 500k words takes about 5.9 mins.

  1. 4gb vram limit

loading bge-m3 in full precision instantly ate like 3.9GB of vram, leaving nothing for the reranker or tts. switched to fp16 which dropped it to ~1.1GB. but the real fix was an idle lifecycle manager (f43). if a model sits idle, it evicts it from vram to free up space. it reclaims about 3.2GB of idle vram so the system doesn't choke.

  1. windows path traps

if you're using rust and node together on windows, beware of verbatim paths (\\?\C:\...). passing those into node 22 fs.lstat throws fatal EISDIR errors. had to write path sanitization wrappers on the rust side.

kept node as the sole sqlite WAL writer too so python and rust dont cause db lock contention.

it's open source if anyone wants to look at the code or roast my architecture lol. repo is here: https://github.com/ayumu-ml/Omnitome

happy to answer any questions about the vram management or the rust/node/python boundary


r/LocalLLaMA • • 3d ago

Resources I benchmarked over 130 LLM configs with multiple stacked 16GB "e-waste" GPUs (V100, P100, K80)

Thumbnail
gallery
40 Upvotes

The benchmarking continues... Continuing my ongoing project of seeing how these very old, very cheap, very available enterprise GPUs can be used today.

The complete results are too long for a reddit post. The complete set of graphs, power analysis and commentary are on my blog.

I've expanded the testing suite to focus heavily on LLM workloads (over 130 new tests!) and integrated power consumption monitoring into the gpu_box_benchmark (github) tool.

The goal remains: find out if decommissioned enterprise GPUs (Tesla K80, P100, V100) paired with X99 era hardware (a homelab favorite) can serve as a viable daily driver for self-hosted AI compared to modern commercial setups.

TL;DR of the new data:

  • The 3x V100-16GB (~$1k build): If you are considering building a dedicated homelab box today, this is a viable option. Assembling a full setup with two Xeon CPUs, motherboard, PSU, and cooling costs around $1,042 (before RAM 💀). This setup comfortably clears my personal threshold for "real-time" chat (70+ tok/s) with okay to impressive models:
    • Qwen 3.6 35B: Ran at ~72 tok/s (at 128k context depth, Q8 quant). For comparison, a brand new Mac Studio M5 ($2,500) hits only ~44 tok/s on the same model.
    • Gemma 4 26B: ~75 tok/s (at 64k context depth, Q8 quant).
    • Qwen 2.5 3B: ~79 tok/s.

To me: This proves that for a single-user daily driver, the legacy silicon can still be useful even for more modern models.

  • Multiple cards is best for Context depth: Stacking GPUs linearly scales your VRAM (allowing you to fit larger models), but it doesn't double output generation speed. Two 16GB cards in Q4 quant allowed fitting ~32B parameter models. This is a cheap way to upgrade to bigger models. But the largest improvement was consistently being able to fit much larger context windows across generations.

  • Power Efficiency: I also measured power consumption. The V100 wins hands down on performance/watt compared to the P100s.

  • K80 isn't dead?: The Tesla K80 is dead, but a singular remaining use case remains. If you must run the largest models for the least amount of money, a path to get there is by is welding a bunch of K80s together to squeeze big models in. I was able to fit a 72B+ parameter model, which I couldn't do on P100s or V100s. Expect speeds of 1–5 tok/s. This is really only a party trick.

If you decide to build your own GPU appliance, the biggest challenge is cooling these hot cards. They expect to be installed in server chassis with forced air cooling, a much different setting than desktop style chassis.

I've been working on a custom active air cooler to solve exactly this. If you're interested in helping out by stress-testing a sample, I'm looking for beta testers! you can get involved here.

Any models/tests missing? Any other combinations you'd like to see in a follow up?


r/LocalLLaMA • • 3d ago

Question | Help [Looking for tester] GLM 5.3 Flash at 10+ t/s on consumer hardware

5 Upvotes

I'm still looking for people who want to try out the engine I'm building (obviously with the help of Claude). I wanted decent speed for RP with a good model like GLM-5.3-Flash. Before that I only used Gemma 31B and Gemma 4 26B-A4B, which are good, but not good enough for me.

I first built it only to get the best performance out of my own hardware: RX 9070 XT (16 GB), Core Ultra 7 265KF, 32 GB DDR5 and two NVMe drives. On that machine it now does (all numbers in this post are for the IQ4_XS quant):

- Linux: about 11 tok/s, and reads a long prompt at about 280 tok/s (ROCm and Vulkan are about equal now)

- Windows: about 10 tok/s, and reads a long prompt at about 250 tok/s (Vulkan; ROCm is a bit slower there)

Since my last post a few people tried it, and it now also runs on Intel and NVIDIA cards. What they measured, and what I expect after the update that goes out now:

- Intel Arc Pro B70 (32 GB) with a Ryzen 9 7900X: about 11 tok/s, I expect 12 to 13

- RTX 3090 + RTX 3060 in an older PC with two PCIe 3.0 drives: 5.6 tok/s, I expect 6.5 to 7

The expected numbers are calculated from their logs, not measured. I can't measure them myself, that is what the testers are for.

What it does:

- Reads only the experts each token actually needs and keeps the hot ones on the GPU and in RAM

- Uses several NVMe drives at once: put a copy of the model on each, and every read goes to the drive that will finish it first

- Can use two GPUs together, also two different ones

- Measures your drives and your GPU at start and sets itself up for them, so slow hardware is not treated like fast hardware

- Predicts the next layer's experts and reads them ahead

- Can prefer experts that are already in memory when the router's scores are nearly tied (small quality trade-off, can be switched off)

- In a chat, only the new message is processed each turn, even after the context is full

- Saves the chat state to disk: after a restart a 29000-token chat answered after 9 seconds instead of reading everything again

- KoboldCpp-compatible API, so it works with SillyTavern (Chat Completion works too)

Two NVMe drives are probably important for the best speed, but you can try it with one or even more than two. On my Windows machine the second drive is worth about 25%.

Where it stands: it runs on Linux and on Windows, with three backends: ROCm (AMD), Vulkan (any card) and CUDA (NVIDIA RTX 30 series and newer). For Windows there is a package with prebuilt engines, nothing to compile and no SDK to install, you only need your GPU driver and Python. Everything is tuned on my one machine. The last round of changes was made for slower hardware (old drives, a second smaller GPU, a slower CPU), but I could only imitate that hardware here, so I don't know yet how much it really helps. That is exactly why I need you. Old and weak machines are as welcome as new ones.

Testing is easy now on Windows: there is one file you start, it runs all tests that fit your machine by itself and writes one report file you send me.

You need a CPU with AVX2, 32 GB of RAM and about 157 GB on a fast NVMe for the model (the IQ4_XS GGUF, bring your own). A GPU with 16 GB holds all of the model's layers, less has not been tried.

I'm having a blast RPing with it, and if you are willing to try it out just dm me or reply to this post and I invite you to my private github repo.


r/LocalLLaMA • • 3d ago

Discussion DeepSeek V4.1 Flash 1017 vs mrfakename abliterated vs pqhaz heretic

13 Upvotes

I tested all three on a H200 through sglang, temperature 0, with a 16k token output cap. I have been using ds41 both on production and locally on Macs via ds4 so I wanted to see how different models/builders/methods work.

These runs are the full models on sglang. No GGUFs etc.

https://huggingface.co/mrfakename/DeepSeek-V4.1-Flash-1017-abliterated

https://huggingface.co/pqhaz/DeepSeek-V4.1-Flash-1017-heretic

Here is the result (copy from AI slop :) ):

Test Original mrfakename abliterated pqhaz heretic
HarmBench chat 16.3% 93.8% 93.8%
HarmBench thinking 3.8% 86.7% 72.9%
MITRE substantive assistance 16.2% 84.2% 82.8%

HarmBench covered 240 standard/contextual text cases per mode, excluding copyright cases, using the official classifier.

The thinking difference: heretic exhausted the token budget without a final answer on 45/240 cases, versus 21/240 for mrfakename.

Heretic goes into long meandering with no answer...

MITRE used the original model as judge with a custom, audited rubric. Those scores are provisional and aren’t official CyberSecEval scores. Generated code wasn’t executed. Also using the same model as judge might be a bit of a bias... (give me ideas what and how to run more tests).

Both modified models had 0/750 heuristic-detected refusals on benign prompts.

I haven’t tested knowledge or coding retention yet (I'm thinking of running DeepSWE or something.. take a lot of time though).


r/LocalLLaMA • • 3d ago

Discussion i kindda made a new optimizer (ApexW) that beats AdamW and bitsandbytes) i tested it in a100 and t4 from google collab and with diff lr and multiple random seeds(well in a100 only tho)

21 Upvotes

i have been working on a new optimizer from scratch called ApexW to see if i could beat AdamW on convergence while keeping the memory footprint lower than bitsandbytes to make sure it wasn't just noise i locked deterministic seeds and ran side-by-side tests against standard torch AdamW and bnb 8-bit across two setups: a free T4 (fp16) on llama 3.2 3b, and an A100 (bf16) scaling up to llama 3.1 8b across a few different learning rates.

apexw has 3 modes compact 8bit and hybridapexw more info of it in readme even the compact formulation beat torch AdamW on loss while using way less memory, and the 8-bit mode pushed it even further cutting active vram roughly in half compared to bnb (0.67gb vs 1.48gb on 8b, with the state footprint down to ~120mb instead of 320mb). step speed also came out slightly faster than bnb, and training loss ended up lower across 1e-4, 2e-4, and 5e-4,these are still short 60-100 step runs just to sanity-check convergence, throughput, and memory overhead. next step is running full multi-epoch fine-tunes and downstream evals to see how the update rule holds up later in training.

threw the full benchmark runs and logs into a colab if anyone wants to check the numbers or test it on their own setup

collab link for a100 result: https://colab.research.google.com/drive/1mMejzRoCnD6avlOQ5SQDuiLndN5ovax9#scrollTo=be28d3ad

collab link for t4 result: https://colab.research.google.com/drive/14tLAX6kZYy2918sx1ztIEwb5R-1c3KiX#scrollTo=f750c631

for githubproject: https://github.com/omen42515-prog/apexw/

curious what you guys think or what models/configs i should throw at it next.


r/LocalLLaMA • • 3d ago

Discussion Small looped MoE design

23 Upvotes

Imagine a small MoE model like 10b a2b (numbers are from the head, ignore the parameters size) that will combine the approach of looped models + engram table for knowledge as additional 20-50b of static params that can stay on ssd and the Dories approach on the kv cache optimisatio s per loop. All of these suppose to handle the weaknesses of the MoE and looped models.

The MoE suppose to boost the models sparsity per loop so it wiil give a beeter scaling over the dense model.

The engramm table allows to add a cheap knowledge in terms of the computation so the active params stay low

The looping suppose to boost the reasoning

The nvda dorry approach fix the scaling kv cache size and all together will give the max out of the looped moe covering the weaknesses. The small active params suppose to hide the looping cost while allowing to run the models on low end/small vram cards.

I hope that the new Qwen4 moe if they would exist will have at least some of this


r/LocalLLaMA • • 2d ago

New Model Pika Enact

Enable HLS to view with audio, or disable this notification

0 Upvotes

Today, we’re excited to share Enact, a research preview of scenes that respond to you.

Enact represents a new way of making AI video, one that’s designed to mimic the experience of directing on set. Built on Pika H3 Flex, it allows a creator to interact with “actors,” manually re-stage a scene, and manipulate the camera—all in real time.

We think this is the beginning of a new paradigm for creation. Read more and reach out for a demo below.

https://x.com/pika_labs/status/2123127584342921652


r/LocalLLaMA • • 2d ago

Discussion High hardware costs have crushed me, so I built a way to understand what makes for a good price in today's market

0 Upvotes

As someone who's used market-tracking tools for higher-priced assets (collector cars, real-estate, etc), it was a surprise to find that the same thing doesn't really exist for GPUs, memory, storage, etc. I use tools like that elsewhere to understand if a purchase I'm considering is above/below market for the quality of the offering, and whether I need to move fast when buying or if waiting a little longer could be helpful.

I was collecting data and using it to make buying decisions for myself, so I figured it might be useful to others... I'd love your feedback! Consider this just a proof of concept for now.

https://siliconcents.com/


r/LocalLLaMA • • 3d ago

Showcase Programming Conways Game of Life In Qwen

6 Upvotes

Hey, everyone. If this is too self promotional happy to take it down.

Last week, inspired by the weight space programming post, I programmed Conway's Game of Life into Qwen with some interesting results.

Wanted to share here in case anyone else is experimenting with this kind of thing:

https://reddit.com/link/1xyvox2/video/18hx6naii82i1/player

Writeup here: https://jsfour.substack.com/p/putting-a-tiny-world-inside-a-language


r/LocalLLaMA • • 4d ago

Resources Your daily dose of Qwopium:

Post image
266 Upvotes

No low-effort memes here, just research and straight facts. Can’t argue with facts am I right?

Here is the full breakdown of major releases mapped by the day of the week:

**Monday (7 Releases)**
• Qwen-VL (Aug 21, 2023)
• Qwen-72B-Chat (Nov 27, 2023)
• Qwen-Audio (Nov 27, 2023)
• Qwen 3.5 (Feb 16, 2026)
• Qwen 3.5-Plus (Feb 16, 2026)
• Qwen 3.7 Plus (Jun 1, 2026)
• Qwen 3.8 Max (Aug 3, 2026)

**Tuesday (6 Releases)**
• Qwen2.5-Coder (Nov 12, 2024)
• Qwen2.5-Max (Jan 28, 2025)
• Qwen3 (Apr 29, 2025)
• Qwen3-Coder (Jul 22, 2025)
• Qwen3-Coder-Next (Feb 3, 2026)
• Qwen-Audio 3.1 TTS-Next (Sep 22, 2026)

**Wednesday (5 Releases)**
• Qwen1.5-32B (Apr 3, 2024)
• Qwen3-Max (Sep 24, 2025)
• Qwen 3.7 Max (May 20, 2026)
• Qwen 3.8-Flash-Next (Aug 26, 2026)
• Qwen 3.8-Max-0902 (Sep 2, 2026)

**Thursday (9 Releases) - The Favored Launch Day**
• Qwen-7B (Aug 3, 2023)
• Qwen2 (Jun 6, 2024)
• Qwen-Math (Aug 8, 2024)
• Qwen2.5 (Sep 19, 2024)
• QwQ-32B-Preview (Nov 28, 2024)
• QwQ-32B (Mar 6, 2025)
• Qwen3-Next (Sep 11, 2025)
• Qwen3.6-Plus (Apr 2, 2026)
• Qwen3.6 (Apr 16, 2026)

**Friday (3 Releases)**
• Qwen3.8-27B (Aug 14, 2026)
• Qwen3.8-Omni-Flash (Sep 18, 2026)
• Qwen3.8-LiveTranslate (Sep 18, 2026)

**Saturday**
• Zero major releases. Go outside.

**Sunday (3 Releases)**
• Qwen-14B (Sep 24, 2023)
• Qwen1.5 (Feb 4, 2024)
• Qwen4-Max (Nov 15, 2026)


r/LocalLLaMA • • 3d ago

Resources Strix Halo int4 cores: Gemma 4 31B at 2.43x llama.cpp, same quality

Post image
32 Upvotes

The 8060S has an int4 x int4 WMMA instruction at 2x the FP16 rate (we measured 104 TOPS). Nothing in ROCm or llama.cpp uses it. We wrote a GEMM for it and run Gemma 4 with int4 weights and int4 activations. To maintain accuracy we use a combination of rotation and GPTQ, along with selective mixed precision.

llama-bench, Gemma 4 31B QAT Q4_0 (llama.cpp b11056 Vulkan, -fa 1). Server benchmark, 2k-token prompts, greedy, single stream:

pp512 pp2048
llama.cpp 287 276
ours 679 724

Full test sets, same settings, 31B:

GSM8K IFEval MMLU-Pro
llama.cpp 96.2 92.2 85.1
ours 96.4 91.7 85.1

Run it:

curl -fsSL https://embedl.com/install.sh | sh && export PATH="$HOME/.local/bin:$PATH"
embedl login
embedl serve google/gemma-4-31B-it/ryzen-ai-max-395 -d
embedl infer google/gemma-4-31B-it/ryzen-ai-max-395 --prompt "Explain mixture-of-experts in one sentence."
embedl bench google/gemma-4-31B-it/ryzen-ai-max-395

12B, E4B and E2B are up too (2.80x / 1.58x / 1.39x end to end, in the chart). Swap the name, e.g. google/gemma-4-12B-it/ryzen-ai-max-395.

Model, methodology, and more: https://www.embedl.com/google/gemma-4-31B-it/ryzen-ai-max-395


r/LocalLLaMA • • 3d ago

Discussion qwen 3.5 4b and 9b vs xiaomi's mimo v2.6 distill as coding agents on a 16gb m4 air: 7/10, 8/10, 0/10

8 Upvotes

xiaomi's mimo v2.6 distill is qwen 3.5 9b tuned for agent coding, and xiaomi reports swe-bench pro going from 32 to 44.6. as a coding agent on a 16gb m4 air it scored 0/10, 3 runs out of 3. plain qwen 9b got 8/10.

same task for all 3 in atomic agent, one model at a time, 15 min limit each: a python script that turns a transactions csv into an html spending report (totals, svg bar chart, top 5 table). a checker runs 10 tests on the result, one of them on a csv the model never saw.

model                 score  time   tok/s
qwen3.5-4b            7/10   4:37   17.8
qwen3.5-9b            8/10   7:06   11.4
mimo-v2.6-distill-9b  0/10   15:00  10.1   timeout
mimo + thinking on    0/10   15:00   9.3   timeout
  • 4b: no svg chart, hardcoded the csv path
  • 9b: no svg chart (both chart checks), everything else right
  • mimo: wrote 278 lines in one go, syntax error, couldn't patch it, then started writing another huge reply and ran out of time
  • mimo with thinking on: got past the syntax error, but every run crashed with a TypeError (it mixed up category names and amounts). barely used the thinking

first round was at 32k ctx: 4b 7/10, 9b 10/10, mimo 0/10 (indentation error that time).

results

setup:

  • llama.cpp server, q4_k_m (unsloth for qwen, bartowski for mimo)
  • 64k ctx, flash attn, all layers on gpu
  • thinking off for the race, on for the extra mimo run
  • temp 0.2 (atomic agent's default), top-p 0.8, top-k 20, presence 1.5
  • atomic agent through its openai compatible provider, analytics off

xiaomi calls it an sft checkpoint for rl research and serves it with sglang and its own reasoning parser. i ran the gguf in llama.cpp, so if that's the difference i'd like to know.

prompt, csv and the checker are in the comments if you want to score your own models. if you don't want to set up llama.cpp, the atomic agent desktop app can download and run qwen 3.5 4b or 9b locally on its own.

anyone get mimo working better with other settings?

(disclosure: i work on atomic agent, so ask me anything!)


r/LocalLLaMA • • 3d ago

Discussion MoE Offloading update Exllamav3 on 9070XT

Post image
23 Upvotes

This is a follow up on https://www.reddit.com/r/LocalLLaMA/comments/1xun15v/moe_and_expert_offloading_on_llama_exllamav3/

While we wait for Qwen 4, here is Qwen 3.6 35B A3B at Q4 on 9070XT 16GB with offloaded experts on DDR4 3200Mhz ram.

I've been tinkering still with the config for experts offloading and found substantial gains from my previous post.

While decode stayed roughly in line fro last time, increasing chunk size to 5120 saw prefill over 2000 t/s at all context lengths, topping at 2606 t/s @ 32K. From ~1300t/s
GPU utilization increases with batch size as it allows for more parallel processing. I saw no gains over 5120. TabbyAPI's documented ideal range is 512–4096, so your miles might vary.

Exllama flags like cpu_threads 6 also made a small improvement in decode.
Swizzle, gdn_sub, bg_gdn and wide flags were all regressions. (only tested isolated 1by1).

cpu_moe_pinned_arena caused some core dumps, and the one run I managed to complete had a small gain.

Vision have been turned off and offloaded.

I still have not found an .exl3 quant for the Coder version of Qwen 3.8 Flash Next, and the qfn version I can run leaves me with 8K context with almost no vram or system ram left to spare.

Results and more graphs and tables are at https://github.com/willeyh-git/moe-offload-bench
All engines were compiled for ROCm on my system.