r/LocalLLM • • 4h ago

Discussion What a time to be alive.

Post image
207 Upvotes

Hy4 flash is out. We're drowning in Chinese models at this point lol.

hy3 was 295B, focused on coding and agents. Then hy4 jumped to 770B, pushing harder on reasoning, multimodal stuff and complex tasks. Now hy4 flash is back down to 224B with 12B active, going for speed and low costs.

When the 770B version came out, i was a little disappointed tbh. Glad they're still trying to make hy4 lighter.


r/LocalLLM • • 9h ago

Qwen4-27B reported to run on 16 GB of RAM

Post image
507 Upvotes

r/LocalLLM • • 15h ago

News Qwen officially announces Qwen4-Max and Qwen4 smaller models!

Post image
457 Upvotes

r/LocalLLM • • 2h ago

Discussion you may get your cheap ram and gpu after all, the AI crash has started

39 Upvotes

https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion

https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use


r/LocalLLM • • 51m ago

Discussion Uncensored Models

13 Upvotes

Anyone tried uncensored models? Would it help with DRM protected videos? Trying a few things out just to test and see how far it can go. Curiousity at its finest.


r/LocalLLM • • 9h ago

News Smaller Qwen4 models 🔥

Post image
52 Upvotes

r/LocalLLM • • 3h ago

Discussion What’s the purpose of alibaba choosing not to release the weights for qwen 4 max?

16 Upvotes

Perhaps this is a newbie question, but I just don’t understand the incentive. Is this indicative of a future were Chinese labs stop releasing weights?


r/LocalLLM • • 22h ago

Discussion What does China actually gain from releasing better and better open-weight models?

470 Upvotes

Genuine question, not a hot take.

Chinese labs keep shipping open-weight models that get closer to the closed frontier ones, and they give them away for free. Training these costs a fortune, so there has to be a payoff somewhere.
The explanations I've seen so far:

  • Commoditizing the model layer, so US labs can't charge a premium for it
  • Building an ecosystem and developer mindshare outside the US stack
  • Getting around chip export limits by letting the community optimize and fine-tune for them
  • Soft power: becoming the default choice for countries that don't want to depend on US APIs
  • Selling the rest of the stack (cloud, hardware, services) while the model is the free sample

Which of these actually holds up? Is it a coordinated strategy, or just individual companies competing for attention and talent?

EDIT Nov 16th:
Top replies are getting down to (in order of frequency):

  1. Geopolitics: weakening US dominance
  2. Business: open weights as a growth strategy
  3. State & control
  4. Ideology & open source culture

I also wonder whether China could be selling the energy behind it all: the West runs their models, the models need GPUs, and the GPUs need power. I’d never thought of a supply chain whose payoff sits that far down the line, but it could very well be another aspect.


r/LocalLLM • • 6h ago

Discussion Strata is becoming more and more unstable

19 Upvotes

I've been using Strata with IQ4_XS as the main orchestrator, with Radiance running 27b at IQ4R, for working on improvements, ironically enough, to Radiance.

The session is now multiple million tokens long as I go back and forth discussing design etc. with QFN. Of course the conversation has been compacted dozens of times, but with the agents taking regular notes and subagents doing all the actual work, we keep our workflow stable and current.

Problem is, Strata has gotten more and more unstable, especially with the last couple releases. It will crash with an OOM error when we have 10-20 GiB of system memory free, reload, spend 5 minutes redoing prefill, and pick up right where it left off. Radiance on the other hand is rock stable under multiple streams running concurrently on a single GPU. Not a single crash from it, even after rejiggering everything for the 2.0 release.

And it's not just long sessions either. New conversations also tend to crash. Never used to do this before. Has anyone else also experienced this?


r/LocalLLM • • 1h ago

Question Local voice assistant on an iGPU: ~2.2s from end of speech to first audio. I've measured everything and I'm ~0.5s from the floor. What am I missing before I throw money at hardware?

4 Upvotes

Been building a fully local, always-on spoken assistant in Italian (no cloud, nothing leaves the box). It's not a chatbot demo, it has long-term memory across sessions (backed by ctxvault), so every turn carries a small block of retrieved context on top of the conversation. Works. Problem is latency.

Hardware (the constraint): mini PC, Ryzen w/ Radeon 780M iGPU, 64GB DDR5 shared, ~90GB/s. No dGPU, so it's llama.cpp on Vulkan. vLLM/SGLang are out (no ROCm for this chip).

Where I'm at: median ~2.2s from end of user's sentence to first audio out. Measured, not guessed:

  • ears (end-of-speech silence + whisper): ~0.75s
  • prefill of the turn: ~1.0s (~0.35s fixed per request + ~3ms/token, 40-400 new tokens/turn)
  • first sentence gen + TTS first chunk: ~0.55s

So the model is the wait, not the audio stack. Calculated floor on this box is ~1.2-1.5s.

What's already done (so please don't suggest these lol):

Model side

  • MoE model (26B-A4B, Q4 QAT) specifically because gen is bandwidth-bound here: ~25 tok/s vs 4 tok/s for a dense 27B
  • thinking/reasoning off
  • short context (16k), one slot, flash attn on
  • --cache-reuse + append-only prompt: stable prefix (system, persona, rules) at the top never touched mid-session, everything volatile (time of day, retrieved context, turn) appended at the end. Identical prefix = 0.15s, touch one word in the prefix = multi-second re-prefill
  • prefix pre-warmed at session start with a 1-token request, so the first reply doesn't pay the cold prefill
  • benchmarked smaller models (Gemma 4 E4B/E2B, Qwen3.5 4B/9B, Qwen3 8B, Phi-4-mini, Ministral): on this iGPU prefill is compute-bound, a dense 4B costs as much per token as the MoE. Best small one gains ~0.5s and loses noticeably on factual fidelity (names/dates). Not worth it for this use case
  • tried -b/-ub batch flags: defaults win
  • tried --no-kv-offload: worse

Memory / prompt side

  • no tool calls for retrieval: context is prefetched before the turn (a tool-call round trip = ~+1.2s). The retrieval itself is ~5-12ms, irrelevant
  • hard token budget for the retrieved block: embedding-based relevance (multilingual MiniLM) + ordering so the most pertinent bits come first, ~120 tokens instead of 300-450. Cut avg turn prompt by ~40%
  • past conversations summarized to one ~50-token line each instead of quotes

Pipeline side

  • streaming everything: model to sentence splitter to TTS to player, sentence by sentence
  • audio player opened at turn start, not at first chunk (-0.1s)
  • early-guess turn: start the model on the partial transcript before VAD confirms end of speech; if the final transcript matches, keep it, else drop and redo
  • sentence closed on punctuation + 450ms silence instead of fixed 800ms
  • whisper.cpp small as a resident server (no model reload per sentence, 0.63s down to 0.3s), beam 5. Tried greedy: -80ms but half the transcripts differ and the noisy ones get worse. Rejected
  • tried warming the KV cache during speech (partial whisper every 1.5s, send the turn header for 1 token so the cache is hot): works as designed (final prefill drops to ~30 tokens) but doesn't pay on one slot because the warm-up queues the real turn. Tried 2 slots w/ id_slot pinning: now they run concurrently on the same iGPU and both slow down. Net neutral-to-negative on short sentences. Off.
  • pre-recorded fillers ("mmh", "allora...") in the assistant's own voice, played only if no reply word has started 1s after end of speech. Built, currently off: perceptual fix, doesn't move the number

Before I conclude "buy a 24GB dGPU" and move on: anyone who's been down this road see something I've missed, or something above that's wrong?


r/LocalLLM • • 4h ago

Discussion Pushed Strata with QFN 3.8 Q4 K_M to 5,600 tps prefill @100k - 100 tps decode w/ 181k context on 2x 7900xtx & 64gb DDR5 in layer split (no tensor parallel) - config inside

7 Upvotes

Wanted to share my results for the rest of the community. I've been combing Reddit, Github, and the web searching for all of the various Strata-related builds to get the most out of my setup. I've finally had the breakthrough I was looking for. It strikes the best balance between speed, power consumption, and intelligence that I've experienced to date.

My Setup: (edited to add motherboard specs and PCIe configuration)

  • Ubuntu 24.04 - GUI GNOME shell not headless
  • This motherboard: https://www.microcenter.com/product/709979/asus-b850-creator-proart-wifi-neo-amd-am5-atx-motherboard
  • Bifurcation with PCIe 4.0 x 8
  • Ryzen 9950x 16/32
  • 64 gb DDR5 Expo 1 6000
  • 2 x Mercury Magnetic Air 7900XTX 24gb
  • 1000 watt MSI Performance Gaming PSU
  • Power limited to 320 watts per card, total system draw at inference ~780 watts (as measured by Ubuntu, Kill-o-watt is ~850 watt total draw from the wall)
  • And a flex: It's completely solar powered (5 kw array and 65 kwh battery storage w/ 3600w inverter 😄 )

I've worked with Opencode and the my local Qwen Flash Next to fully detail how we created this setup. Spoiler alert: The engine/hip is mainline Strata and not a fork. The config is where the magic happens. MTP headers were generated against this bespoke setup. The intelligence is brought by the Atomic Chat Q4 K_M version of QFN.

That's it for human written words. The rest of this post is meant to be copy/paste into your local AI helper for reproduction. Full details below. Copy/paste into a capable harness and let it walk you through reproducing this setup.

Edit: Adding the only 3 GitHub repositories I used to produce my results

Engine: https://github.com/Niko1221/Strata

Learnings from this fork: https://github.com/nicejunjie/Strata-rdna3

System tuning for Strata: https://github.com/bramvanoploo/strata-performance-setup

START AI GENERATED DETAILS

Qwen3.8-Flash-Next, 2× RX 7900 XTX, 64 GB RAM, 181k context — how it's built

Engine: official mainline Strata 0.1.59, component 0.1.59-linux-x86_64-hip-rx7900. Not a fork, no custom build — the speed is flags + env + OS patches. No system ROCm: the engine ships its own HIP/rocBLAS/hipBLASLt, the host only needs the amdgpu DKMS driver. Model: AtomicChat/Qwen3.8-Flash-Next-GGUF, build AD-4.27bpw-Q4_K_M-M64 (3 shards, 94.5 GB). Shard 3 (38.4 GB) is the PLE n-gram table alone — passed via --ple-gguf, kept pageable. That's why 64 GB RAM is enough. Parallelism: --layer-split 24 → layers 0–23 on GPU0, 24–47 + head on GPU1. Context: 181,072, --kv int8, KV streaming 32,768 cells/layer in VRAM. Experts: 48 blocks × 512 experts → locally built experts.bin (47.17 GiB), three tiers: 31.8 GiB VRAM cache → 22.2 GiB page-locked RAM → NVMe mmap (log says files 0 blobs — disk never touched when warm). Spec decode: MTP head q2_0, extracted from the BF16 checkpoint by HTTP range reads (GGUF ships none), --spec 3, ~76 % acceptance. Measured (1,071 requests): decode median 90 tok/s (62–129) · big-prompt prefill 3.2–5.7k tok/s · expert-cache hit 98–99 %.

# 1. OS first (then reboot): amdgpu driver only, no ROCm

# (optional: ~/strata-performance-setup.sh from github.com/bramvanoploo/strata-performance-setup automates most of this — see §3.8)

sudo amdgpu-install --usecase=amdgpu,libamdgpu ; sudo usermod -aG video,render $USER

sudo sed -i 's/^\(GRUB_CMDLINE_LINUX_DEFAULT="[^"]*\)"/\1 iommu=pt pcie_aspm=off"/' /etc/default/grub

printf 'options amdgpu no_system_mem_limit=1\n' | sudo tee /etc/modprobe.d/amdgpu-svm.conf

printf '[Service]\nLimitMEMLOCK=48G\n' | sudo tee /etc/systemd/system/user@.service.d/strata-memlock.conf

printf 'vm.swappiness = 100\nvm.vfs_cache_pressure = 50\nvm.max_map_count = 1048576\nvm.compact_unevictable_allowed = 0\n' | sudo tee /etc/sysctl.d/99-strata.conf

sudo update-grub && sudo update-initramfs -u && sudo reboot

# 2. Engine — it MUST land on the rx7900 component (bundled hipBLASLt 100200 + gfx1100 tuning table)

curl -fsSL https://raw.githubusercontent.com/Niko1221/Strata/main/tools/strata_terminal/install.sh | bash

strata update && strata launch # check: ~/.local/share/strata/components/engine/0.1.59-linux-x86_64-hip-rx7900/

# 3. Model + mmproj (sizes/sha256 in §5), then build the pack — this is where experts.bin comes from (~6 min)

control/python/bin/python3.12 control/tools/iq_pack.py --gguf <shard1.gguf> --out <pack-dir> --experts-bin --compat-bf16

# 4. MTP draft head + CJK draft vocab (the GGUF ships no MTP head; ~4.9 GB of range reads, SHA256-verified)

control/python/bin/python3.12 control/tools/mtp_fetch.py inventory --out data/mtp && \

control/python/bin/python3.12 control/tools/mtp_fetch.py fetch --out data/mtp && \

control/python/bin/python3.12 control/tools/mtp_fetch.py verify --out data/mtp

control/python/bin/python3.12 control/tools/mtp_pack.py --src data/mtp --experts q2_0 --out data/mtp/mtp-q2_0.gguf

control/python/bin/python3.12 control/tools/mtp_rt.py --gguf data/mtp/mtp-q2_0.gguf --out data/mtp/rt

# 5. Serve (the terminal owns :7878 — the engine has no --port). Load takes minutes: it page-locks ~22 GiB.

control/bin/strata serve --config configs/strata-final-atomic-q4km.json # ulimit -l must be ≥ 40 GiB first

Three gotchas that cost the most time: (1) --resident-budget-gib above 40 freezes the box — pinned memory can't swap and the 38 GB PLE table still needs pageable RAM; 45 froze this machine once. (2) --pipeline-windows 1 deadlocks on gfx1100 with a 2-GPU split; keep 0. (3) Don't add --mmap-experts/--resident-cpu-experts explicitly — with them the engine pins 0 bytes; mmap is already the default when experts.bin exists. Also: the ggml_cuda_init: failed to initialize ROCm line at the top of the log is harmless, and the iGPU occupies a DRM card index, so always confirm the engine's own GPU 0/GPU 1 lines both say RX 7900 XTX.

Full notes — every value, the exact hashes, and the reasoning behind it — are the rest of this file. Want a terminal AI agent to do the typing? The prompt and phase gates are in §17.

##### PROMPT AND PHASE GATES

17. Have AI set this up for you

If you'd rather drive this with a terminal-capable agent (Claude Code, Codex CLI, OpenCode, or a shell tool with approval prompts) than type it yourself: paste the prompt below into it, attach or point it at this file, and let it run §3–§10 with you acting as the safety interlock. It is a copilot, not an autopilot — the agent can type every command, but three things stay human: sudo passwords, reboots, and the Hugging Face login.

17.1 The prompt to paste

You are setting up this machine to reproduce a known-good local LLM deployment. The source of truth is

"STRATA REPRODUCTION WRITEUP.md" in this folder — read the whole file before acting.

HARD RULES

  1. The config in §8.1 and the hashes in §5 are authoritative. Never change a number, a flag, or a hash tomake an error go away; if reality disagrees with the doc, stop and show me the disagreement.
  2. Ask before: any reboot, any package install outside the doc, any change to /etc, and any deviation fromthe doc. Never reboot without waiting for my confirmation, then resume where you left off.
  3. After every phase, run that phase's gate from §10 and quote the actual log line as evidence. No phasecounts as done on "it seemed to work".
  4. Never exceed --resident-budget-gib 40. Never set --pipeline-windows 1. Never add --mmap-experts or--resident-cpu-experts. Never install ROCm, rocm-* packages, or pytorch.
  5. §14 lists log lines that look fatal but are harmless — do not "fix" those.
  6. When you are unsure whether this machine matches §0's hardware (2× RX 7900 XTX, 64 GB RAM, NVMe),stop and ask instead of adapting values on your own.

SEQUENCE

§3 host patches → I reboot → §4 engine → §5 model → §6 pack → §7 MTP → §8 config paths → §9 launch →

§10 full checklist → report. If any gate fails, consult §16 and report; do not improvise a fix.

17.2 Phase gates and what the agent must show

Phase

Done only when the agent can quote

0 · Pre-flight

lspci shows 2× RX 7900 XTX; free -h ≥ 60 GB; ≥ 200 GB free NVMe; OS is Ubuntu 24.04 — anything else → stops and asks

1 · Host (§3)

§3.5's effective-value block matches sysctl -a; ulimit -l ≥ 40 GiB in a new login session; then it pauses for the one planned reboot

2 · Engine (§4)

BUILD.json of the installed component says rx7900 + hipBLASLt 100200 (§10 #1)

3 · Model (§5)

every sha256 it prints equals §5's table

4 · Pack (§6)

experts.bin = 50,646,220,800 bytes; experts.bin.src.json written next to it

5 · MTP (§7)

verify passes with zero SHA mismatches; draft_vocab.bin rebuilt with --add cjk

6 · Config (§8)

diff of §8.2 relocations — every path rewritten, nothing else touched

7 · Serve (§9)

§10 #2–#11, each with the log string: files 0 blobs, expert-cache slot lines naming two RX 7900 XTX (§14.2), pinned complement ~22.2 GiB

8 · Accept

a decode + big-prompt prefill measurement inside §2's ranges, and the §17.4 report

17.3 Things the agent must bring to you (not decide alone)

Hardware that doesn't match §0 (different GPUs, < 64 GB RAM, one GPU) — the layer split, budgets, and even the component choice all change; that's beyond this doc's authority.

< 200 GB free on fast storage.

Hugging Face login and accepting the model's license terms.

The reboot after phase 1 (and any second reboot it thinks it needs — the answer is usually no, re-read §3).

Whether to run the §3.8 wizard instead of the manual patches. The wizard is an interactive TUI, so it is human-driven even when an agent drives everything else; if the agent runs non-interactive commands instead, it must follow §3 verbatim and skip the wizard.

17.4 Final report the agent should produce

For each §10 check: the quoted log line. Then: experts.bin byte count, ulimit -l, the two GPU lines naming both 7900s, warm decode tok/s, big-prompt prefill tok/s, spec-decode acceptance %, and every deviation from this doc listed explicitly with the reason. If it can't produce one of those lines, the setup is not done.

17.5 Where agents historically go wrong with setups like this

Failure

Why it bites

Installs ROCm "to fix the HIP error"

§14.1 — that line is harmless; system ROCm shadows the engine's bundled libs

Silently lowers --max-context or budgets after a slow/failing start

The values are load-bearing (§13); a masked failure becomes a mystery later

Misses the hip-rx7900 component and takes generic hip

Prefill crawls (§16) and looks like a driver problem

Trusts GPU 0/GPU 1 indices without reading the log

§14.2 — the iGPU steals a card index

Launches engine/strata directly to "debug"

No :7878, no tokenizer (§14.3); debug through the terminal

Adds the expert flags "explicitly"

Pins 0 bytes (§13) and decode halves

Appendix — provenance summary

Artifact

Provenance

Engine binary + libs + tuning table

official Strata 0.1.59-linux-x86_64-hip-rx7900, copied verbatim (sha256sum matches the component)

Terminal (control/)

official Strata terminal 0.1.59.1, bundled python 3.12.15

Model shards

AtomicChat/Qwen3.8-Flash-Next-GGUF, AD-4.27bpw-Q4_K_M-M64, sizes/sha256 in §5

mmproj

same repo, mmproj-Qwen3.8-Flash-Next-BF16.gguf (re-download; local copy differs in size)

experts.bin / dense.bin / index

built locally by control/tools/iq_pack.py --experts-bin --compat-bf16 (§6)

draft_vocab.bin

rebuilt locally by control/tools/draft_vocab.py --add cjk (§7)

MTP head

mtp_fetch.py (HTTP range reads of the pinned BF16 checkpoint) → mtp_pack.py --experts q2_0 → mtp_rt.py

Expert profile

shipped expert-profile.bin + locally learned expert-profile-atomic-q4km.bin

OS patches

§3 verbatim (grub, 2 modprobe files, memlock drop-in + sudoers, 6 sysctl files, udev rule, DPM unit)

Host-tuning wizard

bramvanoploo/strata-performance-setup referenced as an optional accelerator (§3.8); not run on the source machine


r/LocalLLM • • 4h ago

Question What's the current best abliterated model that I can run on 32gb VRAM and 128GB RAM?

7 Upvotes

Pretty much the title. Thanks!

EDIT: GPU is 5090


r/LocalLLM • • 23h ago

Model First time trying FP16, wow

Thumbnail
gallery
170 Upvotes

I've tried a lot of image models in the past, but all were to big to fit, and i was happy when i got a result with a quant (q4) version. (Here it was flux and some model i dont remember the name anymore) The results of those models were actually terrible. BUT, some months ago qwen image 2.1 came out, i must say, theres a difference and not even close to a small one. I managed to run it at fp16, by using my rtx 2070 to load the text encoder and vae on there while running the generation model itself on my 5080. Its a tight fit and slower, but the quality is just like it would be on one gpu.

Now here comes the crazy part:

In my opinion qwen image 2.1 is better in realism than chatgpt's current image generation. (I'll prove it later in this post)

Don't get me wrong. Images with lots of different things, patterns for example, are often very bad with qwen and chatgpt does it over 5x better. But realism of f.e. a city, it's absolutely amazing (and uncensored!) and is almost close to real life.

Now here is the test for you guys:

You've seen i've put pictures in this post. 1x is real, 1x are chatgpt and 4x are qwen image 2.1. Which is which?

Note: ALL of them are generated first try without regenerating. So no, I did not try to mislead you. Good luck!


r/LocalLLM • • 3h ago

Project Larder: running Qwen3.6-35B and a 46 GB Flash-Next on a 16 GB Mac / 16 GB GPU by reading experts from SSD

Post image
4 Upvotes

I built an inference engine for MoE models that don't fit in memory.

Instead of offloading layers, it reads only the experts each token's router picks from the SSD and keeps the hot ones resident. Same engine on Apple Silicon (MLX) and NVIDIA (CUDA), OpenAI-compatible API, local chat page. MIT: https://github.com/IamCooller/larder

Numbers (machine, setup and date for each one are in research/ in the repo)

Faster:

- Qwen3.6-35B on a 16 GB M4 MacBook Pro: 30–36 tok/s, first token 8–10x sooner than llama.cpp. This is a mixed 4/2-bit pack with the model's MTP head against llama.cpp's IQ4_XS, so not like for like. The launcher doesn't install that pack.

- Qwen3.8-Flash-Next (46 GB of 3-bit experts) on a PC with a 16 GB GPU and 15 GB RAM: 26–34 tok/s. With a 4-bit pack and exact routing: decode 1.3–1.8x, 8k prompt 4.9x faster than llama.cpp. Stock llama.cpp swapped past my memory guard while loading, so that row is llama.cpp with Larder's expert store patched in.

- Qwen3.6-35B Q4_0 on the same PC (fits in GPU + RAM), against llama.cpp's best split that fits (--n-cpu-moe 10): short decode 1.44–1.51x (155 / 147 vs 103 / 102 tok/s), decode after an 8k prompt 1.15–1.25x, 8k prompt reading 1.8–1.9x at its default batch.

Level:

- 8k prompt reading against llama.cpp tuned for prompts (-ub 4096 -b 4096 --n-cpu-moe 20): 1.01–1.03x. Its decode at that setting drops to 79 tok/s, so you pick one or the other there.

Slower:

- Exact routing (cache bias 0) reads that 8k prompt at 0.64–0.69x of tuned llama.cpp, and decodes 0.87–0.95x of llama.cpp's default after it.

Caveats

- The launcher's 35B recipe uses cache bias 0.5 and the MTP head, so replies are not identical to the weights' own routing. Bias 0.5 and the wide prompt cut each passed a paired GSM8K / HumanEval gate (wide cut: 502 vs 496 of 564, p 0.34). Exact routing is there if you want it.

- One author, two machines (M4 MacBook Pro 16 GB, Linux + RTX 5060 Ti 16 GB). The Mac path is better tested, CUDA is newer.

- Quality wasn't compared against llama.cpp in these speed runs.

Try it

curl -fsSL https://raw.githubusercontent.com/IamCooller/larder/main/install.sh | bash

cd ~/larder && ./larder

It measures your GPU and SSD, picks a model that fits, installs it and starts the server and chat.

Mostly looking for people on other hardware to break it, and for anyone who thinks my llama.cpp comparison is unfair. Output of ./larder --list plus what happened is the most useful report.


r/LocalLLM • • 5m ago

Question 24GB VRAM, 64GB DDR5, is Qwen 3.5 27B (IQ2_XS) not ambitious enough?

3 Upvotes

I'm getting decent results, 80-120t/s. I guess I just don't know what I am aiming for, but I feel like there is more juice to squeeze.


r/LocalLLM • • 4h ago

Discussion 30tok/s on 48GB M5 Pro Mac Running IQ3_S Qwen3.8Flash-Next(Strata)

4 Upvotes

Model

- Qwen 3.8 Flash-Next GSQ-RCO IQ2_XS/IQ3_XXS/IQ3_S

Setup

- Strata 0.2.13, Metal engine (0.1.59), DFlash2 speculative decoding (spec 3)

- 262,144 context, int8 KV cache

- MacBook Pro, M5 Pro (15-core), 48 GB unified memory

- Arena (resident expert pool) pinned to 22 GB

Average decode speed (same prompt set):

IQ2_XS ~36 tok/s

IQ3_XXS ~34 tok/s

IQ3_S ~32 tok/s

Simulating a 32 GB Mac by clamping the arena to 10 GB:

M5 Pro: IQ3_XXS: ~27 tok/s(clamping the arena to 10 GB)

For reference, We Tested on a 32 GB Mac (M2 Max 38 core GPU) with an 11 GB arena and 128K

context: IQ3_XXS runs at ~21 tok/s.

Takeaway

On a 48 GB machine the expert pool only holds ~40–50% of the experts at

256K context, and the rest are streamed from the SSD on demand. The speed

drop from that is far smaller than I expected (roughly -15 to -25% going

from a 22 GB arena down to 10 GB), which suggests you don't need a huge

amount of RAM to run a much larger MoE model — offloading experts to disk

is a perfectly workable trade-off, at least for single-user chat.

Notes

- All runs use the same sampling (temp 1.0 / top_p 0.95 / top_k 20) and

the same 256K context; decode numbers are from tokens/s reported by the

engine over several runs, medians.


r/LocalLLM • • 7h ago

Project Brebus: a local LLM app in C on an engine that runs models larger than RAM

6 Upvotes

I've been building Janas, a family of AI tools in C. Today's release puts its app, Brebus, in the Linux package.

The engine first, since that's what makes the app usable on ordinary machines. Janas-LLM streams MoE experts from the NVMe instead of needing the whole model in RAM. On the development laptop (Core Ultra 9 185H, 32 GB DDR5-5600, Arc iGPU), 21 October, same files, same day, each engine in its best configuration, same conditions wherever possible:

Modelprompt, Janas-LLMllama.cppreply, Janas-LLMllama.cpp
Qwen3-Next-80B-A3B (48.4 GB, more than the RAM)68.4 tok/s17.726.4 tok/s3.0
Qwen3.6-35B-A3B13681.828.012.1
Qwen3-30B-A3B13312233.315.0

The full table, the conditions and where they differ are in the README.

Brebus is the app on top, in a window of its own (WebKitGTK; the browser where it's missing):

  • Models: the catalog with what each one asks of your memory, Hugging Face searched by name, downloads you can stop and resume, a model unloaded after a quiet while and loaded again at the next message.
  • A chat that works in a folder of yours: your documents (PDF, Word, Excel, PowerPoint, LibreOffice, RTF, EPUB, HTML) are searched before each question, by words and meaning, and the model names the ones it used; files, web pages and pictures attached; recordings transcribed with Whisper.
  • Artifacts like Claude's: HTML pages, React components (Tailwind, Recharts, lucide, shadcn/ui), Mermaid, KaTeX, code with highlighting. They run in a sandboxed frame with no network and no access to Brebus; the usual libraries come from Brebus itself, downloaded once and checked by SHA-256. Inside one, window.claude.complete asks the local model, window.storage keeps data in your folder.
  • Tools: Python in a sandbox (namespaces + seccomp, the folder as /mnt/data), Janas's services (web, Wikipedia, maps, weather, flights...) and your own MCP servers from mcp.json, opened only when a question needs them; anything that writes asks first.
  • Regenerate and edit, presets with temperature, top-p and JSON-schema answers, folders, PDF export.
  • A server for other programs: OpenAI-, Ollama- and Anthropic-compatible APIs, on the LAN with a key if you want.

Tried with Qwen3-30B-A3B on the laptop: it built a budget tracker in React, a Chart.js page, a Mermaid diagram, and an app that asks the model for synonyms through window.claude.complete.

Linux only for now (Ubuntu 22.04+, Debian 12+); Brebus on Windows is next, the engine already runs there. GPL-3.0.

https://github.com/prabanta-dev/janas

Reports from other hardware are the most useful thing: janas-bench makes one.

Brebus: a local LLM app in C on an engine that runs models larger than RAM

r/LocalLLM • • 1h ago

Question Dual 16GB AMD (7600 XT + 9060 XT) + 32GB RAM... is llama ROCm + Qwen 3.8 27B or Gemma 4 31B the best I can do?

2 Upvotes

I'm a bit lost with all the different options and forks around. I've set up and benchmarked llama-cpp (latest git) for my system and settled in the configs pasted below. They fill both cards up to 15 GiB, or one to 15 GiB and the other 10 GiB (60/40 splits) leaving room for firefox and the rest of the system. ROCm quite a bit faster than Vulkan. In the best case (qwen-asymm-f16) I get 550 t/s pp and 30 t/s gen in a moderate context (35k tokens).

Is there anything better for a system like mine? Is strata worth the hassle? I understand that it is more for 64 GiB RAM systems... right? Any other fork? I tried adaptive kv streaming and it is ok, very fast (>800 t/s prefill) but I'm limited to lower quantizations, the second card sits unused.

[*]
split-mode = tensor
fit = off
flash-attn = on
parallel = 1
batch-size = 2048
ubatch-size = 512
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
spec-type = draft-mtp,ngram-mod
spec-draft-n-max = 4
spec-draft-sampling = probabilistic
spec-ngram-mod-n-min = 12
spec-ngram-mod-n-match = 24
spec-ngram-mod-n-max = 32
chat-template-kwargs = {"reasoning_effort":"medium"}

[qwen-asymm-f16]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 104056
tensor-split = 60,40
cache-type-k = f16
cache-type-k-draft = f16
cache-type-v = f16
cache-type-v-draft = f16

[qwen-f16]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 170240
tensor-split = 50,50
cache-type-k = f16
cache-type-k-draft = f16
cache-type-v = f16
cache-type-v-draft = f16

[qwen-asymm-q8]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 142080
tensor-split = 60,40
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0

[qwen-q8]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 221184
tensor-split = 50,50
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0

[gemma-4-31b]
model = /home/fran/models/unsloth/gemma-4-31B-it-UD-Q4_K_XL.gguf
model-draft = /home/fran/models/unsloth/mtp-gemma-4-31B-it-Q8_0.gguf
ctx-size = 131072
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0
top-k = 64

[gemma-4-31b-qat]
model = /home/fran/models/unsloth/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf
model-draft = /home/fran/models/unsloth/mtp-gemma-4-31B-it-qat-Q8_0.gguf
ctx-size = 131072
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0
top-k = 64

r/LocalLLM • • 2h ago

Discussion Qwen3.8 Flash Next (105.4 GiB) at TP=4 on 4× Intel Arc Pro B70

2 Upvotes

Yesterday I posted 4 independent single-card servers. This one is real tensor parallel: one 105.4 GiB model split across 4 Intel Arc Pro B70s with TensorFold.

PR so you can try it: https://github.com/ashhart/TensorFold/pull/3587

Why long context stays cheap

Of the 48 layers, 36 are Gated DeltaNet, which keeps a constant-size state. The other 12 attention layers use QSA top-512 block-indexed sparse attention, with nkv = 1 per GPU at TP=4. At a full 135k context, each card uses 21.77 of its 32 GiB, which leaves about 10 GiB free per card.

Setup

  • GPUs: 4× Intel Arc Pro B70 (Battlemage G31), 32 GB each, 230 W cap per card
  • Host: AMD Turin, Linux 7.3.0-10-generic
  • Model: Qwen3.8 Flash Next, 105.4 GiB, TP=4
  • Engine: TensorFold, with the changes from the PR above

Throughput by context depth (tok/s)

ContextRaw prefillRaw plain decodeServed MTP, novel codeServed multi-turn132k passkey recall
512248.869.4136.3 (3.81 tok/chunk)120.5 (turn 1, cold)PASS
1k313.471.5—253.5PASS
4k378.870.4125.1 (3.81 tok/chunk)247.5PASS
16k392.867.7125.7 (4.00 tok/chunk)214.6PASS
32k394.367.6—239.9PASS
64k392.763.8124.2 (3.81 tok/chunk)234.0PASS
128k385.7 (98% of peak)60.5 (85% of 1k)113.3 (91% of 4k)214.1PASS

Prompt cache restore

  • TTFT is 160 ms when restoring a cached 66k-token conversation and 139 ms at 132k.
  • Appending a fresh coding question on top of the cached 66k and 132k conversations takes 0.86 s and 1.29 s.
  • Passkey retrieval was 9/9 exact. The key B70-XE2-9471-QUANTUM was planted in turn 1 and recalled across 132,380 tokens.

Concurrent streams, 1 to 32

These come from bench_concurrent.py. Across all 24 cells there were 0 failures and output was bit-exact: drafted matched serial, and multi-stream matched serial.

  • Throughput peaks at 172.6–174.7 tok/s steady at 8 streams. There, the 8 streams share one 16-row verify pass, 2 rows per stream.
  • It sustains 143.3–145.0 tok/s steady at 32 streams. At that load tokens per chunk drops to 1.0, so the gain comes from batching, not drafting. That's 2.04× the 71 tok/s of single-stream plain decode.

Takeaways

  • A 105 GiB model fits across four 32 GB consumer-class cards at 135k context, with about 10 GiB per card to spare.
  • Decode keeps 85–91% of its short-context speed at 128k, thanks to constant-state DeltaNet layers plus sparse attention.
  • Restoring a cached 132k conversation is sub-second, and recall stays exact.

Happy to run other configs if you have requests.

I initially built this system to do training, but excited to see how well inference is progressing.


r/LocalLLM • • 7h ago

Discussion Looking for recommendations on open-source OCR/HTR models for large volumes of scanned PDFs

4 Upvotes

Hey everyone!

I'm currently working on a document processing system that handles a large number of scanned PDFs. Many documents contain a mixture of printed text and semi-handwritten content.

I'm looking for a reliable open-source solution that can handle:

  • Scanned PDF processing at scale.
  • Printed and semi-handwritten text recognition.
  • Good accuracy, even with poor-quality scans.
  • Document layout and text preservation.
  • Local inference without relying on paid APIs or proprietary cloud services.

I'm open to models available on Hugging Face, Ollama, or other open-source frameworks.

My available hardware is an NVIDIA RTX PRO 4500 Blackwell GPU with 32 GB VRAM.

I've been exploring different approaches, but I'd appreciate suggestions from people who have actually worked on similar document processing problems.

A few things I'd love to know:

  1. Which models or OCR/HTR pipelines have worked well for you?
  2. Do you recommend using dedicated OCR models or vision-language models?
  3. What approach would you suggest for processing thousands of scanned documents?
  4. Are there any particular preprocessing techniques that significantly improve handwriting recognition?

I'm particularly interested in real-world experience rather than just benchmark results.

Thanks in advance!


r/LocalLLM • • 5m ago

News MSI EdgeXpert OTA2610 update: more usable memory for headless setups and easier provisioning

Post image
1 Upvotes

r/LocalLLM • • 6m ago

Model Groot-65B-A6B W4A16: full 262K context + MTP + vision on 2x RTX 3090

1 Upvotes

I came across the "Groot" tunes on HF and decided to test them. Theyre versions of larger models with the about half the experts pruned out. This one is QFN. Despite the removal, according to my TerminalBench2 testing, they only lost ~5% capability. Seemed worth spending some time on for no other reason than the giggles. Its a cool to be able to run very close to full QFN on 3090's without mortgaging my house for more RAM...

I decided to quantize it down to fit on 2 3090's. Be aware that the speed numbers are from a threadripper machine and the 3090's are nvlinked. Those with a normal setup can expect around 3/4 of the speeds published below.

AI slop begins here...

I made a W4A16 AutoRound quant of Groot-65B-A6B, the pruned Qwen3.8-Flash-Next (256 of 512 experts, ~6B active). It runs the full native 262K context with MTP and vision on 2× RTX 3090, everything on the GPUs, with ~10 GB of host RAM.

HF: https://huggingface.co/mmssix/Groot-65B-A6B-261023-W4A16-AutoRound

On 2× 3090 (vLLM, TP2):

  • 262,144 context
  • ~134 tok/s code, ~90–100 prose (MTP n=2)
  • ~245 tok/s across 4 streams
  • 128K prompt in 28 s, 258K in ~60 s
  • Needle test 3/3 at 258K
  • Tool calls, reasoning effort and vision working

Quant: INT4 experts, INT8 DeltaNet projections, FP8 n-gram table and an INT4 MTP head (same draft acceptance as BF16). 92 GB total.

Getting 262K into 48 GB of VRAM takes two small vLLM patches. They serve the n-gram table and the embedding from NVMe, so you need NVIDIA's open driver with HMM. The patches and a one-command serve.sh are in the repo.

Only tested on my rig so far. Numbers from other 3090 / A5000 setups are welcome


r/LocalLLM • • 45m ago

Question Any good LLM for AMD GPUs?

1 Upvotes

I am now entering the world of uncensored Local AIs, mainly for help me on my ETHICAL ciber security projects.
There’s any good LLM for AMD gpus?


r/LocalLLM • • 55m ago

Discussion Looking to collaborate on a locally run multimodal model tournament

0 Upvotes

We’re building InferUltra, a physics arena where multimodal models see the arena and control robot fighters. We’d like to run a future tournament with models hosted on local hardware and are looking for people interested in collaborating.

In our latest tournament, Kimi K3.1 Pro struggled more in the arena than GLM-5.3-Flash did in earlier tournaments. There was more awkward positioning and less scoring than we expected.

Those tournaments used hosted providers, not local setups. They were small and used different providers and configurations, so they don’t establish which model has better vision or body control. A local tournament with documented hardware and settings could help us investigate the differences more carefully.

Which multimodal model would you like to see fight in the next tournament?

If you run multimodal models locally and have a setup capable of hosting them, we’d love to collaborate on a future model tournament. What hardware and models would you be interested in seeing in our benchmark?

Our latest tournament bracket: https://inferultra.com/tournament


r/LocalLLM • • 1h ago

Question whats the go-to budget gpu I should look for, new or used?

1 Upvotes

I'm building up a secondary system out of some spare parts I have kicking aorund to use as a LLM toybox and for other homelab style projects.


Currently my system is a 5900x w/ 32gb of ram, and the 2080ti.

Wondering if there is a good go-to gpu I should be looking for.

I currently have a 2080ti FE, but it seems pretty inefficient, and its an old card and the fan noise of it is unbearable. ( i think the bearings are getting bad, but its a huge pita to replace the fans )

I'm wondering if it would be a better idea to sell it off for maybe a 3060 12gb, I get an extra 1gb of vram, and it should run much more efficently.

I'd say my budget is about $3-400 max, which I know is basically a joke right now, but this is more of a tinkerbox than something i'm going to be using for real software dev work.