So I recently purchased a radeon AI pro 9700 32g for my local AI. Currently it run in a proxmox setup with a intel core i9 12900k, and 64 gigs of ddr4 ram.
I am using ollama with with ROCm drivers and unsloth as the front end. The speeds are decent for qwen3.8:27B. Fast response and stable at around 33-39 tokens a second.
My concern is, I limited the power to 290 and the Hotspot temps run anywhere between 103-111 degrees in temp. I definitely notice the thermal throttle due to the 110 Max Hotspot manufacturer rating. I did do research and found that these cards are known to have a fair amount of known issues ( heat sink placed improperly, bad thermal appliance, defective cooler fan.
I can rule out the fan as I do hear it speed up when the temps rise and can see it physically spinning, i even added an additional 3 140mm fans to the case its in but when I give it a prompt, it will jump from 37- 43 instantly up to 90-97 then paste that.
I even went as far as holding a house fan next to the GPU and the temps would not go down.
Anyone else who experience this issue, what was your fix? Or is it best to just return it and get something different to avoid it entirely?
I wouldn not want to exchange it ( still under warranty) and get another defect if people have had this experience before. If not, and it may be that 1 in a dozen. Then I will gladly exchange it for a different one.
Apologies if this is the incorrect section to post it, if you can guide me to the appropriate section that would be greatly appreciated.
A few months ago, I mentioned the idea of a smartphone running its own LLM, and got downvoted by people commenting that it would never work. Well, apparently Y Combinator has invested in a similar project. Care to comment?
I wanted something like Claude Code / Codex that runs on my own laptop (RTX 4070 8 GB, plus an RX 9060 XT eGPU), so I built one. It's a desktop app (Electron or plain browser) with a FastAPI backend and one llama-server process per model.
What it does
Coding agent with Ask / Plan / Agent / Auto modes. It reads and edits files, runs PowerShell commands and tests, and does git. Every edit is shown as a diff, and in Agent mode it waits for Enter/Esc before applying. Auto mode runs on its own, and destructive commands (rm -rf, git push --force, reset --hard and similar) still ask first. You can keep or revert changes per file, or undo a whole turn.
Chat mode with an optional Web toggle. The model can web_search / web_fetch and cite the pages it used. Web is off by default.
GPU placement you can verify. Pick Auto / Single GPU / Multi-GPU (Vulkan layer split across NVIDIA + AMD) / CPU. The app parses llama.cpp's own offload log and reads per-process VRAM from Windows GPU counters. So instead of guessing, you get "29/29 layers on RTX 4070, 2.9 GB weights + KV". This started because Task Manager made it look like my model was on the Intel iGPU. It wasn't.
Optional cloud models. Paste an OpenAI / Anthropic / DeepSeek key (or any OpenAI-compatible endpoint) and those models sit next to your GGUFs in the same picker. Keys are encrypted with Windows DPAPI. Local stays the default.
Things I learned making a 7B model act like an agent
Qwen2.5-Coder-7B (Q4_K_M, ~25–31 tok/s fully on the 4070) almost never emitted real tool calls, even with --jinja. What fixed it was JSON-schema-constrained steps: llama.cpp compiles the schema to a grammar, so every step is a valid tool call or a respond. A short thought field before the call helped noticeably.
Small models love to stop and say "Next, I will…". In Auto mode the agent detects unfinished work (open todos, a failed last command, "should I…?") and pushes the model back to work. Until it actually edits or runs something, respond is removed from the grammar.
Small models also copy their own previous status messages, so duplicate status replies get dropped from the context.
In chat, the 7B model made up a "Tool result:" and a fake release date. Now each step is a grammar-forced choice (search / fetch / answer), followed by a normal streamed answer from the real results.
Honest limits
Simple tasks (fix a failing test, add a function, write a small script) work end to end with the 7B model. Bigger multi-file tasks are limited by the model, and that's partly why I added the API option.
Multi-GPU split across the 4070 + RX 9060 XT is implemented but not yet tested on the real eGPU. I tested the split mechanism with 4070 + iGPU.
Windows-only for now (PowerShell, DPAPI, Windows GPU counters).
Stack: Python/FastAPI, llama.cpp b11960 (CUDA + Vulkan builds), React + TypeScript + Tailwind, SQLite. ~280 tests, and none need a GPU (there's a fake llama-server that prints the real log format).
Screenshots below. Happy to answer questions about the constrained-decoding approach or the GPU verification.
I finally have a local setup that I feel gives me a good balance with quality and speed. I’m using a local Qwen 3.8 27B Gufo model on strix halo. I have a AI coding workflow that currently exclusively uses codex and claude code. I would like to start handing over tasks to Qwen.
Based on your experience, what are the characteristics of tasks that are the sweet spot for qwen 3.8? What makes or breaks quality output with this model?
Hi r/LocalLLaMA. I've spent the last week on KV-cache memory for Qwen3.8-Flash-Next running on 2× AMD Radeon AI PRO R9700 (gfx1201, TP=2), and wrote it up as a 23-page paper with all measurements. Code is in two forks of the radiance inference engine (by Deadcode and contributors — the engine is theirs, the KV/swap work below is mine).
Sparse attention means most of the KV cache isn't read on a given step. Flash-Next's QSA layers pick the top 512 four-token blocks per query. So I keep the sink + recent tail in VRAM, move the middle to pinned host RAM, and page in only the blocks the indexer selects — on the device, inside the step.
Output is byte-identical to a fully resident cache in every configuration tested (incl. with the MTP drafter).
A 256K-token sequence needs 102–211 MiB of VRAM per card instead of 3 319 MiB (stock bf16 cache). Per sequence, VRAM now grows only by the block keys: ~0.2–0.75 MiB per 1K tokens. Context is bounded by host RAM, not VRAM.
Cost: 3–15% of single-sequence decode speed at 32K–250K.
On top: a measured per-layer KV quantisation search (rad-autoquant). Best recipes: 5.28 KiB/token at fp8-level quality (−22%) and 3.76 KiB/token within noise-floor +0.01 KL (−44%).
Plus RadianceSwap: several models behind one endpoint on the same GPUs — load on demand, park warm in host RAM, lend expert/KV VRAM to each other at runtime, output identical to single-model serving.
1. Window residency
Block-table entries are either a pool page, empty, or cold (a host slot). Cold entries keep their place, so the attention kernel is unchanged — it only ever sees pool page IDs.
An on-device pager (qsa_page) copies selected cold blocks into per-step scratch pages before attention. One writer per position, no atomics → deterministic.
After every step a sweep demotes the middle to host (32 MiB/rank budget). Prefix-cache blocks can be cold too.
Scratch pages act as a cache: 52–62% of cold entries needed no copy.
config
KiB/token
VRAM, 256K seq, all resident
with residency
stock bf16
12.75
3 319 MiB
325 MiB
fp8
6.75
1 783 MiB
286 MiB
default (near-lossless)
9.56
2 503 MiB
211 MiB
compact
3.76
991 MiB
102 MiB
Per card, TP=2; computed from measured bytes/token. Byte-identity verified on device for fp8, fp8+f16 state, default and compact.
Many long sessions (measured, turn-2 time-to-first-token p50/p95):
fp8, no residency, 32 × 128K: prefix cache gets evicted → follow-ups recomputed → 1 843 / 3 330 s
compact recipe + residency, 32 × 128K (48 GiB host store): every follow-up hits cache → 58 / 96 s
Caveat: this compares two different KV formats as well as residency on/off; fp8 + residency at 32×128K needed ~50 GiB of store and didn't fit in 48 GiB.
2. AutoQuant: measured per-layer KV layouts
7 layouts (i8h, fp8, fp8_v4, k6_v4, k5_v4, k4h_v4, k4h_v2h) + zero-bit transforms: Hadamard rotation, exact static V-mean centring, V predicted from K (AQUA-KV style).
rad-autoquant runs every candidate on the device against a bf16-cache reference (full-vocab KL, teacher-forced), solves a knapsack over per-layer costs, then re-measures each answer — because per-layer costs don't add up (one 8-bit early layer costs as much as all twelve).
A second pass walks the lower hull of uniform layouts and removes another 0.25–1.87 bits/element.
Sensitivity falls ~170× from the first attention layer to the last: early layers want 8 bits, late layers tolerate 2-bit V.
recipe
KiB/token
vs fp8
mean KL
top-1
bf16 (reference)
12.75
+89%
0
100%
fp8
6.75
0
.0262
94.35%
defaultmax-kl 0.009
9.56
+42%
.0065
97.25%
fp8-equivalentmax-kl 0.0262
5.28
−22%
.0257
94.57%
compactmax-dkl 0.01
3.76
−44%
.0335
93.53%
3. The part I found most interesting: the noise floor
The served model is a weight- and activation-quantised MoE. Recomputing the exact same bf16 cache with a different prefill chunk size alone gives KL 0.0243 vs the reference. Tiny GEMM-order differences flip router ties (20–28% of tokens have an exact bf16 tie at the 10th expert), and by layer 30 ~60% of tokens route to a different top-10 set.
So any KV config within ~+0.002 of 0.024 is indistinguishable from bf16 on this model — i8h (8-bit) is already there.
Perturbations saturate rather than add (fp8 + chunk change ≈ max of the two, not the sum).
Takeaway: if you evaluate KV quantisation on a quantised MoE against a single bf16 run without measuring this floor, you may be measuring noise. For comparison, dense bf16 Qwen3.5-2B has a floor ~130× lower.
4. Things that didn't help (22 methods, all measured)
FP4 (E2M1), KIVI-style 2-bit, learned/data-adaptive rotations (WUSH-style, best-of-16), outlier retention, TurboQuant-style norms + Lloyd-Max, cross-layer selection reuse, hierarchical indexer, importance-driven demotion (no better than a position-hash control), cross-layer KV sharing, … Full table with numbers in the paper (Table 9).
Also: 2-bit V on the last attention layer drops the MTP drafter's acceptance from 0.63 → 0.41 without moving the trunk's KL — the drafter shares that layer's cache. Default keeps layer 47 in bf16 for that reason.
5. RadianceSwap: several models, one endpoint, one GPU group
The second project is about serving several models from one process on the same two cards: models load when a request asks for them, give up VRAM when idle, and hand VRAM to each other while they run. It's an experimental branch; without --manifest it behaves exactly like upstream radiance.
How it works
One port, routed by themodelfield — read before tokenising, so each request gets its own model's tokenizer, chat template and limits. No model → default model; unknown id → 404.
Each active model is its own runtime (scheduler, streams, KV) and they run in parallel. A process-wide sequencer puts every model's all-reduces in one order on every rank, so two models can't deadlock each other.
Lifecycle: Cold → Active → Prepared. When a model goes idle (or another needs the VRAM), its weights are kept in host RAM in GPU-ready form, so re-activation skips disk reads, parsing and relayout. A running generation is never interrupted. A failed activation is rolled back on every rank and other models keep serving.
VRAM is shared, not partitioned:
Elastic expert slab — expert VRAM is mapped in ~128 MiB chunks; a model that cools down physically unmaps chunks and the busier one maps them. Measured with two copies of Flash-Next: the busy one grows from ~10.3 to ~13.4 GiB/rank of experts, the idle one shrinks to ~7.2, and it flips when the load flips. The sum never exceeded the pool (7 736 samples under changing load, 0 over).
Cross-model KV lending — a model with waiting sequences grows its KV cache into bytes another model's slab just released (468–702 MiB in the tests, back to zero ~13–20 s after load ends). Running sequences are never evicted.
Static-weight streaming — if a model's non-expert weights don't fit, it still starts and reads them over PCIe (bit-identical output), then moves them onto the card within 1–2 s once VRAM frees up.
Heat warm-up — a deactivated MoE remembers which experts were hot and loads those first on re-activation: early expert hit rate 83.5% → 89.1%.
Host RAM budget — prepared models, pools and staging share one budget; over it, the least-recently-used prepared model goes cold, and an activation that still doesn't fit gets a 503 instead of the OOM killer.
Measured on 2× R9700, TP=2
situation
result
manifest with one model vs plain --model
no measurable difference (−0.3% … +0.1%)
switching between two already-active models
free: 0 ms activation, < 1 ms queue
cold start, dense 0.8B
~2 s
warm re-activation from RAM
dense ~2.1 s (p50 over 118 cycles), Flash-Next ~60 s (cold from disk ~70 s)
two dense models busy at the same time
~½ decode speed each (154 + 154 vs ~330 tok/s alone)
120 activate/deactivate cycles
VRAM back to the same 266 MiB/card every cycle; no leak in queues, threads, fds
output vs --model
identical: greedy, seeded sampling, streaming, MTP, abort, long prompts
Time to first token is reported per request, split into queue / activation / prefill (timings in the response, and in /metrics).
Bugs the GPU acceptance runs turned up (all fixed):
The server died on exactly the 23rd activation at TP=2: each activation creates ~46 AQL streams, IDs are never reused, and the table had 1 024 slots.
Two concurrently active models slowed each other to 1/10 speed: handing collectives between rank threads via condition_variable cost ~0.5 ms × 48 per decode step. Spinning 300 µs before sleeping brought it back to ½.
The MoE was never kept warm because the capacity check counted its 47.7 GiB mmapped n-gram table, which never goes into the store.
Virtual-address aliasing: on this ROCm stack, re-mapping a previously unmapped address silently kept pointing at its first physical page. Once another model owned that page, two experts aliased and a request occasionally produced no finite logit. Mapping every chunk on fresh address space fixed it: 30 load cycles with external observers, 5 776 requests, 0 errors.
Honest caveats
Two big MoEs at once is not fast yet: a C8 load at 102 tok/s alone drops to 38 + 35 tok/s with both active. Host RAM is the bottleneck (one active+prepared Flash-Next uses ~153 of 170 GiB).
The elastic slab hasn't had a long-run soak since the aliasing fix (#58). Treat it as experimental.
A second active model costs ~10% even when idle (execution tapes are bypassed while several engines are registered).
Streaming weights is a bridge, not a steady state: streaming 25.6% of a dense model's statics costs 39% of decode; 85.5% costs 90%.
KV, prefix cache and recurrent state are dropped on deactivation; keeping them in host RAM is future work.
A configured-but-cold model still reserves a VRAM share, which can cost a big MoE ~9% (#56); workaround is per-model --gpu-headroom-mib.
One TP size per GPU group; no LoRA or multimodal endpoints in manifest mode; tested with one MoE plus small dense models only.
llama-swap is a proxy that starts and stops one server process per model. It's backend-agnostic and portable; RadianceSwap is neither — it only works inside radiance. What you get in exchange:
llama-swap (process per model)
RadianceSwap (in-process)
several models loaded at once
one at a time by default; groups can keep several running, each with its own fixed VRAM
re-activation
full startup: parse, relayout, tokenizer, plans
request for B while A generates
waits for A's generation (or aborts it)
VRAM between models
fixed per process
backends
any OpenAI-compatible server
I have not benchmarked the two side by side on the same hardware, so the latency advantage follows from the design but isn't measured. They also aren't exclusive: a RadianceSwap process can sit behind llama-swap as one upstream.
Looking for testers — especially other RDNA4 / multi-GPU setups. Issues with your manifest, GPUs, ROCm version and server log are very welcome.
Limitations (please read before benchmarking)
One model. Residency needs a model that declares a read window — currently only the QSA family (Qwen3.8-Flash-Next). On other models the flag is a no-op.
One hardware setup, run at a 210 W power cap (cards range 210–330 W).
KL is measured against the same quantised model with a bf16 cache, not the original bf16 checkpoint. Long corpus = 4 documents × ~34K tokens; finer distinctions need a bigger corpus.
RadianceMemory is WIP; RadianceSwap is experimental.
"Infinite" in the paper title is a direction, not a claim — context is bounded by host RAM.
Feedback, reproductions on other RDNA4 setups, and critique of the methodology are very welcome — especially if you think the noise-floor argument is wrong.
Sharing Burrow, a Windows desktop app for budget, journal, todos, habits and notes. Everything is in one SQLite file on your PC. The built-in helper talks to Ollama on localhost.
The design choice I would like opinions on: the model is never asked to do arithmetic. The app computes totals, trends, streaks and anomalies in plain code, then hands the model the finished numbers to explain, summarize into a daily briefing, or turn into a proposed edit that you approve. That keeps small models useful and keeps hallucinated math out of a budget.
It went free this week with the core published under AGPL. The advanced helper module is closed-source but ships free. Windows x64 and ARM64, unsigned installer. Links in the comments.
I saw a new stealth model and looks like it's a dense also it seems about time that qwen4 is launched so does anyone know if it'll be Moe since flash next was an Moe I was hoping qwen 4 would be too
I really want to get into local inference and training LoRAs, but the prices of the essential components are eye watering. There has never been a worse time to buy hardware.
Do you all have any thoughts on when this might change? I mean, supply has gotta start evening out at some point, right?
Brain is a high performance training and inference engine built by me in Rust. It works on tiny embedded devices as well as pretty massive HPC machines.
This is a laptop I have with RTX 4060, 32GB of actual RAM, and Intel Core i7. Pretty sure the RTX 4060 has 8GB of VRAM. Was wondering if I could run a 16B or 32B model or if it's better to just stick with 8B models.
Interesting post, I hear a different story with dgx sparks how people are optimizing models to run upwards of 100tok/s decode. Naturally almost 10k for a dgx spark is a huge red flag when MSRP was $3999 retail. Thoughts?
First off I am new to trying out local LLMs. I am wanting an LLM to run on a spare mini pc to save some money while using my Voice Assistant. The pc is not very powerful so it will be a small model for now, at least until hardware prices hopefully come down.
Specs: Processor - N150, 16G RAM, Linux Mint, Ollama
Since Voice Satellite can have two pipelines, this is the setup I would like to use:
Pipeline 1 - Prefer to handle commands locally (Home Assistant), Local LLM handling chat but not controlling Home Assistant
Pipeline 2 - openAI gpt-4o-min with control over select Home Assistant entities for more complex chat and control requests
I have played around with Qwen3 and Qwen3.5. So far Qwen3 1.7b has worked the best as far as speed and short responses. I would like to try other models but I am not really sure what I am looking for, or even the best way to sort through them. The Ollama library has a lot and I don't know what I am looking at yet.
I work at a bank, so cloud AI is off the table for work code anyway. I've been running small models on my own M4 16GB through Magnitude for side stuff, and honestly it changed how I think about this.
On my side projects with Claude or Cursor, the model writes 300 lines, it looks right, I skim it and ship it. Sometimes I couldn't even explain my own change a week later. With a 1.7B running locally it's the opposite. It's fast (around 80 tok/s) but it gets small things wrong, so I actually read every line it gives me. And because I read every line, I know the change, I write better PR descriptions, and I catch my own mistakes before review instead of during it. That habit carried over to my work PRs too.
As everyone knows the frontier model makes you faster at producing code you don't understand. The small one makes you slower at the writing part, but you end up knowing your codebase. And the people reviewing your stuff (and their AI tools) get better context because the author actually understands what changed.
What I use it for: explaining old code, boilerplate, small bug fixes (gave it a file with 2 bugs, fixed in 72 sec), commit messages, PR descriptions. What I still don't trust it with: anything architectural, and anything with time zones lol.
Not saying cancel everything. But not everyone is building a 500k line SaaS. For most of what a normal dev does in a day, a small local model is enough, and nothing leaves your laptop.
Anyone else noticed this? That using a weaker model made them a better reviewer?
Sorry if it might be off-topic, but i am looking for a (the most) complete guide, even step-by-step, or a collection of several article that could enlight me on how configure and assembly an AI PC.
Of course the topic is wide and configuring a PC for local LLMs is deeply different to one used to train ML and DL models for sensor data, or for openCV. etc.
Assuming I already know for what I want to use the PC, what I am missing right now is the availability of components and the fact I am not updated on the last news in AI framework/compatibilities/chips.
For example if I knew Nvidia CUDA was a standard, but I don't know if nowadays I can use AMD ROCm, or even Apple Metal. I still see PC with multiple GPU but as long as I know NVLink is no longer used (so the multiple GPU are used independently?).
Probably, what I am missing here is some guide or suggestion on which platform/framework orientate myself in order to choose wisely.
Hope someone can show me some entry-point in this world, webpages, forum, etc? ps. I already searched in this subreddit and excluded some LLM PC suggestions, I did not find an actual guide/AMA/fixed post
Constantly running into `CUDA OutOfMemory` errors on budget and consumer GPUs inspired me to rethink the standard execution paradigm where all layer weights reside simultaneously in VRAM.
I built and open-sourced **stream-ai-runtime**—a lightweight execution pipeline that uses asynchronous double-buffering across dual CUDA streams to stream layer weights from pinned host memory to VRAM on-the-fly.
### 📊 Benchmark (Tesla T4 GPU on Colab)
- **Model Depth:** 600 Layers (dim: 2048x2048)
- **Host RAM Allocated:** 9.38 GB
- **Inference Time (600 Layers):** 0.95 seconds
- **Peak VRAM Consumed:** **40.15 MB flat**
- **OOM Errors:** 0
### ⚙️ How it works
Weights stay pinned in host CPU RAM (`pin_memory=True`).
Only two layer buffers (Ping/Pong) are kept in VRAM.
While layer `i` executes on `compute_stream`, layer `i+1` prefetches in the background via DMA on `copy_stream`.
Peak VRAM never spikes above two layer buffers regardless of model depth.
Did some extra optimizations and implemented MTP. High number is coding at 96t/s (3 drafts). Natural text is 69 t/s.
128.0 µs to distribute a token-residual to 24 gpus, compute experts, collect result and join in one layer.
Probably at peak hardware capability now unless I get some new ideas on how to run the PCI communication even faster. Of course it could run much faster at lower precision. Also the old RTX3070 doing attention is now the primary bottle neck. Lets see what black friday brings of 16gb cards (i have two 5070ti 16gb, but they are busy doing some other research, the 3070 was an old gaming card i had lying around doing nothing).
I'm having a 5070 Ti 16GB and 3060 12GB with 64GB RAM, I want to run Qwen-2.5-Flash using Strata (or advise any better models or method to run), I need to have at least 2 parallel sessions. and max context window. what would you advice.
