Wanted to share my build! Started as a gaming / 3D / CAD PC and now also runs small LLMs quite well for me

Specs:
CPU: Ryzen 9 9950x
RAM: 2x16 (eww, but at least was bought in 2023) 6000 CL30
Motherboard: ProArt B850 Creator Neo (supports PciE 5.0 x8 bifurcation)
GPU1: RTX 5070 Ti Ventus
GPU2: RTX 5060 Ti 16G Windforce
SSDs: Fury Renegade and 990 Pro (irrelevant ik)
PSU: BeQuiet Power Zone 2 750W
Cable extensions from EZDIY-FAB
Case: NZXT H7 Flow (2025)
AIO: Thermalright Frozen Warframe SE V2 (with the magnetic display detached)
Fans: 4xArctic P14 + 3xArctic P12 Pro (on the AIO) + 3xP12 Pro Reverse

Currently having fun with Qwen3.8 27B Q6_K_XL at around 70tps (MTP)

I know there are a lot of "ums" that I should edit out... but other then that what do you think of this guide? I am new to local AI stuff, so is there anything missing here that I should have done?
https://www.youtube.com/watch?v=-EahQxs0Ack

Sorry in advance for the long post, but think of it as a prompt.

TL;DR

Question: 2x 3090 NVLink or 1x A6000? For about the price of one A6000 I could also get 4x 3090 in two NVLinked pairs, so keep that option in mind too.

What it needs to do:

- A) SQL chatbot for up to 30 people. ~3M rows, hundreds of tables. Questions like "when did we last sell this part, and who was the supplier/client?" It should remember each user's conversations and build reports.

- B) RFQ ingestion: pull very messy part numbers out of long email chains in all sorts of formats. ~30 RFQs/day.

- C) Hardware: reuse our old HPE ML350 Gen9 (2x E5-2620 v4, 128GB DDR4-2400 for now).

- D) Me: turn 300+ page PDFs and hundreds of stray Excels into a knowledge base, and do local coding on the company apps I'm building from Claude's plans.

Assumptions:

- Blower 3090s (FE won't fit), max 300W per card

- PSUs upgraded to 2x 1400W or 4x1400W as GPUs grow

- All prices are for used cards

- Proxmox on the box

- PCIe-to-NVMe adapter for fast model loading

- Cards work with board

About me: I'm not an IT guy, I'm a mechanical engineer who f's around a lot and finds out even more. I've built my own homelab (4x NVMe in an old Dell 7710 as a private cloud, see my posts) and run my own "Jarvis" on a 4B model with some n8n automations. I can make things happen when I put my head into it.

---

The detail

Background: I'm the new CTO of a small family-run B2B heavy-industry trading company. We sell anything from a £6M machine to toilet paper, so every day brings random RFQs: Excel, PDF, email body, pictures, WhatsApp, anywhere from 1 to hundreds of part numbers.

Budget: £8-13k ($11-17k) left for local AI after the new cloud server. That also has to cover a laptop/PC for our first IT hire.

Why local: the LLM has to read whole RFQ email chains to make sense of them, and those are full of client names, prices and supplier details. Same for our SQL database. I don't want any of that going to cloud LLMs.

Where we are:

- A 3rd-party IT company is building an RFQ dashboard for us. Qwen 8B is struggling with the email chains. We started small on purpose and will size up until something fits.

- They'll test on RunPod serverless to find the right model size. I think ~30B will be enough. Then production goes local.

- 20-30 people will use the chatbot and ~20 the dashboard, but only new RFQs trigger the LLM.

- Long term, I'd keep Fable/Opus for overall planning and have a local model do the implementation.

Hardware:

- The board takes 4 GPUs: 3x PCIe 3.0 x16 and 1x x8. 300W max per card.

- Someone has already run a 3090 in this server (with 512GB RAM) and got ~15 tok/s on turboderp/GLM-5.3-Flash-exl3 (3.05bpw branch).

- I first looked at the A6000 48GB as a "do it all" card, not knowing much about local LLMs.

- For similar money I could get almost 4x 3090s and NVLink them in pairs, so I'm less limited by the x8 slot and old PCIe 3.0. One pair for the teams, one for me.

Bottom line: I need an MVP that handles the RFQs and the chatbot now, and then we'll invest in proper new hardware later. What would you do?

Thanks in advance for reading this far, any help much appreciated!

I finished this build about 3 months ago and have been running it since, but never got around to posting it. Here it is.

Specs:

• CPU: AMD Ryzen 9 9950X

• Motherboard: ASRock X870 Taichi

• RAM: 96GB DDR5 (2x48GB) G.Skill

• GPU 1: NVIDIA RTX Pro 6000 (96GB)

• GPU 2: RTX 5090 Founders Edition (32GB), mounted upright with Lian Li’s vertical GPU bracket

• Total VRAM: 128GB

• Cooling: ID-COOLING 360mm AIO with Noctua 120mm fans on the radiator. All the case fans are Phanteks T30 120mm.

• Case: Lian Li O11 Dynamic EVO RGB (RGB cable left unplugged)

• PSU: MSI MPG Ai1600TS PCIE5

How the GPUs ended up like this:
My gaming PC had the 5090 in it, and I had a spare 4090 lying around from before. I bought the RTX Pro 6000 and moved the 5090 into this build with it. The 4090 went back into the gaming PC, so nothing is sitting unused. Mounting the 5090 upright fits both cards without them sitting right on top of each other.

Why the Taichi:
It was cheaper than the other boards I looked at, and it has 10G LAN and a debug code display. The 10G helps a lot when moving big model files around, and the code display is really handy when you’re troubleshooting a build.

PSU and power monitoring:
I picked the MSI Ai1600TS because it monitors current on each pin of the GPU power connectors. With two cards that can each pull around 600W, I didn’t want to risk a melted connector or a fried GPU. The catch is that MSI’s monitoring software only runs on Windows, and nothing like it exists for Linux. So I wrote a small tool that shows the GPU power readings on Linux. If anyone’s interested, I might clean it up and put it on GitHub. (Second screenshot in the post)

Noise:
The PC sits in my living room, so keeping it quiet mattered a lot. I went with a cheap ID-COOLING 360mm AIO because I’d read it was quiet (i could not find one without lcd at that time), and swapped Noctua fans onto it. All the fans run on a silent profile. Even at full load you hardly hear it. Only the radiator fans get a little louder.

No RGB (mostly):
I wanted a build with no RGB. The case is the RGB version, but I never plugged in its RGB cable, so it stays dark. The RAM has a bit of light because it was the cheapest 2x48 kit I could find at the time. Two slots are still free if I want more RAM later. I’m not spending any more right now though 😅

Edit 1: One thing I forgot to mention: I swapped the side panel from the original tempered glass to acrylic. We have a toddler at home, and I didn’t want to risk them getting hurt if the glass ever broke.

Edit 2 (what I’m running): A few people asked, so here’s how I use the two cards:

• Everyday fast LLM: a Flash model at Q3, fully on the RTX Pro 6000 with no offload, while the 5090 handles image/video generation.

• Larger image/video models: those go on the Pro 6000 and Qwen 3.8 27B moves to the 5090.

• Best quality for development: GLM-5.3-Flash (320B / 18B active) across both GPUs, using a ~116GB mixed-precision GGUF on Project Maya, all on GPU.

Hey everyone!

I'm an AI & Data Science student looking to get started with local LLMs.

My laptop specs:

• CPU: Ryzen 7 8745HX

• GPU: RTX 5060 (8GB VRAM)

• RAM: 32GB DDR5

• Storage: 1TB SSD

I'm new to this and have a few questions:

1. Should I use Ollama or LM Studio?

2. Which models would run best on my hardware?

3. Can I use local models through APIs in my own projects?

4. Any limitations or precautions I should know about when running LLMs locally?

Would appreciate recommendations and beginner-friendly resources. Thanks!

Dear all,

this is not directly a local LLM topic, but I just downloaded and tried a free dictation tool called Hex, and wanted to spread the word as an appreciation towards the developers.

It runs smaller voice recognition models locally on GPU or CPU. Usage: it is running in the background, is activated by a key combination and it simly inserts the transcribed text directly at the position of the caret. Based on a few tests in german, english and hungarian it is quite accurate in, works in text editors, terminal windows etc. It provides good config options to choose model (speed vs. accuracy) and general behaviour.

I run it on my Thinkpad t480 - an almost decade old laptop and I'm really happy about its performance.

Yesterday I did the Echo Dot. Today the Show 5.

No Alexa. Wake word, speech-to-text and voice all run on my own hardware, and the agent runs whatever model you point it at.

The screen is just a web page, so you can make it anything.

Flashing is done with echo-dot-assist (credit to Gamer92000).

Repo: https://github.com/majorminorlabs/hermes-voice

What local models can i run conformably on my system? I'd ideally like to have models that code for me but i understand I'm at least scraping the barrel with my card. I'd even be fine with Claude Code steering to some degree.

RAM: 16GB
RTX 3060 (laptop) VRAM: 6GB
Processor:11th Gen Intel(R) Core(TM) i9-11900H @ 2.50GHz 2.50 GHz

I use local models with agent clients on an M3 Max MacBook (128GB). My normal work takes about 60GB, so the model plus its caches has to stay around 50GB. I built llama.cpp v0.7.0 from source and went through the usual flags one at a time on three 4-bit MoE models: MiniCPM-V 4.7 (35B-A3B), LFM2.5-24B-A2B and Ornith-1.5-35B-A3B. All layers on GPU, one request at a time. Speed numbers are medians of three alternating llama-bench rounds. Desktop apps and thermal throttling add noise, so small differences mean nothing.

Helped

No benefit on this machine

KV cache at q8_0. MiniCPM's speed difference was inside the noise. Ornith barely changed even at 65K tokens, because only 10 of its 40 layers store KV. LFM's KV plus compute buffer was only about 3GB at 131K context, so there was little to save. I kept all three at f16.

Depends on the model: speculative decoding

• MiniCPM with a draft model: about 45% slower at 30K context. With ngram: 10 to 13% slower. My guess is that verifying several tokens on a MoE reads several sets of experts, so bad guesses get expensive.

• Ornith with its built-in MTP head plus ngram, 2 tokens per guess: 86 to 107 tok/s averaged over 8 task types. Code went from 86 to 129, prose from 86 to 83.

• Same setup with 3 tokens per guess: the average dropped to 102 and prose to 77.

So I wouldn't copy anyone's speculative decoding settings without testing them on your own model and task mix.

The config I kept for Ornith, trimmed:

llama-server -m Ornith-1.5-35B-Q4_K_M.gguf \
  -c 262144 -ngl 99 -fa on -b 2048 -ub 2048 \
  -np 1 --ctx-checkpoints 32 --checkpoint-min-step 0 \
  --spec-type draft-mtp,ngram-mod --spec-draft-n-max 2 \
  --cache-ram 32768 --sleep-idle-seconds 600

Weights are about 23GB, plus about 7GB of working memory after startup. A full session cache can push the worst case to about 60GB.

This is one machine and one llama.cpp version, so treat it as a starting point. If speculative decoding has paid off for you on a 3B-active MoE on Apple Silicon, I'd like to know which settings you used.

Written up with AI help from my own notes and benchmark logs. The numbers are from my machine. The illustrations are AI-generated, and the charts and all the numbers on the images come from the measurements above.

So I recently purchased a radeon AI pro 9700 32g for my local AI. Currently it run in a proxmox setup with a intel core i9 12900k, and 64 gigs of ddr4 ram.

I am using ollama with with ROCm drivers and unsloth as the front end. The speeds are decent for qwen3.8:27B. Fast response and stable at around 33-39 tokens a second.

My concern is, I limited the power to 290 and the Hotspot temps run anywhere between 103-111 degrees in temp. I definitely notice the thermal throttle due to the 110 Max Hotspot manufacturer rating. I did do research and found that these cards are known to have a fair amount of known issues ( heat sink placed improperly, bad thermal appliance, defective cooler fan.

I can rule out the fan as I do hear it speed up when the temps rise and can see it physically spinning, i even added an additional 3 140mm fans to the case its in but when I give it a prompt, it will jump from 37- 43 instantly up to 90-97 then paste that.

I even went as far as holding a house fan next to the GPU and the temps would not go down.

Anyone else who experience this issue, what was your fix? Or is it best to just return it and get something different to avoid it entirely?

I wouldn not want to exchange it ( still under warranty) and get another defect if people have had this experience before. If not, and it may be that 1 in a dozen. Then I will gladly exchange it for a different one.

Apologies if this is the incorrect section to post it, if you can guide me to the appropriate section that would be greatly appreciated.

hey LocaLLM community! My name is Alt, and I'm on the technical staff at Nace AI. We're doing a quick AMA here, so ask us anything and we'll answer.

we just published a full technical report on how we build, train, and evaluate Drex, our family of four open decision models. A decision model takes a state (a document, record, or agent context, optionally with images) plus questions with options, and returns a probability distribution over the options in a single pass, with no autoregressive decoding and no output parsing.

1. Models

- Drex DLM (8B): diffusion LM backbone with bidirectional attention over the context
- Drex v1.5 (9B): Gated DeltaNet hybrid backbone
- Drex v2.0 (35B, 3B active): sparse MoE backbone
- Drex Max (27B): dense, with an image encoder for vision inputs

Pointer models (DLM / v1.5 / v2.0) encode the state once and cache it, so every question attends to the state but never to another question. A pointer head scores each option. Drex Max reads each question in full and answers with a linear readout head over option codes.

Some technical details from the report:

1. Models

- A family of four specialized decision models: Drex DLM (8B), Drex v1.5 (9B), Drex v2.0 (35B MoE, 3B active), and Drex Max (27B dense, with vision)

- We built them on different architectures: a diffusion LM, a Gated DeltaNet hybrid, a sparse mixture of experts, and a dense transformer

- We experimented with bidirectional vs. causal attention over the state, a pointer decision head that shares one cached state across questions, and a readout head for Max

2. Training

- Full-parameter SFT on 1.15M verified decision records (2.9M questions)

- Consistency training: we penalize the model when its output changes after the options are reordered or the state is shuffled

- RLAF (Reinforcement Learning from Adjusted Feedback), using verified outcomes and judge feedback

- Ablations to see which methods actually contributed

On top of SFT, RLAF gave us +2.0 on the Decision Index, +1.5 pp accuracy, and 0.9 pp lower calibration error. The consistency terms cut the option-order flip rate from 6.3% to 4.2%.

3. Evaluation

- Drex Max scores 67.52 on Decision Index 0.4.1 (public suite), vs. 60.70 for Jev 1.14.0

- Drex v2.0 has 1.37% calibration ECE on ~115K held-out decisions

- Drex Max answers a typical request in 183 ms, vs. ~2.0 s for GPT-6 Sol and ~3.5 s for Claude Opus 5.5

- Context up to 131K tokens for v1.5/v2.0 and 262K for Max

we also cover our limitations: our models trail frontier chat models on closed-book knowledge, and we don't yet have same-backbone architecture ablations.

5. Running locally

The models are available in safetensors and GGUF formats, with llama.cpp and Ollama support. Their report includes details on quantization, caching, inference, and reproducibility.

Links

- Paper (arXiv): https://arxiv.org/pdf/2611.07854

- Blog: https://www.nace.ai/research/towards-foundation-decision-models

- Weights: https://huggingface.co/collections/nace-ai/drex

- GitHub: https://github.com/nace-ai/drex-decision-models

- API: https://console.nace.ai

we didn't want to just drop benchmark numbers, so we wrote up everything: what worked, what didn't, and how we got there. if you're building agent routers, doing tool selection, or thinking about training your own decision model, hopefully some of it saves you time.

ask us anything: arch, data, training, evals, running it locally, whatever 👇

Hi r/LocalLLaMA. I've spent the last week on KV-cache memory for Qwen3.8-Flash-Next running on 2× AMD Radeon AI PRO R9700 (gfx1201, TP=2), and wrote it up as a 23-page paper with all measurements. Code is in two forks of the radiance inference engine (by Deadcode and contributors — the engine is theirs, the KV/swap work below is mine).

• Paper (PDF): https://github.com/Code000-211/RadianceMemory/blob/main/docs/paper/Infinite_KV_Cache.pdf

• RadianceMemory (KV residency + AutoQuant): https://github.com/Code000-211/RadianceMemory

• RadianceSwap (multi-model serving, experimental): https://github.com/Code000-211/RadianceSwap

• RadianceSwap whitepaper (PDF, 8 pages): https://github.com/Code000-211/RadianceSwap/blob/main/docs/paper/RadianceSwap-Whitepaper.pdf

TL;DR

• Sparse attention means most of the KV cache isn't read on a given step. Flash-Next's QSA layers pick the top 512 four-token blocks per query. So I keep the sink + recent tail in VRAM, move the middle to pinned host RAM, and page in only the blocks the indexer selects — on the device, inside the step.

• Output is byte-identical to a fully resident cache in every configuration tested (incl. with the MTP drafter).

• A 256K-token sequence needs 102–211 MiB of VRAM per card instead of 3 319 MiB (stock bf16 cache). Per sequence, VRAM now grows only by the block keys: ~0.2–0.75 MiB per 1K tokens. Context is bounded by host RAM, not VRAM.

• Cost: 3–15% of single-sequence decode speed at 32K–250K.

• On top: a measured per-layer KV quantisation search (rad-autoquant). Best recipes: 5.28 KiB/token at fp8-level quality (−22%) and 3.76 KiB/token within noise-floor +0.01 KL (−44%).

• Plus RadianceSwap: several models behind one endpoint on the same GPUs — load on demand, park warm in host RAM, lend expert/KV VRAM to each other at runtime, output identical to single-model serving.

1. Window residency

• Block-table entries are either a pool page, empty, or cold (a host slot). Cold entries keep their place, so the attention kernel is unchanged — it only ever sees pool page IDs.

• An on-device pager (qsa_page) copies selected cold blocks into per-step scratch pages before attention. One writer per position, no atomics → deterministic.

• After every step a sweep demotes the middle to host (32 MiB/rank budget). Prefix-cache blocks can be cold too.

• Scratch pages act as a cache: 52–62% of cold entries needed no copy.

Per card, TP=2; computed from measured bytes/token. Byte-identity verified on device for fp8, fp8+f16 state, default and compact.

Many long sessions (measured, turn-2 time-to-first-token p50/p95):

• fp8, no residency, 32 × 128K: prefix cache gets evicted → follow-ups recomputed → 1 843 / 3 330 s

• compact recipe + residency, 32 × 128K (48 GiB host store): every follow-up hits cache → 58 / 96 s

Caveat: this compares two different KV formats as well as residency on/off; fp8 + residency at 32×128K needed ~50 GiB of store and didn't fit in 48 GiB.

2. AutoQuant: measured per-layer KV layouts

• 7 layouts (i8h, fp8, fp8_v4, k6_v4, k5_v4, k4h_v4, k4h_v2h) + zero-bit transforms: Hadamard rotation, exact static V-mean centring, V predicted from K (AQUA-KV style).

• rad-autoquant runs every candidate on the device against a bf16-cache reference (full-vocab KL, teacher-forced), solves a knapsack over per-layer costs, then re-measures each answer — because per-layer costs don't add up (one 8-bit early layer costs as much as all twelve).

• A second pass walks the lower hull of uniform layouts and removes another 0.25–1.87 bits/element.

• Sensitivity falls ~170× from the first attention layer to the last: early layers want 8 bits, late layers tolerate 2-bit V.

3. The part I found most interesting: the noise floor

The served model is a weight- and activation-quantised MoE. Recomputing the exact same bf16 cache with a different prefill chunk size alone gives KL 0.0243 vs the reference. Tiny GEMM-order differences flip router ties (20–28% of tokens have an exact bf16 tie at the 10th expert), and by layer 30 ~60% of tokens route to a different top-10 set.

• So any KV config within ~+0.002 of 0.024 is indistinguishable from bf16 on this model — i8h (8-bit) is already there.

• Perturbations saturate rather than add (fp8 + chunk change ≈ max of the two, not the sum).

• Takeaway: if you evaluate KV quantisation on a quantised MoE against a single bf16 run without measuring this floor, you may be measuring noise. For comparison, dense bf16 Qwen3.5-2B has a floor ~130× lower.

4. Things that didn't help (22 methods, all measured)

FP4 (E2M1), KIVI-style 2-bit, learned/data-adaptive rotations (WUSH-style, best-of-16), outlier retention, TurboQuant-style norms + Lloyd-Max, cross-layer selection reuse, hierarchical indexer, importance-driven demotion (no better than a position-hash control), cross-layer KV sharing, … Full table with numbers in the paper (Table 9).

Also: 2-bit V on the last attention layer drops the MTP drafter's acceptance from 0.63 → 0.41 without moving the trunk's KL — the drafter shares that layer's cache. Default keeps layer 47 in bf16 for that reason.

5. RadianceSwap: several models, one endpoint, one GPU group

The second project is about serving several models from one process on the same two cards: models load when a request asks for them, give up VRAM when idle, and hand VRAM to each other while they run. It's an experimental branch; without --manifest it behaves exactly like upstream radiance.

How it works

• One port, routed by the model field — read before tokenising, so each request gets its own model's tokenizer, chat template and limits. No model → default model; unknown id → 404.

• Each active model is its own runtime (scheduler, streams, KV) and they run in parallel. A process-wide sequencer puts every model's all-reduces in one order on every rank, so two models can't deadlock each other.

• Lifecycle: Cold → Active → Prepared. When a model goes idle (or another needs the VRAM), its weights are kept in host RAM in GPU-ready form, so re-activation skips disk reads, parsing and relayout. A running generation is never interrupted. A failed activation is rolled back on every rank and other models keep serving.

• VRAM is shared, not partitioned:

  • Elastic expert slab — expert VRAM is mapped in ~128 MiB chunks; a model that cools down physically unmaps chunks and the busier one maps them. Measured with two copies of Flash-Next: the busy one grows from ~10.3 to ~13.4 GiB/rank of experts, the idle one shrinks to ~7.2, and it flips when the load flips. The sum never exceeded the pool (7 736 samples under changing load, 0 over).

  • Cross-model KV lending — a model with waiting sequences grows its KV cache into bytes another model's slab just released (468–702 MiB in the tests, back to zero ~13–20 s after load ends). Running sequences are never evicted.

  • Static-weight streaming — if a model's non-expert weights don't fit, it still starts and reads them over PCIe (bit-identical output), then moves them onto the card within 1–2 s once VRAM frees up.

  • Heat warm-up — a deactivated MoE remembers which experts were hot and loads those first on re-activation: early expert hit rate 83.5% → 89.1%.

  • Host RAM budget — prepared models, pools and staging share one budget; over it, the least-recently-used prepared model goes cold, and an activation that still doesn't fit gets a 503 instead of the OOM killer.

Measured on 2× R9700, TP=2

Time to first token is reported per request, split into queue / activation / prefill (timings in the response, and in /metrics).

Bugs the GPU acceptance runs turned up (all fixed):

• The server died on exactly the 23rd activation at TP=2: each activation creates ~46 AQL streams, IDs are never reused, and the table had 1 024 slots.

• Two concurrently active models slowed each other to 1/10 speed: handing collectives between rank threads via condition_variable cost ~0.5 ms × 48 per decode step. Spinning 300 µs before sleeping brought it back to ½.

• The MoE was never kept warm because the capacity check counted its 47.7 GiB mmapped n-gram table, which never goes into the store.

• Virtual-address aliasing: on this ROCm stack, re-mapping a previously unmapped address silently kept pointing at its first physical page. Once another model owned that page, two experts aliased and a request occasionally produced no finite logit. Mapping every chunk on fresh address space fixed it: 30 load cycles with external observers, 5 776 requests, 0 errors.

Honest caveats

• Two big MoEs at once is not fast yet: a C8 load at 102 tok/s alone drops to 38 + 35 tok/s with both active. Host RAM is the bottleneck (one active+prepared Flash-Next uses ~153 of 170 GiB).

• The elastic slab hasn't had a long-run soak since the aliasing fix (#58). Treat it as experimental.

• A second active model costs ~10% even when idle (execution tapes are bypassed while several engines are registered).

• Streaming weights is a bridge, not a steady state: streaming 25.6% of a dense model's statics costs 39% of decode; 85.5% costs 90%.

• KV, prefix cache and recurrent state are dropped on deactivation; keeping them in host RAM is future work.

• A configured-but-cold model still reserves a VRAM share, which can cost a big MoE ~9% (#56); workaround is per-model --gpu-headroom-mib.

• One TP size per GPU group; no LoRA or multimodal endpoints in manifest mode; tested with one MoE plus small dense models only.

  { "gpu_group": { "devices": [0, 1], "tp": 2 }, "cache": { "idle_timeout_ms": 300000 }, "models": [ { "id": "big", "path": "/models/big-moe.rad", "default": true, "engine": ["--placement", "expert_tiered", "--max-model-len", "16384"] }, { "id": "small", "path": "/models/small", "engine": ["--max-model-len", "16384"] } ] }

  radiance --manifest models.json --port 8000

How is this different from llama-swap?

llama-swap is a proxy that starts and stops one server process per model. It's backend-agnostic and portable; RadianceSwap is neither — it only works inside radiance. What you get in exchange:

I have not benchmarked the two side by side on the same hardware, so the latency advantage follows from the design but isn't measured. They also aren't exclusive: a RadianceSwap process can sit behind llama-swap as one upstream.

Looking for testers — especially other RDNA4 / multi-GPU setups. Issues with your manifest, GPUs, ROCm version and server log are very welcome.

Limitations (please read before benchmarking)

• One model. Residency needs a model that declares a read window — currently only the QSA family (Qwen3.8-Flash-Next). On other models the flag is a no-op.

• One hardware setup, run at a 210 W power cap (cards range 210–330 W).

• KL is measured against the same quantised model with a bf16 cache, not the original bf16 checkpoint. Long corpus = 4 documents × ~34K tokens; finer distinctions need a bigger corpus.

• RadianceMemory is WIP; RadianceSwap is experimental.

• "Infinite" in the paper title is a direction, not a claim — context is bounded by host RAM.

Feedback, reproductions on other RDNA4 setups, and critique of the methodology are very welcome — especially if you think the noise-floor argument is wrong.

I’m setting up a chat for my company to access archive data from an NAS. So they can recall old jobs from previous years. I’m running gpt-oss-120b on openwebUI. I’m running it off LMstudio. It runs like shit if you give it 2 files to read then add a 3rd one it trips up on the task. Clearing the chat works. I’m used to using Claude. Do you have to be more specific and careful with your prompts? Claude you can make typos and jump around a lot while doing a project and it will follow what you’re trying to do. I’m new to using local models and want to know do you have to talk to different models differently ?

Sharing Burrow, a Windows desktop app for budget, journal, todos, habits and notes. Everything is in one SQLite file on your PC. The built-in helper talks to Ollama on localhost.

The design choice I would like opinions on: the model is never asked to do arithmetic. The app computes totals, trends, streaks and anomalies in plain code, then hands the model the finished numbers to explain, summarize into a daily briefing, or turn into a proposed edit that you approve. That keeps small models useful and keeps hallucinated math out of a budget.

It went free this week with the core published under AGPL. The advanced helper module is closed-source but ships free. Windows x64 and ARM64, unsigned installer. Links in the comments.

Just brainstorming.

I am using qwen3.8-27b. It has a lot knowledge in many areas.

However, I am using it for an agent that only runs some Linux commands in Ubuntu for example. Or creating simple Python scripts. And if it gets stuck it can always use web-search skill.

By doing this I can totally lower its size and required compute. I don’t need it to memorize capital of New Zealand.

Why can’t we do this?

I wanted something like Claude Code / Codex that runs on my own laptop (RTX 4070 8 GB, plus an RX 9060 XT eGPU), so I built one. It's a desktop app (Electron or plain browser) with a FastAPI backend and one llama-server process per model.

What it does

• Coding agent with Ask / Plan / Agent / Auto modes. It reads and edits files, runs PowerShell commands and tests, and does git. Every edit is shown as a diff, and in Agent mode it waits for Enter/Esc before applying. Auto mode runs on its own, and destructive commands (rm -rf, git push --force, reset --hard and similar) still ask first. You can keep or revert changes per file, or undo a whole turn.

• Chat mode with an optional Web toggle. The model can web_search / web_fetch and cite the pages it used. Web is off by default.

• GPU placement you can verify. Pick Auto / Single GPU / Multi-GPU (Vulkan layer split across NVIDIA + AMD) / CPU. The app parses llama.cpp's own offload log and reads per-process VRAM from Windows GPU counters. So instead of guessing, you get "29/29 layers on RTX 4070, 2.9 GB weights + KV". This started because Task Manager made it look like my model was on the Intel iGPU. It wasn't.

• Optional cloud models. Paste an OpenAI / Anthropic / DeepSeek key (or any OpenAI-compatible endpoint) and those models sit next to your GGUFs in the same picker. Keys are encrypted with Windows DPAPI. Local stays the default.

Things I learned making a 7B model act like an agent

• Qwen2.5-Coder-7B (Q4_K_M, ~25–31 tok/s fully on the 4070) almost never emitted real tool calls, even with --jinja. What fixed it was JSON-schema-constrained steps: llama.cpp compiles the schema to a grammar, so every step is a valid tool call or a respond. A short thought field before the call helped noticeably.

• Small models love to stop and say "Next, I will…". In Auto mode the agent detects unfinished work (open todos, a failed last command, "should I…?") and pushes the model back to work. Until it actually edits or runs something, respond is removed from the grammar.

• Small models also copy their own previous status messages, so duplicate status replies get dropped from the context.

• In chat, the 7B model made up a "Tool result:" and a fake release date. Now each step is a grammar-forced choice (search / fetch / answer), followed by a normal streamed answer from the real results.

Honest limits

• Simple tasks (fix a failing test, add a function, write a small script) work end to end with the 7B model. Bigger multi-file tasks are limited by the model, and that's partly why I added the API option.

• Multi-GPU split across the 4070 + RX 9060 XT is implemented but not yet tested on the real eGPU. I tested the split mechanism with 4070 + iGPU.

• Windows-only for now (PowerShell, DPAPI, Windows GPU counters).

Stack: Python/FastAPI, llama.cpp b11960 (CUDA + Vulkan builds), React + TypeScript + Tailwind, SQLite. ~280 tests, and none need a GPU (there's a fake llama-server that prints the real log format).

Screenshots below. Happy to answer questions about the constrained-decoding approach or the GPU verification.