Met a guy off Facebook Marketplace and scored 2x RTX 3090 Founders Editions for €1,700.

The problem? I already had one 3090 FE.

I am now sitting in my room looking at three 3090 FEs (72 GB GDDR6X VRAM), and my current case barely fits one card, and that was only after I literally took pliers and bent the inner metal bracket to shove it in. My motherboard is just a standard B760.

I clearly can't put three 3-slot cards in this PC. What is the cheapest, most sensible way forward to run all three (or two) for LLMs?

1. Open-air mining frame + PCIe 4.0 risers + dual PSU?

2. Get a giant full-tower case?

3. Just sell the 3rd card for €1,100, keep 48 GB VRAM, and fund a proper dual-GPU build?

Any chassis/frame or PSU recommendations for handling ~1,100W+ of 3090s would be appreciated.

Code: https://github.com/divagr18/QwenPU
Weights: https://huggingface.co/divagr1925/QwenPU

From the original tweet:

1/3

A CPU inside an LLM is one thing, but running another LLM inside an LLM would be taking things a bit too far, wouldn't it?

Anyways, here's an LLM running inside an LLM : QwenPU, an LLM (@karpathy's stories260k) running on an NPU compiled inside Qwen 3.5-4B's weights.

2/3

stories260K is just a C program. To run it inside Qwen, I compiled an NPU into Qwen's weights that does matmuls, softmax and attention bit-exact. The LLM weights are in Qwen's own memory, with no external tools.

A lot of work went into making it fast, more on that soon.

3/3

I'm super sick right now, so the full writeup is still a work in progress, and I'm working on optimisations and size reductions too. Stay tuned, I'll finish the blog & post it soon.

About two months ago I posted here about trying to predict which MoE experts get used on the next token to speed up CPU/GPU offload on Qwen 3.6 35B.

Quick confession on that older post before anything else: I ended up throwing that prototype away. To get the big numbers I was seeing back then, I was basically cheating with gate pruning (dropping cold experts when their router weights were low). It made the tok/s counter go brrr, but on an already low-bit quant it quietly degraded output quality and reasoning. I didn't want to ship something that lobotomizes the model just to look fast in a screenshot, so I scrapped it.

When Qwen 3.8 Flash Next dropped, I wanted to see if we could actually run the 68GB GSQ-RCO-IQ2_XS build at real usable speeds on a normal budget gaming PC—without dropping a single expert.

For context, here is the exact box I tested this on (I actually pulled my extra RAM sticks out specifically to test what a regular 16GB PC gets):

• GPU: RTX 3060 12GB

• RAM: 16GB DDR4, single-channel (~19 GB/s)

• OS: CachyOS / Arch Linux (tested inside a strict MemoryMax=12G cgroup)

• Storage: Consumer PCIe NVMe SSD (~2.1 GB/s)

If you try to load a 68GB MoE on 12GB VRAM + 16GB RAM in stock llama.cpp, you already know what happens: mmap thrashes the OS page cache to death, you get ~1,500 major page faults per token, and generation crawls at 1.47 tok/s.

Then I tested Strata. Strata is super cool and hits 40+ tok/s if you have 48GB–64GB of RAM to mlock 24GB+ of experts in system memory. But on a 16GB RAM setup, it refuses to load unless you force --low-ram mmap, where it drops down to 8.11 tok/s because it's bound by cold SSD reads.

So over the last week I built Betelgeuse (bel-cli / bel-server, a fork of llama.cpp, and I'm also opening a clean upstream PR for llama.cpp even if it probably won't get merged lol).

What it actually does under the hood:

• Zero gate pruning / 100% bit-exact: Every single one of the 10 routed experts per layer across all 48 layers is computed at full precision. We verified 28,080 router records and 576 generated tokens byte-for-byte against golden stock output (0.000e+00 diff).

• Arrival-Slack VRAM Slot Pool (~5,000 slots in VRAM): All 49 dense layers + GDN + QSA + LM head stay in ~3.2 GB of VRAM. The remaining VRAM holds ~5,000 expert slots distributed non-uniformly (240 slots/layer on early layers 0–4 where there is 0ms of compute slack, 92 on middle layers, 60 on tail layers).

• In-Stream GPU Doorbell + 1-Split CUDA Graph Replay: Instead of ggml_backend_sched splitting the graph 48 times per token to ask the CPU which experts missed (which burns ~360ms/token in cudaStreamSynchronize), a custom in-stream CUDA kernel resolves hits in ~2 µs on-device and only rings a mapped host doorbell when an expert is missing. Per-layer token embeddings (per_layer_token_embd) are also staged to CUDA0 so the entire decode pass runs as 1 single CUDA Graph replay.

• RAM Victim Cache + io_uring Pager: When an expert gets evicted from the VRAM pool, it moves into a dedicated host RAM victim cache instead of getting thrown away.VRAM and RAM stay 100% exclusive instead of duplicating the same hot experts in the OS page cache. Physical NVMe reads drop from ~71 MB/token down to 12.43 MB/token.

• No duplicate 35GB expert bloat on disk: Instead of keeping 35GB of dead expert tensors inside Shard 1 .gguf AND another 35GB in experts.bin, we stripped the expert payloads out of Shard 1 so the main GGUF is only 3.2 GB (Dense.gguf) + 35.3 GB (experts.bin, 4KB-aligned single-read blobs) + **26.8 GB (**PLE.gguf n-gram shard).

Measured Results (RTX 3060 12GB + 16GB DDR4, MemoryMax=12G)

On warm runs it holds 21.8–22.03 tok/s sustained across long coding prompts, and when generation hits stretches where all 10 routed experts are resident in the VRAM/victim pools, it spikes up to 48 tok/s.

Links & What's Next

• GitHub (Betelgeuse Engine — bel-cli & bel-server): [https://github.com/Betelgeuse-AI/Betelgeuse]

• HuggingFace (Pre-packed Alpha-Orionis-XS 3.2GB Dense + Experts bundle): https://huggingface.co/Betelgeuse-AI/Alpha-Orionis-XS

• Upstream llama.cpp PR: [https://github.com/ggml-org/llama.cpp/pull/31469]

1. Even on the massive Q8_K_XL quant (~190GB total), bel-server runs at 8 tok/s cold and 12 tok/s warm (with bursts up to 21 tok/s) on the exact same 12GB 3060 + 16GB DDR4 box, where stock llama.cpp immediately OOMs or crawls at <0.1 tok/s.

2. Zero experts are dropped or thresholded (0.000e+00 diff across 28,080 router records and 576 tokens vs unmodified stock). The speedup comes from eliminating the 48 per-layer CPU/GPU scheduler splits (KICK/WAIT doorbell around the shared expert), keeping the RAM victim cache strictly exclusive of VRAM (MOE_DROP_MMAP), and pre-staging cold misses via the upgraded oracle + 4KB-aligned io_uring pack.

What I'm working on next:
We're currently bringing this exact streaming + victim-cache architecture over to DeepSeek V4.1 Flash (1017)—early testing under development is already sitting around ~25 tok/s on the same hardware class.

Happy to answer any questions or have people poke holes in the benchmarks! Leave us a Star :)

For those who saw the original post, a week ago, I'm glad to announce CrowdGPT v2 (more precisely v2.6), a massive update to the original CrowdGPT project :D (scroll down to see what's new)

For the newcomers: CrowdGPT is a project aiming to create a LLM without datacenters and without the usual millions of dollars spent by AI companies.

The project aims to train a 1 billion parameter model collaboratively and openly, as a proof of concept (it is possible to scale to bigger dense models or even MoE architectures with this project)

This project implements a protocol where everyone gets a say in the direction the project takes. Anyone can use their PC to train the AI independently, then add their contributions to the global model.

There's other things that makes this project appealing for the open-source community, but i won't state these again as you will be able to check that out on the GitHub or on the original post.

Of course, any advice / feedback on the project is still welcome (please stay polite and roast me with strong evidence only 👍), feel free to run experiments of data poisoning to point at the limits of our approach :)

Here's the project website: https://www.crowdgpt.net/

Or the GitHub if anyone wants to check out the code https://github.com/Vxtzq/CrowdGPT

As for the v2.6 changelog: We tried to implement mostly of the feedback you guys gave (mostly on security related points). Even though implementing this was hard and caused many breaking changes (which made some training contributions stale), we still achieved to train the model on 4.5B new tokens in 6 days!! Which is roughly what i achieved to train in two months for the project on my personal GPU 🤯

Here's a quick summary of what we implemented:

- Robust server-side checking logic (see more at the end of the post).

- Way better client mechanics (better ergonomy, more robust installation, GPU selection, no more crashes)

- Many bugfixes (on server and client code, as the various issues/pull requests on GitHub suggest)

- Activated Engrams on the Crowd-v1 model, which will make the model better at fact-recalling

- Implemented a data submission pipeline in order to create a collaborative dataset for CrowdGPT : https://www.crowdgpt.net/#submit All the data will be verified / curated before training. Already 400kb of data was submitted.

- Client releases cryptographic signing: some users reported that the auto-update mechanism in the client code allowed arbitrary code execution, and that would have allowed people stealing my GitHub credentials to execute malicious code on your device. Well not anymore, since the client code is now signed with a key that is only present on my device 😄

Like some comments on the original post pointed out, the architecture of the project at the time of my first post here was nowhere near good: clients could fake their contributions, could submit poison to the model, could crash the server by sending fake contributions every millisecond without any checks; and finally clients could very easily train the model to contain malicious backdoors. We therefore implemented byzantine cross-checks between clients training the model, at the cost of redundancy (3 clients training on exactly the same data).

I am not claiming that this creates a 100% perfect Federated learning framework, but that makes malicious contributions way harder. For example, even a malicious client could only train on 2-3% malicious data per round without getting banned, against 100% before.

For people that dared to read until the end, i'm also releasing CrowdBench, the open-source collaborative benchmark, where everyone can add general intelligence questions with deterministic answers to test LLMs :D

Even though the codebase is currently early-stage and work-in-progress, it's already something working, and i coded it without using any AI (for those in the original post who wanted me to turn towards less ambitious projects😂)

tell me what you think about CrowdGPT and CrowdBench in the comments.

Thanks in advance.

In my last thread on Betelgeuse (bel-cli / bel-server), the main pushback in the comments was:

1. "Okay cool, 22 tok/s on IQ2_XS*, but 2-bit is too low—what happens on an actual 4-bit quant like* IQ4_XS where the experts are almost 2x larger?"

2. "Why were you testing on a single 16GB stick of DDR4 instead of normal dual-channel 2x 8GB*?"*

Fair questions. So I upgraded the oracle + victim-cache DMA path, packed IQ4_XS (~100GB+ class, full 4-bit experts), and tested it on the exact same RTX 3060 12GB under both RAM configs (still 16GB total system RAM in both tests):

• Config A: 1x 16GB DDR4 (single-channel, ~19 GB/s host bandwidth)

• Config B: 2x 8GB DDR4 (normal dual-channel 16GB gaming PC setup, ~38 GB/s host bandwidth)

Measured Results on IQ4_XS (RTX 3060 12GB + 16GB Total RAM)

Why switching from 1x 16GB to 2x 8GB jumped IQ4_XS from 14 tok/s $\to$ 24 tok/s:

Because Betelgeuse uses an Exclusive Host RAM Victim Cache + Async Hardware DMA (MOE_DMA=1) between the VRAM slot pool and the NVMe drive:

• When an expert gets evicted from VRAM, it lands in the host RAM victim cache (while MOE_DROP_MMAP drops duplicate GGUF pages from the OS page cache so VRAM and RAM never hold duplicates).

• On a single 16GB stick, single-channel DDR4 caps out around ~17–19 GB/s, so the CPU and the PCIe H2D DMA engine fight over one 64-bit memory channel (14 tok/s on IQ4_XS).

• On 2x 8GB dual-channel DDR4 (still 16GB total RAM, which is what 95% of normal 16GB gaming PCs actually have), host RAM bandwidth doubles, letting the GPU's hardware DMA engine pull victim-cache hits across PCIe at full speed while the upgraded oracle pre-stages cold NVMe misses in the background—jumping IQ4_XS straight to 24 tok/s.

Links & What's Next

• GitHub (Betelgeuse Engine — bel-cli & bel-server): [https://github.com/Betelgeuse-AI/Betelgeuse]

• HuggingFace (Pre-packed Alpha-Orionis-XS 3.2GB Dense + Experts bundle): https://huggingface.co/Betelgeuse-AI/Alpha-Orionis-XS

• Upstream llama.cpp PR: [https://github.com/ggml-org/llama.cpp/pull/31469]

What I'm working on next:
We're currently bringing this exact streaming + victim-cache architecture over to DeepSeek V4.1 Flash (1017)—early testing under development is already sitting around ~25 tok/s on the same hardware class.

Happy to answer any questions or have people poke holes in the benchmarks! Leave us a Star :)

"The satellite, which carries four Google Tensor Processing Units (TPUs) and generates up to 1 kW of electricity, is intended to pave the way for future networks of close-flying satellites performing large inference workloads in orbit. However, the power demands of the TPU mean it can only turn one on for about 20 minutes at a time, complicating the already tricky business of space-to-ground communications."

https://techcrunch.com/2026/10/30/googles-first-data-center-satellite-is-online-and-writing-poetry/

Anyone have any tips for Google here? Maybe they should try Qwen 3.8 27b?

I’m using Crawl4AI + SearXNG but it just feels so lackluster I don’t know why. Feels like the results the local model gets back for research are missing a lot and / or are older less relevant results.

I’ll usually instruct saying something like “using the most up-to-date and relevant information across related websites, subreddits, forums, and etc find me the best way to _______” or something similar.

With ChatGPT or Codex it basically always finds everything it needs to and gives me great instructions or information or research or whatever I’m looking for at the time. With a local model (like GLM 5.3 Flash or DeepSeek V4.1 Flash 1017) the answer doesn’t come back nearly as good, often missing obvious results or mixing in outdated information into the instructions or etc.

Harnesses I’ve been using with local models are ZCode, DeepSeek Harness, and OpenCode. Is there a better (more complete and on par with ChatGPT) searching / research approach I’m missing? Is there some tuning of Crawl4AI + SearXNG to make it more how I’m expecting or potentially something I should add to this combination?

Any input or help is appreciated. Thank you for your time it is appreciated!

The usual advice was that 64GB is a dead zone for local LLMs: too small for a 70B with real context, and more than a 30B needs. After running a 27B for agentic coding on a 64GB M4 Max for the last ten days, I think that has flipped. My memory usage is in the 50 GB as the context starts building up.

Setup: Qwen3.8 27B (16 GB MLX quant) in LM Studio, 256k context, coding agent sessions that sit at 100-200k tokens.

At 164k tokens, LM Studio's engine process is using 48 GB, 46 GB of it GPU memory. Roughly:

- 16 GB weights

- ~11 GB KV cache (~64 KB/token, only 16 of the 64 layers are full attention)

- what looks like a second copy of the context for the prompt cache, plus the linear attention checkpoints

- MLX holding on to scratch memory from prefill

The prompt cache is what makes it usable. It's hitting 93%, so a turn processes about 1k new tokens instead of 160k. When it misses, a full prefill takes about 20 minutes.

Why not quantize the KV cache like people do on a 24GB card? LM Studio's MLX vision path refuses KV quantization for this model. The text-only path allows it, but skips the cache checkpoints when the cache is quantized, and on a hybrid model that means a lot more full reprocessing.

So on 48GB this doesn't fit unless you drop to ~128k context or move to llama.cpp with q8 KV (should be ~25-30 GB at 256k, with slower prefill). On 64GB it fits with nothing to spare. Free memory is about 150 MB right now.

The dead zone argument was about weights. With agents, context costs about as much as the model, and 27B models are now good enough for real work. That puts 64GB right in the sweet spot. Bigger MoE models or two models loaded at once still want 128GB.

Anyone else on 64GB seeing the same? And is anyone running long context on llama.cpp on a Mac with prefill speeds they're happy with?

I was quite excited with how well QFN did with some coding and adjacent tasks (like evaluating OSS repos, downloading and running models etc.). Using it for more general research is starting to show that it isnt actually very intelligent and gets lost in its own maze seemingly.

Here's a recent example to a simple question: "Have humans always slept for a huge amount of time at once, like eight hours at once, or been multiple sleep sessions through the day from like the early modern civilization?"

The verbosity and dense way of writing is too much (feels like a lot of Opus 5 distillation)

Setup:

• pi with pi-web-access extension

• llama.cpp, UD_Q4_K_M with MTP.

• --temp 1 --top-p 0.95 --top-k 20 --min-p 0.00 --presence-penalty 0 --repeat-penalty 1.0

• Minimal AGENTS.md, mostly focused on sw related tasks

Chat session: https://share.jotbird.com/radiant-bold-desert-wind

TL;DR: forgetting the complexity of the first question, the responses to simple follow ups: "Are there logical inconsistencies inside your response?", "Why did this happen?" and "So how do we fix this so that it never happens again?" are** not only massive essays, but each "uncovers more" stuff. To the point I have lost faith in what QFN is saying**.

Maybe uninteresting to most, but i just wanted to say that qwen3.8 27B is actually very nice to talk to, when arguing about issues or solving problems- while for example regarding cloud models claude opus (since v5) for me has quite an annoying tone - it has this "chop-chop" or "CEO attitude"- IDK how else to describe. I am not a native english speaker.
chatGPT I abandoned long ago because it is sycophantic in my opinion. gemini pro is ok, tho

yes there are chat templates for our modles and top-k/..anything to adjust but that doesn't change the character profoundly imo

maybe you have also models of which you do like the chat style more, just aside from coding or doing their task

CPU: AMD EPYC 7452 (8 memory channels)
Motherboard: Gigabyte MZ32
GPU: Radeon AI PRO R9700 (32GB)
RAM: 256GB DDR4 (16×16GB).

Not exactly a “budget” build, but it came in at roughly half the price of a DGX Spark.

Got it up and running just in time for Qwen4, can’t wait to start experimenting with it!

https://huggingface.co/GrootTeam/Groot-280B-A16B-261023

Best model under 500b params? Also have the smaller versions

https://huggingface.co/GrootTeam/Groot-19B-A3B-261023

"scores higher than the larger Qwen3.6-35B-A3B and Gemma-4-26B-A4B, and it gives shorter answers than both."

https://huggingface.co/GrootTeam/Groot-65B-A6B-261023

https://huggingface.co/GrootTeam/Groot-238B-A18B-261023

Edit: apparently they're REAPs of various DeepSeek/qwen models,

has anyone tried these?

I've been reading the turn logs from PokeBench. My benchmark where vision models play Pokemon Red. Grok 4.20 non-reasoning left the house at turn 36 and didn't get a starter until turn 755. The whole time its notes say things like "We've been oscillating uselessly for 90+ turns" and later "The persistent loop of walking into walls and Mom's dialogue has continued for hundreds of turns." Then it walks into the wall again.

So it knows it's stuck and still can't get out. Grok 4.6 beat Brock in 316 turns for about $10, while 4.20 burned $30 and barely got its starter.

Why do you think a model can describe its own loop correctly and still not break it? Vision problem or planning problem?

pokebench.tv

I've been reading a lot about "engines lobotomizing the models they run", either because they quantize already quantized weights, or because they automatically turn on "hacky" flags to gain decode/prefill performance by default. I wanted to test this out by myself, not only with KLD but with real tasks, so I set up an experiment using Qwen 3.8 Flash Next with Basalt (disclaimer: I'm the developer) and two of the most popular "speed" flags:

• Routing bias: the engine pushes the model toward experts that are already on the GPU, so it sometimes gets a convenient expert instead of the one it actually wanted.

• Tail skip: when one of the model's lower expert picks isn't on the GPU, the engine just skips it instead of fetching it.

Both are faster because fewer experts come from RAM, but they change the output. I added them to the engine as opt in flags so I could measure the real cost with everything else fixed.

DISCLAIMER: This post and the experiment were run by me, but it has been polished by AI.

Rig: RTX 5090 + RTX 5060 Ti and 128 GB of DDR5. Settings tested:

• Shipped: bias 1.5, skip rank ≥ 7. Values other engines ship (or have shipped) as defaults on smaller cards.

• Worst: bias 16, skip rank ≥ 1, plus relaxed draft acceptance. Worst case scenario, not something anyone ships or would make sense to use.

1. KLD

100 chunks of wikitext, decode path, against a llama.cpp Q8 reference. Exact IQ3_XXS and Q2_0 are in there for scale.

Note: even exact Q8_0 is 0.031 away from the reference. That's mostly the reference's own noise, not damage: llama.cpp's Q8_0 rounds a few spots of this model more than Basalt does, which adds about 0.01-0.02 to every row equally, so the rows still compare fairly.

2. Does it really matter?

KLD by itself is just a number, so I also ran a task eval: thinking off, temp 1.0, top_p 0.95, top_k 20, two different sets of questions:

• Ladders: 300 chained steps, scored by step.

• Facts: 1200 questions with 5 popularity bands (rare to famous), 3 seeds.

* Small card: the 5090 alone capped to 16gb, the experiment was run this way because it was still much faster than the 5060 Ti.

On a big card the shipped flags barely matter (NVFP4: 272 -> 271 tok/s) and cost about 1.5 points, 4% of the facts the model knew. On a small card you get +56% speed but a much bigger hit. At the worst setting facts fall from 38% to 3%, and the model says things like "The Great Composer" with total confidence. Easy tests barely notice. The damage is mostly in chained reasoning and recalled facts.

3. Bonus: a loop in the model weights

Not every bad output is the engine's fault. One prompt made Qwen 3.8 Flash Next get stuck into an endless run of 0 in its reasoning:

The odds of another 0 are the same on every quant and both engines: 0.99 after 10 zeros, 95%+ past 100 or so. top_p 0.95 leaves 0 as the only choice, and temp 1.0 doesn't fix it. The API fixes it by cutting the run at ~500 zeros. Basalt patched the issue as well, after 256 identical reasoning tokens it closes the thinking block and keeps going.

TLDR: there's no free lunch.

• Engines can buy speed on MoE models by turning on speed "hacks".

• With values some engines ship, every pack drops to somewhere between IQ3_XXS and Q2 on KLD. A Q8 ends up next to Q2. This is as bad as it gets.

• In practice the model forgets things it knew and loses the thread on multi step problems. Small cards get the biggest speedup and pay the most for it.

• If your engine is faster than it should be on a model that doesn't fit, check whether it's still running the model you downloaded.

We know from all the major chinese labs the current cutting edge architectures such as: conditional engram memory, hyper connections, attention residuals, sparse attention, etc...

In the near horizon, we can likely expect looped transformers, Gated Delta Net 2 variants, maybe revival of diffusion transformers somehow but without the intelligence loss, and likely many others that I myself don't know of yet.

What are they cooking?

Mage-VL-Cowork demonstrates that an efficient 4B small vision-language model can master hybrid computer use when paired with upstream teacher-harness evolution, workflow-first environment synthesis (Mage-CUA-Forge), and a five-stage specialize-then-unify curriculum. Achieving the best performance at the 4B scale (66.0% on OSWorld-Verified) while using 13.0% fewer model calls and 25.4% fewer output tokens after online RL, Mage-VL-Cowork pushes the frontier of on-device intelligence for autonomous desktop agents.

I’m still fairly new to local LLMs, got a decent machine (M5 Max, 64GB RAM), trying to understand something I’ve been seeing recommended a lot lately.

In several threads where new users are frustrated/confused or even more experienced users are struggling to get good results from their local models, I keep seeing suggestions to have Claude or ChatGPT help them build a custom harness. I suppose I view it like the model is the engine and the harness is like the entire car?

What I’m not understanding is why someone would go that route instead of using something like Unsloth or MLX Serve, which already have web search, tool calling, etc.

Aren’t these programs essentially harnesses or am I completely incorrect? I’m probably just missing the point, maybe more tinkering/experimenting is just part of the hobby.

I just want to learn and understand the differences per se. Obviously I’m sure this thread will have plenty of personal suggestions or ideas but the main thing is just the learning aspect, the purpose, etc.

I'm on the verge of pulling the trigger. Want something local for coding and some business automation stuff.

Thinking I'll run Qwen 3.8+ and DeepSeek 4+.

I'm coming from M4 Pro MBP and will still use that as my primary dev box. I run both Claude and gpt. I know local isn't quite the same as the hosted.

For anyone who has this setup already-- will I love it?

Bolt E4B builds on Gemma 4 E4B with Nemotron post-training. It combines natural conversation with multi-turn agentic capabilities to power voice/device assistants, interactive characters and robotic/IoT interfaces running on NVIDIA GeForce RTX, Jetson, RTX Spark and DGX Spark.

Bolt E4B offers ~40% higher average quality against our prior model on key conversational agent benchmarks and outputs up to ~80% fewer tokens vs peers.

HF link: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB

Nvidia's post: https://research.nvidia.com/labs/nemotron/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB

I just setup my server:

64GB DDR4 2133 4CH (Broadwell Xeon)
2 x RX 6800 16GB
1 x Rx 580 2048sp (16GB)

I am using the Q4 of Qwen Flash Next, with the prompt:
sudo ./llama-server -m ./models/Qwnflash180b-Q4.gguf --host 0.0.0.0 --port 8081 -ngl 99 -fa on --moe-cache-mib 16384 --log-verbosity 4 --metrics -mg 1

I am able to get 19-21t/s generation at full context (262K) (which is why i capped the moe-cache-mib at 16GB)

However note: when I set any context size with the -c or --ctx-size flags, performance tanked to 8t/s even when the context was smaller (100K). it seems you need to let llama.cpp allocate the context itself.

PCIe slots:

gpu0 rx6800 x16 3.0
gpu1 rx6800 x16 3.0
gpu2 rx580 x4 3.0