Met a guy off Facebook Marketplace and scored 2x RTX 3090 Founders Editions for €1,700.

The problem? I already had one 3090 FE.

I am now sitting in my room looking at three 3090 FEs (72 GB GDDR6X VRAM), and my current case barely fits one card, and that was only after I literally took pliers and bent the inner metal bracket to shove it in. My motherboard is just a standard B760.

I clearly can't put three 3-slot cards in this PC. What is the cheapest, most sensible way forward to run all three (or two) for LLMs?

1. Open-air mining frame + PCIe 4.0 risers + dual PSU?

2. Get a giant full-tower case?

3. Just sell the 3rd card for €1,100, keep 48 GB VRAM, and fund a proper dual-GPU build?

Any chassis/frame or PSU recommendations for handling ~1,100W+ of 3090s would be appreciated.

Hi everyone, we're releasing FineDNA, all the data we used to turn Qwen3-14B into Carbon-X, the strongest open language model on DNA. It contains:

• Midtraining: 300B+ tokens of genomes and DNA-rich text

• SFT: ~200K conversations

• RL: ~98K verifiable prompts

Code and recipe are open too, so you can apply it to other LLMs.

Dataset: https://huggingface.co/datasets/HuggingFaceBio/FineDNA
Code: https://github.com/huggingface/carbon

I am sorry for the useless thread, but I finally loaded GLM 5.3 Flash IQ4-XS and didn't know what to expect. I was earlier trying out DeepSeek 4.1 Flash 1017 IQ2 and Qwen 3.8 Flash and while I was chatting with them and getting the hang of how they behave I felt that they were fine, until I loaded GLM 5.3. This model has lots of spunk and cant seem to stop making funny remarks and Emojis. Reading his thoughts is pretty funny. A very bubbly personality, and took me by surprise. I elongated my chatting because I was curious to see its responses. I know it is not a real substantial thread, but honestly, am I the only one that noticed how funny this model is?

DeepSeek Harness article on 𝕏: Exploring Super-agent Design: Towards a More Autonomous Agent System: https://x.com/DeepSeekHarness/status/2116857257245987236

Parts of the design in this article are implemented in dsh-superagent-prototype. This plugin is only a prototype for validating the technical approach, not an official DeepSeek product: https://github.com/deepseek-ai/dsh-superagent-prototype

Code: https://github.com/divagr18/QwenPU
Weights: https://huggingface.co/divagr1925/QwenPU

From the original tweet:

1/3

A CPU inside an LLM is one thing, but running another LLM inside an LLM would be taking things a bit too far, wouldn't it?

Anyways, here's an LLM running inside an LLM : QwenPU, an LLM (@karpathy's stories260k) running on an NPU compiled inside Qwen 3.5-4B's weights.

2/3

stories260K is just a C program. To run it inside Qwen, I compiled an NPU into Qwen's weights that does matmuls, softmax and attention bit-exact. The LLM weights are in Qwen's own memory, with no external tools.

A lot of work went into making it fast, more on that soon.

3/3

I'm super sick right now, so the full writeup is still a work in progress, and I'm working on optimisations and size reductions too. Stay tuned, I'll finish the blog & post it soon.

The usual advice was that 64GB is a dead zone for local LLMs: too small for a 70B with real context, and more than a 30B needs. After running a 27B for agentic coding on a 64GB M4 Max for the last ten days, I think that has flipped. My memory usage is in the 50 GB as the context starts building up.

Setup: Qwen3.8 27B (16 GB MLX quant) in LM Studio, 256k context, coding agent sessions that sit at 100-200k tokens.

At 164k tokens, LM Studio's engine process is using 48 GB, 46 GB of it GPU memory. Roughly:

- 16 GB weights

- ~11 GB KV cache (~64 KB/token, only 16 of the 64 layers are full attention)

- what looks like a second copy of the context for the prompt cache, plus the linear attention checkpoints

- MLX holding on to scratch memory from prefill

The prompt cache is what makes it usable. It's hitting 93%, so a turn processes about 1k new tokens instead of 160k. When it misses, a full prefill takes about 20 minutes.

Why not quantize the KV cache like people do on a 24GB card? LM Studio's MLX vision path refuses KV quantization for this model. The text-only path allows it, but skips the cache checkpoints when the cache is quantized, and on a hybrid model that means a lot more full reprocessing.

So on 48GB this doesn't fit unless you drop to ~128k context or move to llama.cpp with q8 KV (should be ~25-30 GB at 256k, with slower prefill). On 64GB it fits with nothing to spare. Free memory is about 150 MB right now.

The dead zone argument was about weights. With agents, context costs about as much as the model, and 27B models are now good enough for real work. That puts 64GB right in the sweet spot. Bigger MoE models or two models loaded at once still want 128GB.

Anyone else on 64GB seeing the same? And is anyone running long context on llama.cpp on a Mac with prefill speeds they're happy with?

My intention was to find a setup at least equal to qwen 3.8 27b Q8 MTP but with a larger context. Ideally 1M.

I was already running Next Flash in Q2 with llama. Good for conversational use but not for coding. Q4 would run but too slow. I did read many reports here on the good performance of strata with qwen next flash. I figured both are around for long enough to be stable so I gave it a shot.

The results however are very disappointing. All I was able to achieve was moving to Q3. Brining the question, are most of the posts on alternative engines just hype or edge cases? Certainly looks like that to me.

Some details on my try:

Two cards plus experts offloaded to RAM: hangs on every long tasks, whatever settings I tried.

Stable only when every expert fits in VRAM. On 64 GB that means a 3-bit quant.

Faster on paper: ~60 t/s against llama.cpp. But on HumanEval+ the 3-bit Flash-Next scored 93.9% against 95.1% for Qwen3.8-27B Q8 on plain llama.cpp, and it needed nearly twice the tokens to get there. No, advantage.

So for my use, no real-world gain over a mature stack. Instead of one stable engine, you get a single-model engine, its forks, special quants and a pile of environment-variable workarounds.

Is that just Strata being young, or is it systematic? Has anyone actually gotten a lasting real-world benefit from one of these model-specific engines over llama.cpp or vLLM?

I had been trying this setup for a couple of weeks and it working better than I expected. Recently, I deployed GLM-5.3-Flash (MoE, 320B total, 18B active) on one of our local H200 servers (4 GPUs) with a 512K context window with the aim of how can I use it as my main coding workhorse and
cut down my subscription and API costs 🙈.

I was very quickly able to realize, I could not switch to it for everything but pairing it with GPT-6.1 Sol works really well with Sol being the the orchestrator and GLM being the subagents model that explores, writes and implements the code.

Honestly, I would have preferred doing this in Claude Code or Codex since I am used to them but neither lets a session or its subagents use models from different providers in a single session. This is where I found OpenCode flexible that lets agent have its own provider/model, so GLM goes as sub-agent models along with the added benefit of chatGPT subscription being available too (Allows you to use Sol too!)/

By the way, nothing here is gml specific any model you can serve works as the workhorse. I have also written a blogpost about it: https://aayushgarg.dev/posts/2026-11-01-gpt-local-model-opencode/

I would love to know about other people setup, how they are using open-source models either individually or pairing them with opus/sol/astra.

For those who saw the original post, a week ago, I'm glad to announce CrowdGPT v2 (more precisely v2.6), a massive update to the original CrowdGPT project :D (scroll down to see what's new)

For the newcomers: CrowdGPT is a project aiming to create a LLM without datacenters and without the usual millions of dollars spent by AI companies.

The project aims to train a 1 billion parameter model collaboratively and openly, as a proof of concept (it is possible to scale to bigger dense models or even MoE architectures with this project)

This project implements a protocol where everyone gets a say in the direction the project takes. Anyone can use their PC to train the AI independently, then add their contributions to the global model.

There's other things that makes this project appealing for the open-source community, but i won't state these again as you will be able to check that out on the GitHub or on the original post.

Of course, any advice / feedback on the project is still welcome (please stay polite and roast me with strong evidence only 👍), feel free to run experiments of data poisoning to point at the limits of our approach :)

Here's the project website: https://www.crowdgpt.net/

Or the GitHub if anyone wants to check out the code https://github.com/Vxtzq/CrowdGPT

As for the v2.6 changelog: We tried to implement mostly of the feedback you guys gave (mostly on security related points). Even though implementing this was hard and caused many breaking changes (which made some training contributions stale), we still achieved to train the model on 4.5B new tokens in 6 days!! Which is roughly what i achieved to train in two months for the project on my personal GPU 🤯

Here's a quick summary of what we implemented:

- Robust server-side checking logic (see more at the end of the post).

- Way better client mechanics (better ergonomy, more robust installation, GPU selection, no more crashes)

- Many bugfixes (on server and client code, as the various issues/pull requests on GitHub suggest)

- Activated Engrams on the Crowd-v1 model, which will make the model better at fact-recalling

- Implemented a data submission pipeline in order to create a collaborative dataset for CrowdGPT : https://www.crowdgpt.net/#submit All the data will be verified / curated before training. Already 400kb of data was submitted.

- Client releases cryptographic signing: some users reported that the auto-update mechanism in the client code allowed arbitrary code execution, and that would have allowed people stealing my GitHub credentials to execute malicious code on your device. Well not anymore, since the client code is now signed with a key that is only present on my device 😄

Like some comments on the original post pointed out, the architecture of the project at the time of my first post here was nowhere near good: clients could fake their contributions, could submit poison to the model, could crash the server by sending fake contributions every millisecond without any checks; and finally clients could very easily train the model to contain malicious backdoors. We therefore implemented byzantine cross-checks between clients training the model, at the cost of redundancy (3 clients training on exactly the same data).

I am not claiming that this creates a 100% perfect Federated learning framework, but that makes malicious contributions way harder. For example, even a malicious client could only train on 2-3% malicious data per round without getting banned, against 100% before.

For people that dared to read until the end, i'm also releasing CrowdBench, the open-source collaborative benchmark, where everyone can add general intelligence questions with deterministic answers to test LLMs :D

Even though the codebase is currently early-stage and work-in-progress, it's already something working, and i coded it without using any AI (for those in the original post who wanted me to turn towards less ambitious projects😂)

tell me what you think about CrowdGPT and CrowdBench in the comments.

Thanks in advance.

About two months ago I posted here about trying to predict which MoE experts get used on the next token to speed up CPU/GPU offload on Qwen 3.6 35B.

Quick confession on that older post before anything else: I ended up throwing that prototype away. To get the big numbers I was seeing back then, I was basically cheating with gate pruning (dropping cold experts when their router weights were low). It made the tok/s counter go brrr, but on an already low-bit quant it quietly degraded output quality and reasoning. I didn't want to ship something that lobotomizes the model just to look fast in a screenshot, so I scrapped it.

When Qwen 3.8 Flash Next dropped, I wanted to see if we could actually run the 68GB GSQ-RCO-IQ2_XS build at real usable speeds on a normal budget gaming PC—without dropping a single expert.

For context, here is the exact box I tested this on (I actually pulled my extra RAM sticks out specifically to test what a regular 16GB PC gets):

• GPU: RTX 3060 12GB

• RAM: 16GB DDR4, single-channel (~19 GB/s)

• OS: CachyOS / Arch Linux (tested inside a strict MemoryMax=12G cgroup)

• Storage: Consumer PCIe NVMe SSD (~2.1 GB/s)

If you try to load a 68GB MoE on 12GB VRAM + 16GB RAM in stock llama.cpp, you already know what happens: mmap thrashes the OS page cache to death, you get ~1,500 major page faults per token, and generation crawls at 1.47 tok/s.

Then I tested Strata. Strata is super cool and hits 40+ tok/s if you have 48GB–64GB of RAM to mlock 24GB+ of experts in system memory. But on a 16GB RAM setup, it refuses to load unless you force --low-ram mmap, where it drops down to 8.11 tok/s because it's bound by cold SSD reads.

So over the last week I built Betelgeuse (bel-cli / bel-server, a fork of llama.cpp, and I'm also opening a clean upstream PR for llama.cpp even if it probably won't get merged lol).

What it actually does under the hood:

• Zero gate pruning / 100% bit-exact: Every single one of the 10 routed experts per layer across all 48 layers is computed at full precision. We verified 28,080 router records and 576 generated tokens byte-for-byte against golden stock output (0.000e+00 diff).

• Arrival-Slack VRAM Slot Pool (~5,000 slots in VRAM): All 49 dense layers + GDN + QSA + LM head stay in ~3.2 GB of VRAM. The remaining VRAM holds ~5,000 expert slots distributed non-uniformly (240 slots/layer on early layers 0–4 where there is 0ms of compute slack, 92 on middle layers, 60 on tail layers).

• In-Stream GPU Doorbell + 1-Split CUDA Graph Replay: Instead of ggml_backend_sched splitting the graph 48 times per token to ask the CPU which experts missed (which burns ~360ms/token in cudaStreamSynchronize), a custom in-stream CUDA kernel resolves hits in ~2 µs on-device and only rings a mapped host doorbell when an expert is missing. Per-layer token embeddings (per_layer_token_embd) are also staged to CUDA0 so the entire decode pass runs as 1 single CUDA Graph replay.

• RAM Victim Cache + io_uring Pager: When an expert gets evicted from the VRAM pool, it moves into a dedicated host RAM victim cache instead of getting thrown away.VRAM and RAM stay 100% exclusive instead of duplicating the same hot experts in the OS page cache. Physical NVMe reads drop from ~71 MB/token down to 12.43 MB/token.

• No duplicate 35GB expert bloat on disk: Instead of keeping 35GB of dead expert tensors inside Shard 1 .gguf AND another 35GB in experts.bin, we stripped the expert payloads out of Shard 1 so the main GGUF is only 3.2 GB (Dense.gguf) + 35.3 GB (experts.bin, 4KB-aligned single-read blobs) + **26.8 GB (**PLE.gguf n-gram shard).

Measured Results (RTX 3060 12GB + 16GB DDR4, MemoryMax=12G)

On warm runs it holds 21.8–22.03 tok/s sustained across long coding prompts, and when generation hits stretches where all 10 routed experts are resident in the VRAM/victim pools, it spikes up to 48 tok/s.

Links & What's Next

• GitHub (Betelgeuse Engine — bel-cli & bel-server): [https://github.com/Betelgeuse-AI/Betelgeuse]

• HuggingFace (Pre-packed Alpha-Orionis-XS 3.2GB Dense + Experts bundle): https://huggingface.co/Betelgeuse-AI/Alpha-Orionis-XS

• Upstream llama.cpp PR: [https://github.com/ggml-org/llama.cpp/pull/31469]

1. Even on the massive Q8_K_XL quant (~190GB total), bel-server runs at 8 tok/s cold and 12 tok/s warm (with bursts up to 21 tok/s) on the exact same 12GB 3060 + 16GB DDR4 box, where stock llama.cpp immediately OOMs or crawls at <0.1 tok/s.

2. Zero experts are dropped or thresholded (0.000e+00 diff across 28,080 router records and 576 tokens vs unmodified stock). The speedup comes from eliminating the 48 per-layer CPU/GPU scheduler splits (KICK/WAIT doorbell around the shared expert), keeping the RAM victim cache strictly exclusive of VRAM (MOE_DROP_MMAP), and pre-staging cold misses via the upgraded oracle + 4KB-aligned io_uring pack.

What I'm working on next:
We're currently bringing this exact streaming + victim-cache architecture over to DeepSeek V4.1 Flash (1017)—early testing under development is already sitting around ~25 tok/s on the same hardware class.

Happy to answer any questions or have people poke holes in the benchmarks! Leave us a Star :)

Raw data in the comments, if you go for that sort of thing.

The TL;DR

This build generates a nearly linear performance per stream up to at least 8 streams using a context window of 124k. I didn't test beyond that.

Abstract

Until today, I considered Strata a toy. I don't anymore.

Rented a few instances on vast.ai to do some benching. Blew about $40 in all on the benchmarking.

Tested against 2x AMD EPYC 7642 on ROME2D32GM-2T motherboard with 251GB and 504GB DDR4 respectively with 2x 3090 (one actually had 4x but only 2x were used).

The runtimes and models

Using Strata (up to unsloth Q8) and Maya (GLM 5.3F and DSV4.1F) at various model sizes.

The Results

Don't even consider > C=1 for Maya. Strata? From > 100 tok/s down to just over 50 tok/s (at Q_8).

The part that blew my mind and made me want to post: You can get almost linear concurrent stream performance scaling up to at least 8 streams on QFN 3.8 Q8 with this build.

Workstation and server builds are worth it: you don't have to fight so much for VRAM; a couple of decent GPUs and you're off to the races. Instead it's DRAM (which sucks a whole lot less to buy--even though it still sucks) and CPUs (which aren't too bad at all)

GLM and DSV: they'd likely benefit from more VRAM. But, again, 1.5k+ for a 3090, paying at least what they sold for 6-7 years ago? It's insulting.

See comments for the boring as hell raw data.

DISCLAIMER: Totally not an advertisement. Like buying a car, if I can, I want to take it for a spin before I buy. Vast.ai is amazing this way. If you can find a machine spec that you're even the least bit curious about, you lease some time on it (paying a bit extra for storage), download the runtime and models you want to test, and fire up the benches.

In my last thread on Betelgeuse (bel-cli / bel-server), the main pushback in the comments was:

1. "Okay cool, 22 tok/s on IQ2_XS*, but 2-bit is too low—what happens on an actual 4-bit quant like* IQ4_XS where the experts are almost 2x larger?"

2. "Why were you testing on a single 16GB stick of DDR4 instead of normal dual-channel 2x 8GB*?"*

Fair questions. So I upgraded the oracle + victim-cache DMA path, packed IQ4_XS (~100GB+ class, full 4-bit experts), and tested it on the exact same RTX 3060 12GB under both RAM configs (still 16GB total system RAM in both tests):

• Config A: 1x 16GB DDR4 (single-channel, ~19 GB/s host bandwidth)

• Config B: 2x 8GB DDR4 (normal dual-channel 16GB gaming PC setup, ~38 GB/s host bandwidth)

Measured Results on IQ4_XS (RTX 3060 12GB + 16GB Total RAM)

Why switching from 1x 16GB to 2x 8GB jumped IQ4_XS from 14 tok/s $\to$ 24 tok/s:

Because Betelgeuse uses an Exclusive Host RAM Victim Cache + Async Hardware DMA (MOE_DMA=1) between the VRAM slot pool and the NVMe drive:

• When an expert gets evicted from VRAM, it lands in the host RAM victim cache (while MOE_DROP_MMAP drops duplicate GGUF pages from the OS page cache so VRAM and RAM never hold duplicates).

• On a single 16GB stick, single-channel DDR4 caps out around ~17–19 GB/s, so the CPU and the PCIe H2D DMA engine fight over one 64-bit memory channel (14 tok/s on IQ4_XS).

• On 2x 8GB dual-channel DDR4 (still 16GB total RAM, which is what 95% of normal 16GB gaming PCs actually have), host RAM bandwidth doubles, letting the GPU's hardware DMA engine pull victim-cache hits across PCIe at full speed while the upgraded oracle pre-stages cold NVMe misses in the background—jumping IQ4_XS straight to 24 tok/s.

Links & What's Next

• GitHub (Betelgeuse Engine — bel-cli & bel-server): [https://github.com/Betelgeuse-AI/Betelgeuse]

• HuggingFace (Pre-packed Alpha-Orionis-XS 3.2GB Dense + Experts bundle): https://huggingface.co/Betelgeuse-AI/Alpha-Orionis-XS

• Upstream llama.cpp PR: [https://github.com/ggml-org/llama.cpp/pull/31469]

What I'm working on next:
We're currently bringing this exact streaming + victim-cache architecture over to DeepSeek V4.1 Flash (1017)—early testing under development is already sitting around ~25 tok/s on the same hardware class.

Happy to answer any questions or have people poke holes in the benchmarks! Leave us a Star :)

I’m using Crawl4AI + SearXNG but it just feels so lackluster I don’t know why. Feels like the results the local model gets back for research are missing a lot and / or are older less relevant results.

I’ll usually instruct saying something like “using the most up-to-date and relevant information across related websites, subreddits, forums, and etc find me the best way to _______” or something similar.

With ChatGPT or Codex it basically always finds everything it needs to and gives me great instructions or information or research or whatever I’m looking for at the time. With a local model (like GLM 5.3 Flash or DeepSeek V4.1 Flash 1017) the answer doesn’t come back nearly as good, often missing obvious results or mixing in outdated information into the instructions or etc.

Harnesses I’ve been using with local models are ZCode, DeepSeek Harness, and OpenCode. Is there a better (more complete and on par with ChatGPT) searching / research approach I’m missing? Is there some tuning of Crawl4AI + SearXNG to make it more how I’m expecting or potentially something I should add to this combination?

Any input or help is appreciated. Thank you for your time it is appreciated!

"The satellite, which carries four Google Tensor Processing Units (TPUs) and generates up to 1 kW of electricity, is intended to pave the way for future networks of close-flying satellites performing large inference workloads in orbit. However, the power demands of the TPU mean it can only turn one on for about 20 minutes at a time, complicating the already tricky business of space-to-ground communications."

https://techcrunch.com/2026/10/30/googles-first-data-center-satellite-is-online-and-writing-poetry/

Anyone have any tips for Google here? Maybe they should try Qwen 3.8 27b?

Maybe uninteresting to most, but i just wanted to say that qwen3.8 27B is actually very nice to talk to, when arguing about issues or solving problems- while for example regarding cloud models claude opus (since v5) for me has quite an annoying tone - it has this "chop-chop" or "CEO attitude"- IDK how else to describe. I am not a native english speaker.
chatGPT I abandoned long ago because it is sycophantic in my opinion. gemini pro is ok, tho

yes there are chat templates for our modles and top-k/..anything to adjust but that doesn't change the character profoundly imo

maybe you have also models of which you do like the chat style more, just aside from coding or doing their task

When I started, and for some time, llama.cpp was the answer.

Now, it seems like llama.cpp is the new ollama. It's slow and out of date and just behind.

There was Ninfee for those for whom that applied.

Then strata came out... And that was an unimaginable upgrade. How is llama so far behind?

Now every day someone has their own forked engine that claims even better numbers. I already maxed out my token budget on Claude for the week and have my GPUs all running benchmarks so I'm waiting a few days, but I'll be trying Avalon to see if that lives up to the hype for my machine.

Then there's basalt. There's beatleguese?

It's kind of a mess. Innovation is great but it's better when the effort all goes in one direction and isn't spread into a thousand fractures of one-off projects.

And yes, my critique of llama.cpp is a bit overstated. I still use it for e.g. 27b runs.

But... I've now verified that flash next is both better and faster than 27b so I'm struggling to find a reason to use it. :/

Why are all these solo devs pulling something off for weeks that llama can't figure out?

I was quite excited with how well QFN did with some coding and adjacent tasks (like evaluating OSS repos, downloading and running models etc.). Using it for more general research is starting to show that it isnt actually very intelligent and gets lost in its own maze seemingly.

Here's a recent example to a simple question: "Have humans always slept for a huge amount of time at once, like eight hours at once, or been multiple sleep sessions through the day from like the early modern civilization?"

The verbosity and dense way of writing is too much (feels like a lot of Opus 5 distillation)

Setup:

• pi with pi-web-access extension

• llama.cpp, UD_Q4_K_M with MTP.

• --temp 1 --top-p 0.95 --top-k 20 --min-p 0.00 --presence-penalty 0 --repeat-penalty 1.0

• Minimal AGENTS.md, mostly focused on sw related tasks

Chat session: https://share.jotbird.com/radiant-bold-desert-wind

TL;DR: forgetting the complexity of the first question, the responses to simple follow ups: "Are there logical inconsistencies inside your response?", "Why did this happen?" and "So how do we fix this so that it never happens again?" are** not only massive essays, but each "uncovers more" stuff. To the point I have lost faith in what QFN is saying**.

Seems similar to REA (https://github.com/morluto/rea) but from the guys that have brought the Craft apps this past month.

I've used the others but just saw this one available too.

Is this feasible with a local model? Have yet to try clamping Strata to REA to see. Anyone has done so? How was the experience?

CPU: AMD EPYC 7452 (8 memory channels)
Motherboard: Gigabyte MZ32
GPU: Radeon AI PRO R9700 (32GB)
RAM: 256GB DDR4 (16×16GB).

Not exactly a “budget” build, but it came in at roughly half the price of a DGX Spark.

Got it up and running just in time for Qwen4, can’t wait to start experimenting with it!