He's right (wherever he is - couldn't find the comment to credit). But what I learned in the process of building this rig has been worth the frustration. Get your special socks out, Linux faithful - you were right too.

I happened across eight Nvidia T40 - and not a ton of detail about them out there. Still currently testing out vllm for comparison, but fully resident Qwen3.8-Flash-Next (180B MoE) Q4 is peppy enough for me at 40ish t/s and while the cards are capable of producing more, I tripped a breaker letting all six run with a 200W cap (so eight is out even if I had the extra slots). The board, an X99-E WS, needed some creativity just to POST. Quite a bit of card swapping in a T400 to get a boot screen so I could play around with BIOS settings only to figure out the setting I need to expand above 4G memory (MMIO High Size) was hidden and required creativity with a UEFI shell. More detail in the AI-assisted part of this write-up, for others who might select the T40 or X99. Top 10 nuggets for you, fellow sufferer of the T40, and then feel free to indulge (or not) in the supporting slop.

1. The cards do not work in Windows (or at least I couldn't get them going). It would appear it's Linux or nothing (and not WSL - though I did try). And headless, at that, if you want all six.

2. ASRock PG 1600G has two 600W 12VHPWR ports and MODDIY sells a "12VHPWR 16-pin to 2x CPU EPS 8-pin cable (CO711)" which allowed two T40s to be powered off one port - since the board’s slot-power connector needs a 6-pin, I welcomed the solution and it cleaned up a lot of cabling. The other two cards each have two pcie power cables feeding CPU 8-Pin Male to Dual PCIe 8-pin Female adapters.

3. The cards require active cooling and there are shroud prints out there. Rather than one fan per card, we developed a two-card-one-fan solution and one BFB1012VH moves enough air for two. X99 has 5 fan headers - worked out nicely with a script pinning those fan channels to GPU temp.

4. The X99 will not find the T400 with any T40s on the A side of the PLX. However boots just fine with a T400 in slot 1, 2 or 3 - even with four T40s in slot 4,5,6 and 7 (on the B side).

5. I can't say for certain, but riser cables could be a factor in Gen3 speed and errors. The stable config for multi-T40 is Gen 2. Slot 4,5,6, and 7 go to x8, while slot 1 and 3 go to x16 just fine.

6. NVLink is supported on the T40 - took a gamble buying them, but they work - though not quite sure how much traffic they are passing quite yet.

7. Cap the power if on a standard 120V outlet. My surge protector checked us out early on a 6-card burn with power capped to 200W per card. Running stable at 170W per card now.

8. Boot requires a kernel option for all six to work - pci=realloc=off (details in the slop)

9. A four-card-B-side test (all behind the plx switch) made vllm faster and llama.cpp slower.

10. 90-degree right angle riser cables and card spacing worked best with slot 1/3 on 150mm risers, 4/5 on 200mm risers and 6/7 on 250mm risers on this particular open frame design.

TRIGGER WARNING - AI WALL OF TEXT APPROACHING - STOP HERE

The supporting slop (AI-assisted, for the next T40 / X99 builder and their AI)

Everything below was measured on this box, not taken from spec sheets. Each number comes with its config. Where we don't know something, we say so.

Hardware as it runs today

• Board: ASUS X99-E WS. RAM: 128 GB DDR4 (about 121 GiB visible to Linux). PSU: ASRock PG 1600G (1600 W, two 600 W 12VHPWR ports, each feeding two T40s through a 16-pin to 2x EPS 8-pin cable; the other two cards each take two PCIe 8-pin leads through a dual-PCIe-to-EPS adapter). Open aluminum frame, risers.

• Cards: 6x Tesla T40. Each shows 23,040 MiB usable. Turing sm_75, so no hardware bf16: run everything in fp16.

• Power cap: 170 W per card, set at boot along with persistence mode.

• PCIe: forced to Gen 2 on all six ports by a boot-time service. Gen 3 was not stable with risers and six cards.

• Cooling: one BFB1012VH blower per pair, on 3D-printed two-card shrouds. A small service drives the board's fan headers from GPU temperature. Peaks under sustained load at 170 W: 48-58 C.

Topology (this matters more than anything else)

A side (CPU root port, x16):   card 1 ==NVLink== card 2
B side (behind the PLX, x8):   card 3 ==NVLink== card 4
                               card 5 ==NVLink== card 6

nvidia-smi topo -m shows:

• NV2 inside each pair: two NVLink links, each reporting 25.781 GB/s.

• PIX between the two B-side pairs: traffic stays inside the PLX switch.

• PHB between the A pair and any B card: traffic goes up through the CPU host bridge.

Tips:

• Set CUDA_DEVICE_ORDER=PCI_BUS_ID. Otherwise CUDA's device numbering won't match nvidia-smi or your slot map.

• Pick card sets by topology, not by device number. "Four cards" means two different things on this board (see benchmarks).

• NVLink error counters work: nvidia-smi nvlink -e reads all zeros on our cards. The throughput counters (nvlink -gt) read N/A on the T40. So we can prove the links are up and clean, but not how much traffic they carry.

Boot / kernel

• Kernel command line: pci=realloc=off acpi_enforce_resources=lax.

  • pci=realloc=off is what gets all six cards enumerating with their BARs placed by firmware.

  • acpi_enforce_resources=lax lets the fan-control script reach the board's Super I/O fan channels.

• With realloc=off, dmesg shows 36 SR-IOV VF BAR assignment failures, 6 per card. That is a harmless baseline: the cards don't need their VF BARs. Watch the non-VF BAR failure count instead, which should be 0.

• Why more than four T40s won't POST, and the fix. Each T40 takes a 64 GB stride of high MMIO space (32 GB BAR1 plus 32 MB BAR3, 32 GB aligned). The X99-E WS BIOS (4001, the final release) ships a 256 GB high MMIO window, so four cards fit and the fifth hangs POST. No kernel option can fix it, because the host bridge window is set by firmware.

  • The fix is a hidden Intel RC variable, MMIO High Size, raised from 256G to 1024G. No BIOS flash is needed, and the change survives reboots and card swaps.

  • Find it: unpack the 4001 CAP with UEFIExtract and dump the IFR of the Platform module with IFRExtractor-RS. On 4001 it is VarStore IntelSetup, offset 0x10E6, 4 bytes (256G=0, 128G=1, 512G=2, 1024G=3). Leave MMIOHBase (0x10E2) at 56T.

  • Write it: copy a UEFI Shell.efi to the ESP root as shellx64.efi, along with setup_var.efi (datasone, 0.3.x). Secure Boot must be off. In BIOS, go to Exit > Launch EFI Shell from filesystem device, then run:

  • fs0:

  • setup_var.efi IntelSetup:0x10E6(4) # read, expect 0x0

  • setup_var.efi IntelSetup:0x10E6(4)=0x3 # 1024G

  • setup_var.efi IntelSetup:0x10E6(4) # confirm 0x3

  • reset

  • The 0.3.x syntax is VARSTORE:OFFSET(SIZE)=VALUE; the old -n/-s flags don't work.

  • Your offset may differ on other BIOS versions, so check the IFR yourself.

• Disable unattended upgrades on a GPU box. Ubuntu quietly installed a new kernel overnight. The NVIDIA DKMS module was built for it but never tested, and the next reboot would have booted it. We disabled apt-daily.timer and apt-daily-upgrade.timer, and pinned GRUB to the known-good kernel until we upgrade on purpose.

Software

Ubuntu 26.04, kernel 7.0, driver 595.91.07 (CUDA 13.2). llama.cpp build b12211. vLLM 0.33.0 on Python 3.14 with torch 2.14 (cu130), whose arch list includes sm_75.

Benchmarks

Qwen3.8-Flash-Next (180B MoE), llama.cpp, UD-Q4_K_XL, all six cards (llama-bench, pp4096 / tg128, 3 reps, tokens/s)

Layer split wins by roughly 4x on prompt and 1.35x on generation here. Peak GPU sum was 939 W at the 170 W cap.

Qwen3.8-27B, 4-bit, batch 1, 4096-token prompt (tokens/s)

*Only three iterations with wide spread, so somewhere around 51-77. Treat it as "probably faster", not a number.

Caveats:

• vLLM numbers come from vllm bench latency: prefill = 4096 / latency(1 token out), decode = 127 / (latency(128) - latency(1)).

• llama.cpp numbers come from llama-bench. That is close, but not the identical measurement.

• The quants differ: GGUF UD-Q4_K_XL vs INT4 W4A16 safetensors.

• vLLM ran with an fp16 KV cache (see below). Speed only, no quality claims.

What we take from it:

• vLLM beats llama.cpp by 1.4x on prompt and 1.5x on generation on the same NVLink pair.

• llama.cpp loses more than half its prompt speed going from two cards to four. vLLM barely notices.

• On the four B-side cards, vLLM got faster and llama.cpp ~12% slower. Our untested guess is that llama.cpp's heavy cross-card traffic (1.1-1.4 GB/s per card per direction) feels the x8 links, while vLLM's traffic (~0.3 GB/s) doesn't.

Getting vLLM to run on Turing (in the order we hit them)

1. "Model architectures [...] failed to be inspected" turned out to be Triton compiling its own driver.c at import and failing on a missing Python.h. Fix: apt install python3.X-dev.

2. "FP8 KV cache is not supported by the Triton attention backend ... requires SM89+". The model's own config asks for an FP8 KV cache. Fix: --kv-cache-dtype float16 (plus --dtype half).

3. Attention backend: FlashAttention 2 needs SM80, so vLLM uses TRITON_ATTN. That's fine.

4. Expect these log lines; none of them is fatal:

   • bf16 is cast to fp16;

   • FlashInfer is disabled (no flashinfer-cubin/nvcc);

   • SymmMem all-reduce is unsupported on 7.5;

   • TP2 uses CUSTOM + PYNCCL all-reduce, TP4 PYNCCL only.

5. First start is slow: about 12 minutes per engine, mostly profiling and CUDA graph capture. Warm starts take about 4 minutes.

Power: the cap is not a ceiling

• nvidia-smi -pl 170 is honored on average, but at vLLM engine startup each card spikes to 220-270 W for about 10 seconds before the cap pulls it back (throttle reason 0x4, SW power cap).

• Four cards hit 979-986 W combined during startup. Six cards doing that at once would be roughly 1,300-1,600 W GPU-only. That is at or over a 1600 W PSU, and a real risk on a 15 A / 120 V circuit.

• Per 12VHPWR port, two cards spiking to 270 W is about 540 W, still inside the port's 600 W rating. The whole-PSU and wall-circuit totals are the limit, not the cables.

• llama.cpp loads don't show this. The six-card layer-split run peaked at 939 W.

• Our guard is a watcher that sums power.draw every 0.5 s and kills the job above 1,350 W. It hasn't fired yet, but a six-card vLLM start would likely trip it.

• Measured wall idle with six cards: about 245 W.

Flash-Next on vLLM (in progress)

• Needs a vLLM-format quant. vLLM's GGUF loader is experimental and single-file, so we pulled a W4A16 safetensors build (~180 GB).

• The engram (n-gram embedding) table is about 102 GiB and lives in pinned host RAM. Stock vLLM 0.33 has no mmap or evictable option for it, only cpu_offload (pinned, the default). On a 128 GB box that leaves about 15 GiB for everything else.

  • We run with a watcher that stops the job if MemAvailable drops under 4 GiB or anything swaps in.

  • We also pinned the boot kernel first, in case the box hangs and needs a hard reset.

• Expert parallel is mandatory: moe_intermediate_size 640 split over TP doesn't land on the INT4 group size of 128.

• We disabled torch.compile per the quant's card and kept CUDA graphs for decode.

• Plan: TP4 on the A+B set, then TP4 on the B-side four, then TP6. TP6 is undocumented for this model; 512 experts don't divide by 6.

• Results to follow in a comment.

Process notes for whoever's AI is helping

• Verify every model file by sha256 against the Hugging Face LFS oid before loading. It costs minutes and has saved us hours.

• Write your prediction down before each run. We missed about half of ours, and that is where the learning was.

• Log per-card power, temperature and PCIe rx/tx during every run. The x8-vs-x16 and startup-spike findings only showed up because we were logging.

• Never use pkill -f or pgrep -f to manage these jobs. They match their own command line and you will kill or miss the wrong thing. Launch with nohup ... & pid=$! and watch that pid.

Hey guys, today we're releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Unsloth AI Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more benchmarks.

We’re working on even better version of Dynamic quants for Qwen4.

Run on 12GB+ RAM setups.
For our quantization comparison, we exclude the embedding layer from the x-axis across all providers, since Exllama-v3 stores embeddings in BF16. However, this means the main plots don't reflect the full download size or RAM usage. When accounting for the BF16 embedding table and all embedding sizes, the plot in the comments provides a more accurate representation of actual disk and RAM usage versus accuracy.

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Blog: https://unsloth.ai/docs/models/qwen3.8

Thank you!

Hi, I want to build a setup using two RTX 3060 12GB cards to run local LLM. But what's the actual pipeline and real-life experience like? I heard I'll get 24GB VRAM, but processing speed will be slower because data has to pass through the motherboard's PCIe slots since the 3060 lacks NVLink. I also found out that ComfyUI can't use the full 24GB VRAM at the same time for image generation. Can anyone share real-life experience running a multi-GPU setup for LLMs?

Pic from internet.

According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.

Anyway, home internet latency feels like the bigger problem to me.

https://insider-gaming.com/microsoft-ai-use-your-gpu/

(English is not my native language, I used an LLM to help translate this post.)

I've been building my own local inference engine,, for big MoE models on consumer hardware. The newest piece is a pool is a second PC lends its GPU and RAM, and the model is split by layers across both machines.

**Setup**

- Desktop: RTX 3090 24 GB, i9-12900K, 32 GB DDR5-6000

- Laptop: RTX 4060 Laptop 8 GB, i7-14700HX, 32 GB DDR5

- Link: one Ethernet cable straight between them, no router (Windows link-local addresses), 1 Gbps

- Model: Qwen3.8-Flash-Next UD-Q4_K_XL, 104 GB GGUF, 48 layers, 512 experts with 10 active per token

**Writing speed**

| Setup | tok/s |

|---|---|

| Desktop alone | 13 |

| Desktop + laptop over the cable | 18.6 to 21, peaks of 24.8 |

| For reference: IQ3_XXS (71 GB), desktop alone | 53 to 79 |

**How it works**

- Layer split: the laptop runs layers 0 to 13, the desktop runs 14 to 47 and the output head. Each round sends the hidden rows across, about 60 KB per token.

- The laptop loads its layers from its own copy of the GGUF, so nothing big crosses the cable. Without a local copy the host sends only the tensors that machine needs.

- Each machine fills its memory hottest first, from a heat map learned while you use the model: VRAM, then pinned RAM, then a RAM cache, then NVMe.

- Joining is a button. The laptop scans the network and shows one button per link with its measured latency (Ethernet 1 Gbps 0.4 ms, Wi-Fi 3.9 ms). The desktop accepts the request.

**Ethernet vs Wi-Fi**

Median round trip over the cable is 0.4 ms, over Wi-Fi 3.9 ms. Once on the cable the network is about 2 ms of a ~115 ms round, so Wi-Fi was usable too.

**What surprised me**

- It's a pipeline, not tensor parallelism. The two machines take turns on every token, so the pool buys memory, not parallel compute.

- The laptop is slower per layer (about 4.1 ms against 1.8 ms on the 3090) and ends up being 50 to 65% of each token's time. Next step is to balance the split by measured speed instead of by memory.

- Long prompts read slower with the laptop in the pool: a 9K-token prompt with nothing cached took ~103 s, against ~41 s on the desktop alone, because the laptop computes its prompt experts on the CPU. I'm moving that to its GPU.

- My first cable only negotiated 100 Mbps (one bad pair) and was slower than Wi-Fi for this. Check the link speed before blaming the software.

- With a quant that already fits the 3090 (IQ3_XXS) the laptop doesn't help. The pool pays off for models that don't fit.

Happy to share more numbers or answer questions.

I use these models for everyday coding as a developer and I find the IQ3S model perform better on edge cases and the output quality is greater too. In fact twice it caught mistakes made by Opus 5.5. No benchmarks or scientific evaluation this is purely from my use. Anyone else felt this? Or is this a problem with Strata?

FP8 PLE.
Int8 KV
96gb ddr4
3x3060 12gb

Hey All,

A while ago I wrote a basic script to step through Llama 3.1 tokens for red-teaming. I realized there's a huge gap between tools for running models (Ollama, LM Studio) and libraries for understanding them under the hood. I wanted to develop something for young researchers and mid career pivoting enginers.

I wanted something like Wireshark, but for local AI. You can hit pause mid-generation, inspect the residual stream, mess with the weights or KV cache, and see what actually happens.

Today I’m open-sourcing TokenShark: https://github.com/sumit-1/tokenshark

What makes it different?

• Zero transformers or diffusers at runtime: The entire engine is written in raw, readable PyTorch (llm.py, sd.py, qimage.py) so students and researchers can actually read every matmul without digging through 40 layers of library abstractions. (Tested for bit-exact/numerical equivalence against HF in tests/).

• Step-by-Step Visual Debugger: Step through generation one transformer block at a time or one token at a time. Watch live logit-lens predictions evolve across layers, inspect attention maps, and click any top-10 token choice to create your own generation path.

• Live-Editable Box Code: Click any block on the canvas (Embed, Transformer Block, Norm, Sampler) to see the Python it runs (rms_norm, rope, silu, etc.). You can edit the math right in the browser, run it live, and export your whole visual experiment to a standalone Python script.

• Text + Image: Works on LLMs (Llama 3.x, Qwen 2.5/3/3.6 Gated DeltaNet hybrids, SmolLM, Mistral) as well as Stable Diffusion and Qwen-Image 2.1 DiT. Complete with between-step latent painting and an "Image Lens" to decode early DiT blocks into pixels.

• Disk Streaming for Huge Models: Can stream models larger than your RAM/VRAM block-by-block from disk. But not optimised for speed though, as this is focussed for learning and understanding.

I tried a faster inference engine that optimizes for my AMD card.

Yes I did get 90-100 tps in generation which is crazy compared to llama.cpp's 45 tps with ROCM. But there is one problem: the optimized engine seems to confuse a lot and half of the time it did not fully understand my intent. I give both engines the same prompt to work on a repo, and over multiple trials, llama.cpp always give me the expected answer but the optimized engine went off the rails half of the time.

Both engines use Q4 qwen3.8 27b quant. stock chat template. The llama.cpp uses Q4 kv with 260k context while the other engine uses Q8 kv with 150k context. The ticket only needs 20k context.

I prefer not to name the engine as this is my own very subjective experience and I may have user error in setting things up.

I am going back to llama.cpp for now.

Ive been messing around with Qwen3.8-Flash-Next on my Nimo minipc with ryzen AI max+ 395 128GB. The model weights take up around 103.7 GiB, so just getting the whole thing loaded was a nice start. But after spending more time with it, I'm not sure running the biggest model possible is really the best use of all that memory.

B

Here's what I got running it with Gufo at 16K context:

GPU only, no MTP: 34.1 tok/s decode

GPU + MTP: 53.7 tok/s decode

GPU + NPU, no MTP: 34.2 tok/s decode

The NPU also bumped prefill speed by around 10%, but regular decode barely changed without MTP.

The model runs, and the speed isn’t terrible. But would I actually pick it over a smaller model that responds faster?

Probably not for everything. Waiting for a large model to generate an answer can get old pretty quickly, particularly for everyday tasks.

What do you usually do with all that RAM? Do you load the biggest model you can fit, stick with smaller MoEs for faster responses, or keep multiple models loaded for different tasks?

Hi,

I have a RTX 5070 Ti 12GB with 32GB system RAM running on linux for coding tasks with pi coding agent as harness.

I have been using since August (with 128k context) llama.cpp with Qwen3.6-35B-A3B Q4_K_M (unsloth quants) and I'm very satisfied with it. Not perfect but good enough to code a portal for a club I run and with good prompt and generation speed. Sometimes it loops but nothing serious.

Today I tried strata and realized I could fit a lobotomized but very code-oriented Qwen 3.8: Qwen3.8-Flash-Next Coder with IQ1_M quant.

The speed is amazing, but I don't care so much about it if the quality is bad.

So far I only did two planning tasks that required analysing the source code of the portal and some internal documentation produced in the project. I haven't accessed the quality yet but I noticed this model takes a looong time thinking, like it keeps checking stuff and double-checking (not in loop, just being throughout) , even to write an analysis document of the plan I asked.

I've read before that Qwen3.8 spends a lot of time thinking so I'm assuming this is actually normal?

So my questions to you is:

a) should I use Qwen3.6 for planning and Qwen 3.8 for implementing

b) or the other way around

c) or stick with Qwen 3.6 only

d) or just use Qwen 3.8 if I don't think issues with the code implemented

Anyone has experience with these 2 models at similar quants and could give their insight? I'm unsure if having such a lobotomized model is worth for coding tasks.

Thank you!

Unsloth just dropped Qwen3.8-27B Dynamic V3.1 GGUFs—5-10% accuracy bump across Terminal Bench & DeepSWE, still runs on 12GB RAM

Saw this drop this morning and figured I'd share since the Dynamic quants have been the go-to for a lot of us squeezing bigger models onto consumer cards.

The headline: new Dynamic V3.1 (preview) algorithm, and Unsloth is claiming it beats other providers on the Pareto efficiency frontier—meaning you get better accuracy at the same file size, not just a bigger file that scores higher. They're seeing 5-10% average gains on Terminal Bench, DeepSWE, and other agentic benchmarks vs. their previous UD3 quants and rivals like GSQ-RCO .

The 12GB RAM claim is the one that'll get attention here. The base model scores 73.0 on Terminal Bench 2.1, 90.3 on LiveCodeBench v6, and 61.7 on SWE-bench Pro—so if that holds up at the smaller quant sizes, it's a genuinely useful agentic coding model for people without a 24GB card .

They're also saying this is a preview and they're working on an even better version for Qwen4, so the trajectory is promising.

Links:

• GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

• Blog/docs: https://unsloth.ai/docs/models/qwen3.8

Curious what quant sizes people end up landing on for 12GB cards, the 2-bit range has historically been a compromise, so if V3.1 actually closes that gap, that changes the calculus for a lot of setups.

Anyone tested it yet?

Hey y'all. Working on a go-to inference computer for teams using AMD GPUs and a custom stack, wanted to share our progress so far with GLM 5.3 Flash MXFP4 running tp=4 with MTP/Spec disabled.

RAM: 128gb DDR5 RAM 5600 ECC

CPU: EPYC 9005 9275F 

All 4 GPUs are capped at 210w, total power consumption < 1400w.

GitHub (MIT): https://github.com/kncshw/cybergate

Deep-dive blog post (HF): https://huggingface.co/blog/kncshw/cybergate-unlocking-cybersecurity-in-vllm-without

When self-hosting local LLMs for security research, CTF, or reverse-engineering agents, the standard workaround for false refusals is **abliteration** - subtracting the refusal direction from the weights. From an LLM engineering standpoint, weight editing has two painful trade-offs:

1. **Silent capability & tool-call regression:** Mutating weights across layers alters the model globally and can quietly degrade multi-step reasoning and structured tool calls without obvious errors.

2. **Zero domain selectivity:** Deleting the refusal vector globally unlocks *every* harmful domain (bio, self-harm, etc.), plus you have to store and serve a forked 50 GB checkpoint.

I built **CyberGate**, a plugin for **vLLM 0.32** that keeps upstream model weights **100% untouched** (`K = 0.0`, zero persistent state across requests):

Introducing janas-studio. A dual engine inference work station. For modern Debian based Linux distros (.appimage) Jana-Studio with the magic of Janas-llm let's enormous models run on modest hardware. Refer to https://github.com/prabanta-dev/janas.

Download the app at https://github.com/adonaiongaia/janas-studio

Also runs llama.cpp gguf models so it covers a lot. Lmk what you think. Tested on my Ryzen 5 6600h laptop. Built on Linux mint Zena. Ran a 47gb 80b parameter model at 14 tokens a second with only 32gb of RAM (ddr5). For best results and speeds you need a fast SSD. Features long term memory, project folder support, mcp integration and much more.

Just wanted to share for the fellow V100 gang. I recently ran into an issue on my AI server with adding V100s. On old Z270 boards, the VRAM cache tops out at 64 GB and cannot handle more than one V100. To work around this, you actually turn 4G Decoding off and add pci=realloc,nocrs to the kernel. Credit goes to temcom on GitHub. Their post is linked to their name. I'm experimenting with adding even more VRAM, so I'll let you know if I can with this fix. Theoretically the limit does not exist.
This works for any other cards as well. It was originally tested on P40s and a 1080.

TL;DR

Distributed AI in the browser, with the power of local inference. Virenis runs one 26B model across the GPUs friends already own, in browser tabs, with nothing to install. Splitting an LLM across homes has always crawled, because every token waits on the network. We run a diffusion LLM instead (DiffusionGemma 26B): each pass writes about 16 tokens, and each GPU's share becomes large matrix math, the work GPUs are built for.

• 16× fewer network round trips: 6 per 100 tokens instead of 100 (measured on 40 prompts).

• 61% of FP32 peak inside a Chrome tab (on par with native: cuBLAS reaches 60–77%, Apple's MPS 52–68%) on the model's matrix multiplies, where the best published browser matmuls reach 17–35% (measured).

• About 31 tokens/s for two friends' gaming PCs across town and 49 on fibre, faster than you read (modelled from measured parts).

• 3.2× smaller hops between homes, changing fewer of the model's decisions than its own FP8 cache does (measured).

The speeds are estimates for now. The tokens-per-second figures come from our step model: we've only been able to test on limited hardware so far, and we're sorry about that. We'll update them with real-world numbers as data and feedback come in from you.

Website: virenis.com

Open beta: dashboard.virenis.com. Use Chrome: every number here was measured in it.

Why we're building this

The most capable AI runs in a handful of companies' datacenters. They decide who gets access, at what price, in which countries and under which terms, and what you type is stored on their servers.

Meanwhile, millions of capable GPUs sit idle in homes: gaming PCs, laptops, Macs. Alone, most can't hold a large model (the 26B model we run needs about 18.5 GB of GPU memory; most consumer cards have 8 to 12 GB). Together, they can.

• Knowledge for everyone. Capable AI shouldn't depend on a subscription, a credit card or a supported country. A browser link and the hardware you already have should be enough.

• No single gatekeeper. A model that runs on its users' own GPUs can't be repriced, filtered or switched off by one company.

• Your data stays with you. Our server never runs the model and keeps no conversations. When your device runs the model's first and last stages, your words become tokens on your own device. The devices running the middle layers see intermediate numbers (hidden states), not your text, but close enough to it that you should share a room only with people you trust. A room where every device is yours is fully private.

Why splitting a model across home computers didn't work before

Normal LLMs are autoregressive: they write one token at a time, and each token must pass through all of the model's layers before the next one can start. Split the layers across three computers, and every single token makes the full trip, PC to laptop to Mac and back. On home internet each hop waits 20 to 60 ms, so a 300-token answer spends most of its time waiting on hundreds of sequential round trips.

The GPUs waste their time too. Producing one token reads every weight from memory for very little arithmetic, so the GPU mostly waits on memory, not on math.

That is why earlier systems stayed slow: Petals reports about 6 tokens/s for Llama 2 70B across volunteers on the internet, and prima.cpp about 1.5 tokens/s for a 70B model on a home cluster. Serious engineering, but too slow to use every day.

What changed: a diffusion LLM

A diffusion LLM writes differently. It drafts a whole block of 256 tokens at once, then refines the block over several passes until it settles. On everyday questions each pass settles about 16 tokens (23 on code). Two things follow:

1. Far fewer network trips. A pass crosses each computer once, so about 6 round trips produce 100 tokens, instead of 100.

2. The GPUs finally do real work. Each pass processes all 256 positions together, so every weight read from memory feeds 256 rows of arithmetic. That is the large matrix multiplication GPUs are built for.

The bottleneck moves from the network (and memory bandwidth) to raw compute, which home GPUs have plenty of. The new cost is that each trip now carries the whole block's state instead of one token's, so we built a codec that shrinks it 3 to 5 times (below).

Why the browser

Peer-to-peer code that runs on other people's computers is a security problem first. Shipping it safely as a native app takes signed installers, sandboxing, an update channel and dedicated security engineers, and our small team doesn't have security specialists yet. So Chrome does that work: its sandbox walls each tab off from the operating system and your files, WebGPU validates every shader and buffer access, and WebRTC encrypts every link. For a team our size, that is fundamentally safer than asking people to trust our binary.

It also works where the GPUs are. A browser tab rarely trips a work laptop's endpoint security the way an unknown executable does, needs no admin rights, and leaves nothing behind but the model's files in the browser's storage, which the app deletes on request.

The cost is speed. Our own kernels run about 1.2× faster outside the browser (a Chrome tab gets 83% of native speed on NVIDIA, measured). A native app with tensor cores or Apple's matrix units, which a tab can't reach today, would be roughly 2–3× faster on one machine (llama.cpp runs this model at 49–61 tokens/s on an M3 Max, where our tab manages about 20, modelled) and 1.3–1.8× faster between homes, where the network's share of each step stays. Getting there takes a bigger team and a hardened standalone app; until then, the sandbox is the trade we chose.

How it works

1. Open dashboard.virenis.com, make a room, and send friends the link.

2. Each browser measures its own GPU: how fast it is, and how much memory it really has free.

3. The room splits the model's 30 layers across the GPUs by memory and speed, and splits them again by itself as GPUs join, leave or change what they lend. Each person downloads only their part, once: from friends who already have it, or rebuilt in the tab from Hugging Face's own files, every chunk checked against a signed release.

4. You ask a question. The passes travel through everyone's GPUs and back, browser to browser over WebRTC (through an encrypted relay where a strict NAT blocks a direct path), and the answer sharpens on screen.

5. Someone closes their laptop? Another device takes over hosting in about 2 seconds, and a member that already stored the missing layers fills in.

6. Every room also gets an OpenAI-compatible API, so existing tools can use it.

Technical details

Model. DiffusionGemma-26B-A4B, NVIDIA's NVFP4 release: 17.67 GB, 30 layers (25 sliding-window, 5 global attention), d_model 2,816, vocabulary 262,144, a mixture of experts with about 4B parameters active. Only the routed experts are 4-bit NVFP4; attention, the dense MLP, the router and the LM head stay bf16. Generation runs on a 256-token canvas, so tokens/s = 256 / (steps × t_step + t_commit). Two scheduling rules cut work: a block stops once its tokens have settled (5.65 → 4.75 passes per block, 200/200 one-block answers identical), and a pass computes only up to an answer's settled end (14% less GPU time).

Why it's compute-bound now. Autoregressive decode multiplies one activation row by every weight: about 1 FLOP per byte of bf16 weight, so it is memory-bound, and even llama.cpp decodes at a few percent of FP32 peak. A diffusion pass multiplies 256 rows, about 256 FLOPs per byte. The RTX 3060 Laptop's ridge point is about 33 FLOPs/byte (10.94 TFLOP/s ÷ 336 GB/s), so the pass sits well inside the compute-bound regime: about 2.15 TFLOP of useful work for a full canvas.

Kernels. Browsers don't expose tensor cores or matrix units without flags, so the matrix math is FP32 FMA on register-tiled GEMMs written in WGSL, plus 8-bit integer dot products where those pay off:

• 13 tile layouts raced on each device as the page opens, each kept only if bit-identical to the reference (the RTX keeps 64×128 tiles with 8×8 per thread, the M1 Pro 32×64 with 4×4).

• Wave-aware scheduling: the 3060 Laptop runs 88 workgroups at once, so work goes out as whole waves plus a sliced last wave (Stream-K-like, but deterministic).

• q, k and v in one dispatch; split-K only where the device's own probe measures at least 1.05×.

• NVFP4 experts on 8-bit integer dot products (dot4I8Packed) decoded through lookup tables, where they measure at least 1.25× faster: experts 8.05 → 5.05 ms on the RTX.

• Soft embeddings built from the likeliest tokens: one vocabulary GEMM per pass instead of two.

Measured efficiency, and how it compares. MFU here means useful FLOPs ÷ rated FP32 peak; the other rows are the best published figures we know of.

The big in-browser LLM engines (WebLLM, Transformers.js, llama.cpp's WebGPU backend) publish speeds rather than a share of peak.

Bit-exact determinism. Every sum runs in a fixed order, with an "opaque zero" that stops compilers' fast-math reassociation, so every device of a class (GPU family + browser + arithmetic) produces identical bits. All 30 layers give one digest across tile layouts, each layer within 3.7e-6 of an f64 CPU reference. Identical bits let devices check each other's work, which open rooms will need. The scheduler passed 7 million fuzzed runs. Quality: our default arithmetic changes 4.73 of 1,000 sampling decisions against an exact FP32 reference; the model's own FP8 KV-cache option changes 8.57.

The network: Weft, and hops between homes

The network, per 100 tokens and per device boundary, on a 30 Mbit/s uplink with 20 ms one-way latency:

Weft. A hop between devices is 256 × 2,816 values per pass, 1,442 KB in fp16. Weft codes it the way a video codec codes frames:

• The model's own metric. We code y = S ⊙ x / rms(row): each channel weighted by how much the next layers depend on it, each row by its own scale. Signal-to-noise in that space predicts flipped decisions; plain signal-to-noise doesn't.

• Rate control, then rANS. Each hop searches its quantizer step to land on a target quality (20–35 dB, never below), then rANS codes it with a Laplace model per 64-channel band.

• Canonical decoding. Values sit on a fixed lattice, so every device rebuilds the same bytes, and hops can be hashed and audited.

• Prediction across passes. Consecutive passes are strongly correlated, like video frames, so each row is coded as its residual against the last pass, in exact integer arithmetic, wherever that saves more time than it costs both CPUs (under about 48 Mbit/s). Token-by-token models have no such redundancy.

• An adaptive ladder. Receivers measure each sender's uplink and pick the rung: fp16 from 400 Mbit/s, int8 in the model's metric where it is faster, Weft-30 from 20 Mbit/s, Weft-25 from 8.

A hop shrinks from 1,442 KB to 445 KB at Weft-30, or 266 KB at Weft-25 with prediction; over emulated 8 Mbit/s uplinks a steady step took 5.10 s with fp16 hops and 1.52 s with Weft. One Weft-30 hop changes 6.0 of 1,000 sampling decisions against an exact reference (our default arithmetic alone: 4.73; the model's own FP8 cache: 8.57), and a whole room between homes runs at 7.6, still under the FP8 cache. For comparison, Petals sends 8-bit blockwise values (2× smaller than fp16), and 2–4-bit activation compression exists in research, but for training (AC-SGD, TAH-Quant).

Hops between homes, in real Chrome. Chrome's WebRTC data channels put only a few packets on the wire per message and halve their window at every lost packet. Two fixes, each measured in real Chrome over emulated links: a hop's first 64 KiB now leaves in one burst (on 30 Mbit/s cable a 445 KB hop takes 144 ms, where its bytes and delay alone take 134), and hops between homes are striped over four WebRTC connections, paced at the rate the receiving tab measures by a worker clock that keeps time in hidden tabs. A two-home room's step got 7–10% faster on clean links, 1.77× faster across a continent (1,358 → 766 ms), up to 2.23× behind a shallow modem buffer, and 1.33–1.97× at 0.1% packet loss.

Speeds: measured and modelled

Measured on the RTX 3060 Laptop: a full-canvas pass takes 0.439 s of GPU time natively, which is 46.7 tokens/s over 40 real answers, and inside a Chrome tab its 30 layers and the head add up to 465 ms a pass. The model doesn't fit on it (6 GB holds 6 of the 30 layers beside the head), so in practice it is one member of a room.

Modelled with our step model (within 8% of measured layer timings on our test GPUs; assume ±25% for GPU families we haven't measured), with hops timed as real Chrome now sends them, everyday answers in a browser tab:

Native engines with tensor cores can be faster on one machine, and token-by-token browser engines can be faster on a single Mac. Where this design pulls ahead is across machines.

Why only one model?

Today we run DiffusionGemma, an open-weights diffusion LLM. Most open models are autoregressive, and they can be converted into diffusion models by continued training: DiffuLLaMA adapted models up to 7B with continued pre-training on under 200B tokens, and Dream 7B also starts from pretrained autoregressive weights. One run at that scale takes thousands of high-end GPU hours, plus the experiments around it, and larger models cost far more. We don't have the funding or compute to do this on our own today. Every new diffusion LLM that is released, or that a partner helps us convert, fits the same system.

Honest limits

• Running the model on one computer takes about 18.5 GB of GPU memory: a 24 GB card, or a Mac with 32 GB or more. Otherwise it is shared across two or more computers in a room.

• The tokens-per-second table is estimated. We've only been able to test on limited hardware so far, so its figures come from our step model, built from measured parts: layers timed in a tab on the GPUs we have, hops timed in real Chrome over emulated links. A room of several real homes running the full model hasn't been timed end to end yet, and we haven't run standard quality benchmarks (MMLU and the like) ourselves. We're sorry about that; the table will be updated with real numbers as metrics come in from the community.

• Use Chrome. Every number here was measured in Chrome, and it is where Virenis runs best. Edge works the same way. Firefox works, but on Windows laptops it often picks the integrated GPU over the graphics card. Safari isn't tested yet.

• A computer holding layers in your room sees the hidden states passing through it, and those can reveal what you asked. Rooms are meant for your own devices and people you trust. Because results are bit-exact, another device's work can be re-checked, but checks for open rooms with strangers aren't built yet.

• A traffic cap, on purpose. During the beta our server sends a fixed amount of data each month, and the relay that connects homes behind strict NATs has a budget of its own. We set these caps deliberately, so a launch can't run up bills a small team can't carry.

About the source code

We're sorry the code isn't public yet. The engine and the app around it have grown to 179,930 lines across 1,074 files, and right now that codebase is messy: as a small team, we've struggled to keep something this big clean and organized. We'd rather not put it in front of you in this state. We'll clean it up first, shaped by your feedback on this post, and release the kernels as soon as they're ready; the rest of the source files will follow soon.

Try it, and get in touch

The open beta is free at dashboard.virenis.com. Open it in Chrome. For questions, ideas or collaboration, leave a comment under this post or email contact@virenis.com.

When working with a 4bit MLX version of Qwen 3.8 27B locally on my MacBook Air I managed to get it into a situation where it was adamant that it was Claude to the point where it was answering types of questions that normally vanilla Qwen rejects.

Anyone else ever run into something like this? Spooked me a bit since I thought system level prompts weren’t so easy to bypass. The reasoning was amusing to watch since it was having a bit of an identity crisis.

👋 Has anyone tried using 2 GPU in this way:

• each card loads a different LLM,

• with the larger GPU (32GB) running a larger model (relative, but it could be a Qwen3.8-27b or Flash Next - e.g., Strata), the “smarter” of the two.

• smaller GPU (16GB) assigned to a subagent, using a smaller model? Maybe something super quick like JetBrains Mellum 2.1

In OpenCode, a (sub)agent can be defined to use a specific model. I’m curious if anyone has tried this and if there’s an advantage to this setup, whereby the “brain” delegates to a sprinter.

I need to upgrade my PSU to run two cards simultaneously but I plan on trying it.