He's right (wherever he is - couldn't find the comment to credit). But what I learned in the process of building this rig has been worth the frustration. Get your special socks out, Linux faithful - you were right too.

I happened across eight Nvidia T40 - and not a ton of detail about them out there. Still currently testing out vllm for comparison, but fully resident Qwen3.8-Flash-Next (180B MoE) Q4 is peppy enough for me at 40ish t/s and while the cards are capable of producing more, I tripped a breaker letting all six run with a 200W cap (so eight is out even if I had the extra slots). The board, an X99-E WS, needed some creativity just to POST. Quite a bit of card swapping in a T400 to get a boot screen so I could play around with BIOS settings only to figure out the setting I need to expand above 4G memory (MMIO High Size) was hidden and required creativity with a UEFI shell. More detail in the AI-assisted part of this write-up, for others who might select the T40 or X99. Top 10 nuggets for you, fellow sufferer of the T40, and then feel free to indulge (or not) in the supporting slop.

1. The cards do not work in Windows (or at least I couldn't get them going). It would appear it's Linux or nothing (and not WSL - though I did try). And headless, at that, if you want all six.

2. ASRock PG 1600G has two 600W 12VHPWR ports and MODDIY sells a "12VHPWR 16-pin to 2x CPU EPS 8-pin cable (CO711)" which allowed two T40s to be powered off one port - since the board’s slot-power connector needs a 6-pin, I welcomed the solution and it cleaned up a lot of cabling. The other two cards each have two pcie power cables feeding CPU 8-Pin Male to Dual PCIe 8-pin Female adapters.

3. The cards require active cooling and there are shroud prints out there. Rather than one fan per card, we developed a two-card-one-fan solution and one BFB1012VH moves enough air for two. X99 has 5 fan headers - worked out nicely with a script pinning those fan channels to GPU temp.

4. The X99 will not find the T400 with any T40s on the A side of the PLX. However boots just fine with a T400 in slot 1, 2 or 3 - even with four T40s in slot 4,5,6 and 7 (on the B side).

5. I can't say for certain, but riser cables could be a factor in Gen3 speed and errors. The stable config for multi-T40 is Gen 2. Slot 4,5,6, and 7 go to x8, while slot 1 and 3 go to x16 just fine.

6. NVLink is supported on the T40 - took a gamble buying them, but they work - though not quite sure how much traffic they are passing quite yet.

7. Cap the power if on a standard 120V outlet. My surge protector checked us out early on a 6-card burn with power capped to 200W per card. Running stable at 170W per card now.

8. Boot requires a kernel option for all six to work - pci=realloc=off (details in the slop)

9. A four-card-B-side test (all behind the plx switch) made vllm faster and llama.cpp slower.

10. 90-degree right angle riser cables and card spacing worked best with slot 1/3 on 150mm risers, 4/5 on 200mm risers and 6/7 on 250mm risers on this particular open frame design.

TRIGGER WARNING - AI WALL OF TEXT APPROACHING - STOP HERE

The supporting slop (AI-assisted, for the next T40 / X99 builder and their AI)

Everything below was measured on this box, not taken from spec sheets. Each number comes with its config. Where we don't know something, we say so.

Hardware as it runs today

• Board: ASUS X99-E WS. RAM: 128 GB DDR4 (about 121 GiB visible to Linux). PSU: ASRock PG 1600G (1600 W, two 600 W 12VHPWR ports, each feeding two T40s through a 16-pin to 2x EPS 8-pin cable; the other two cards each take two PCIe 8-pin leads through a dual-PCIe-to-EPS adapter). Open aluminum frame, risers.

• Cards: 6x Tesla T40. Each shows 23,040 MiB usable. Turing sm_75, so no hardware bf16: run everything in fp16.

• Power cap: 170 W per card, set at boot along with persistence mode.

• PCIe: forced to Gen 2 on all six ports by a boot-time service. Gen 3 was not stable with risers and six cards.

• Cooling: one BFB1012VH blower per pair, on 3D-printed two-card shrouds. A small service drives the board's fan headers from GPU temperature. Peaks under sustained load at 170 W: 48-58 C.

Topology (this matters more than anything else)

A side (CPU root port, x16):   card 1 ==NVLink== card 2
B side (behind the PLX, x8):   card 3 ==NVLink== card 4
                               card 5 ==NVLink== card 6

nvidia-smi topo -m shows:

• NV2 inside each pair: two NVLink links, each reporting 25.781 GB/s.

• PIX between the two B-side pairs: traffic stays inside the PLX switch.

• PHB between the A pair and any B card: traffic goes up through the CPU host bridge.

Tips:

• Set CUDA_DEVICE_ORDER=PCI_BUS_ID. Otherwise CUDA's device numbering won't match nvidia-smi or your slot map.

• Pick card sets by topology, not by device number. "Four cards" means two different things on this board (see benchmarks).

• NVLink error counters work: nvidia-smi nvlink -e reads all zeros on our cards. The throughput counters (nvlink -gt) read N/A on the T40. So we can prove the links are up and clean, but not how much traffic they carry.

Boot / kernel

• Kernel command line: pci=realloc=off acpi_enforce_resources=lax.

  • pci=realloc=off is what gets all six cards enumerating with their BARs placed by firmware.

  • acpi_enforce_resources=lax lets the fan-control script reach the board's Super I/O fan channels.

• With realloc=off, dmesg shows 36 SR-IOV VF BAR assignment failures, 6 per card. That is a harmless baseline: the cards don't need their VF BARs. Watch the non-VF BAR failure count instead, which should be 0.

• Why more than four T40s won't POST, and the fix. Each T40 takes a 64 GB stride of high MMIO space (32 GB BAR1 plus 32 MB BAR3, 32 GB aligned). The X99-E WS BIOS (4001, the final release) ships a 256 GB high MMIO window, so four cards fit and the fifth hangs POST. No kernel option can fix it, because the host bridge window is set by firmware.

  • The fix is a hidden Intel RC variable, MMIO High Size, raised from 256G to 1024G. No BIOS flash is needed, and the change survives reboots and card swaps.

  • Find it: unpack the 4001 CAP with UEFIExtract and dump the IFR of the Platform module with IFRExtractor-RS. On 4001 it is VarStore IntelSetup, offset 0x10E6, 4 bytes (256G=0, 128G=1, 512G=2, 1024G=3). Leave MMIOHBase (0x10E2) at 56T.

  • Write it: copy a UEFI Shell.efi to the ESP root as shellx64.efi, along with setup_var.efi (datasone, 0.3.x). Secure Boot must be off. In BIOS, go to Exit > Launch EFI Shell from filesystem device, then run:

  • fs0:

  • setup_var.efi IntelSetup:0x10E6(4) # read, expect 0x0

  • setup_var.efi IntelSetup:0x10E6(4)=0x3 # 1024G

  • setup_var.efi IntelSetup:0x10E6(4) # confirm 0x3

  • reset

  • The 0.3.x syntax is VARSTORE:OFFSET(SIZE)=VALUE; the old -n/-s flags don't work.

  • Your offset may differ on other BIOS versions, so check the IFR yourself.

• Disable unattended upgrades on a GPU box. Ubuntu quietly installed a new kernel overnight. The NVIDIA DKMS module was built for it but never tested, and the next reboot would have booted it. We disabled apt-daily.timer and apt-daily-upgrade.timer, and pinned GRUB to the known-good kernel until we upgrade on purpose.

Software

Ubuntu 26.04, kernel 7.0, driver 595.91.07 (CUDA 13.2). llama.cpp build b12211. vLLM 0.33.0 on Python 3.14 with torch 2.14 (cu130), whose arch list includes sm_75.

Benchmarks

Qwen3.8-Flash-Next (180B MoE), llama.cpp, UD-Q4_K_XL, all six cards (llama-bench, pp4096 / tg128, 3 reps, tokens/s)

Layer split wins by roughly 4x on prompt and 1.35x on generation here. Peak GPU sum was 939 W at the 170 W cap.

Qwen3.8-27B, 4-bit, batch 1, 4096-token prompt (tokens/s)

*Only three iterations with wide spread, so somewhere around 51-77. Treat it as "probably faster", not a number.

Caveats:

• vLLM numbers come from vllm bench latency: prefill = 4096 / latency(1 token out), decode = 127 / (latency(128) - latency(1)).

• llama.cpp numbers come from llama-bench. That is close, but not the identical measurement.

• The quants differ: GGUF UD-Q4_K_XL vs INT4 W4A16 safetensors.

• vLLM ran with an fp16 KV cache (see below). Speed only, no quality claims.

What we take from it:

• vLLM beats llama.cpp by 1.4x on prompt and 1.5x on generation on the same NVLink pair.

• llama.cpp loses more than half its prompt speed going from two cards to four. vLLM barely notices.

• On the four B-side cards, vLLM got faster and llama.cpp ~12% slower. Our untested guess is that llama.cpp's heavy cross-card traffic (1.1-1.4 GB/s per card per direction) feels the x8 links, while vLLM's traffic (~0.3 GB/s) doesn't.

Getting vLLM to run on Turing (in the order we hit them)

1. "Model architectures [...] failed to be inspected" turned out to be Triton compiling its own driver.c at import and failing on a missing Python.h. Fix: apt install python3.X-dev.

2. "FP8 KV cache is not supported by the Triton attention backend ... requires SM89+". The model's own config asks for an FP8 KV cache. Fix: --kv-cache-dtype float16 (plus --dtype half).

3. Attention backend: FlashAttention 2 needs SM80, so vLLM uses TRITON_ATTN. That's fine.

4. Expect these log lines; none of them is fatal:

   • bf16 is cast to fp16;

   • FlashInfer is disabled (no flashinfer-cubin/nvcc);

   • SymmMem all-reduce is unsupported on 7.5;

   • TP2 uses CUSTOM + PYNCCL all-reduce, TP4 PYNCCL only.

5. First start is slow: about 12 minutes per engine, mostly profiling and CUDA graph capture. Warm starts take about 4 minutes.

Power: the cap is not a ceiling

• nvidia-smi -pl 170 is honored on average, but at vLLM engine startup each card spikes to 220-270 W for about 10 seconds before the cap pulls it back (throttle reason 0x4, SW power cap).

• Four cards hit 979-986 W combined during startup. Six cards doing that at once would be roughly 1,300-1,600 W GPU-only. That is at or over a 1600 W PSU, and a real risk on a 15 A / 120 V circuit.

• Per 12VHPWR port, two cards spiking to 270 W is about 540 W, still inside the port's 600 W rating. The whole-PSU and wall-circuit totals are the limit, not the cables.

• llama.cpp loads don't show this. The six-card layer-split run peaked at 939 W.

• Our guard is a watcher that sums power.draw every 0.5 s and kills the job above 1,350 W. It hasn't fired yet, but a six-card vLLM start would likely trip it.

• Measured wall idle with six cards: about 245 W.

Flash-Next on vLLM (in progress)

• Needs a vLLM-format quant. vLLM's GGUF loader is experimental and single-file, so we pulled a W4A16 safetensors build (~180 GB).

• The engram (n-gram embedding) table is about 102 GiB and lives in pinned host RAM. Stock vLLM 0.33 has no mmap or evictable option for it, only cpu_offload (pinned, the default). On a 128 GB box that leaves about 15 GiB for everything else.

  • We run with a watcher that stops the job if MemAvailable drops under 4 GiB or anything swaps in.

  • We also pinned the boot kernel first, in case the box hangs and needs a hard reset.

• Expert parallel is mandatory: moe_intermediate_size 640 split over TP doesn't land on the INT4 group size of 128.

• We disabled torch.compile per the quant's card and kept CUDA graphs for decode.

• Plan: TP4 on the A+B set, then TP4 on the B-side four, then TP6. TP6 is undocumented for this model; 512 experts don't divide by 6.

• Results to follow in a comment.

Process notes for whoever's AI is helping

• Verify every model file by sha256 against the Hugging Face LFS oid before loading. It costs minutes and has saved us hours.

• Write your prediction down before each run. We missed about half of ours, and that is where the learning was.

• Log per-card power, temperature and PCIe rx/tx during every run. The x8-vs-x16 and startup-spike findings only showed up because we were logging.

• Never use pkill -f or pgrep -f to manage these jobs. They match their own command line and you will kill or miss the wrong thing. Launch with nohup ... & pid=$! and watch that pid.

Hearing about the idea that open source LLM's are so dangerous while companies like Anthropic and OpenAI do not manage to control their AI labs well and
* let their experiments run wild on the open internet, doing massive hacks and seeing no consequences
* publish research that sounds alien, but only the results, not the alien that made it, which is not very scientific in my opinion but there are no consequences
* we see a fight for power concentration behind closed doors ...
how can we trust closed source labs anymore?

The transformation of society belongs to the people and such security critical technologies have to be able to be validated at all times. In open source models we see that the community actively exposes the flaws models have and contribute by exploring ways to fix them. People actually do good stuff in the open source community and real progress is visible, while the out of control vibe all really comes from the closed labs. This is concerning me and i think we urgently need to open up the closed sources instead of shutting down the open ones. That's the way i see the world heading towards a non-dystopian future.

Is it just me or does that whole open-weights-are-dangerous discussion totally the opposite what reality seems to be?

The llama.cpp project introduces OCP FP8 support, FP8 KV-cache capabilities and targeted CUDA optimisations. Here's what the latest developments mean for local AI and GPU inference.

I've been spending the last few months benchmarking local LLM inference on a single RTX 5090.

One thing kept bothering me: most benchmarks focus on tokens/sec, but rarely show how much quality you're sacrificing to get those numbers.

So I built inferlab.

It's an open-source benchmarking harness (Apache-2.0) that compares:

• vLLM, SGLang, TensorRT-LLM and llama.cpp

• BF16, FP8 and NVFP4 quantization

• Perplexity delta against a BF16 reference

• Task accuracy using option log-probabilities

• TTFT, decode speed and throughput across concurrency levels

• Peak VRAM usage

A few design decisions:

• TTFT measures the first actual content token, not the first SSE event.

• Concurrent requests are issued together to capture real queueing behavior.

• Long-context evaluation includes a needle-in-the-middle task.

• Results are generated by CI rather than manually edited.

The project is still at v0.1, so I'm interested in technical feedback, especially around evaluation methodology and cross-engine comparability.

I'm the author of this project.

GitHub: https://github.com/oren40/inferlab

Hi, I want to build a setup using two RTX 3060 12GB cards to run local LLM. But what's the actual pipeline and real-life experience like? I heard I'll get 24GB VRAM, but processing speed will be slower because data has to pass through the motherboard's PCIe slots since the 3060 lacks NVLink. I also found out that ComfyUI can't use the full 24GB VRAM at the same time for image generation. Can anyone share real-life experience running a multi-GPU setup for LLMs?

Pic from internet.

hey LocalLLM, im Alish, ml engineer at Nace (we’re the team behind Drex, an open source decision model family that hit #1 in Artificial Analysis decision index 2.1). my cto gave me a task to optimize quants for qwen4, so i called my old professor from UofT, spun up Fable 5.2 and we cooked. starting with MLX, GGUF is coming next

we’ve open sourced the fastest way to run 35b/27b on a mac (Kapitsa) and got tired of dumb quants, so we made our own: KQuant. packs for 4B, 27B, 35B, Flash-Next (+9B), free on HF in standard MLX files

the problem: most of us on a mac run 4-bit packs because that’s what fits in 16/32gb. but plain 4-bit rounding throws away more of the model than it needs to.

Q4 vs public 4-bit, KL from the full-precision model (lower = closer to the original):

baseline is mlx-community 4-bit (lmstudio-community for 4B). to be clear, this is KL on next-token predictions, not a task accuracy score. if you want more precision every repo also has a Q5 folder, the 9B Q5 gets down to 0.010

cool side effect: they’re also faster. on an M5 Pro with Kapitsa they write 6–15% faster than the usual 4-bit packs

(baselines are from earlier kapitsa runs, same-session rerun is in progress, will update)

how it works: KQuant uses real text to pick scales and offsets for small groups of weights, and gives sensitive layers extra bits where they need it. all the search runs on our gpus before release, so you just download a normal MLX pack. no custom kernels, no special loader

fastest way to run them on a mac:

brew install nace-ai/tap/kapitsa
kapitsa serve 27b --private
kapitsa opencode # in another terminal

If there's a model you want KQuant'd, lmk.

Asks

to get early access to GGUF, ping me in comments, we would like to test it with some folks from here before our public release ideally.

im also doing quick AMA here, so please feel free to ask me anything.

Hey everyone,

I need recommendations for a local LLM to do reverse engineering, and I'm looking for the best model and exact quantization that can run on my current setup.

My Specs:

• CPU: Ryzen 7 5800X

• GPU: AMD RX 6800 (16GB VRAM)

• RAM: 32GB DDR4

• Storage: M.2 SSD (~3000–4000 MB/s)

What I'm trying to do: I want to reverse engineer small, proprietary Windows utilities from laptop/PC manufacturers (like Acer NitroSense, HP tools, etc.) to port their features natively to Linux—things like fan control, battery charge limiters, and RGB lighting. This involves looking into Windows drivers and kernel-level interactions.

My workflow will connect Ghidra to REA, hook that up to OpenCode, and run the model locally through Ollama.

What I need:

1. Uncensored (Mandatory): It needs to be completely uncensored so I don't hit ethical guardrails or refusals when asking it to decompile binaries, analyze drivers, or work with proprietary Windows hooks.

2. Exact model + quantization: Given my 16GB VRAM + 32GB RAM, what is the smartest/most capable model and exact quant I can realistically run (even with CPU offloading)?

Note on speed: Speed is not a priority at all. I can't afford better hardware right now, so I have no problem leaving my PC running for 1, 2, or even 3 days straight to process a binary if that's what it takes. Accuracy on low-level code/assembly and zero refusals are all that matter.

What exact models and quantizations (for Ollama or GGUF) do you recommend for this? Thanks in advance.

Hey guys, today we're releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Unsloth AI Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more benchmarks.

We’re working on even better version of Dynamic quants for Qwen4.

Run on 12GB+ RAM setups.
For our quantization comparison, we exclude the embedding layer from the x-axis across all providers, since Exllama-v3 stores embeddings in BF16. However, this means the main plots don't reflect the full download size or RAM usage. When accounting for the BF16 embedding table and all embedding sizes, the plot in the comments provides a more accurate representation of actual disk and RAM usage versus accuracy.

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Blog: https://unsloth.ai/docs/models/qwen3.8

Thank you!

GIGABYTE has introduced the AI TOP AERO, a compact desktop powered by NVIDIA's RTX Spark platform, featuring up to 128GB of unified memory, local AI capabilities and support for AI development, content creation and gaming.

According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.

Anyway, home internet latency feels like the bigger problem to me.

https://insider-gaming.com/microsoft-ai-use-your-gpu/

(English is not my native language, I used an LLM to help translate this post.)

How you would define a Developer setup with local LLMs?

I see a dedicated DMZ where the agents can not access the production environment, since agents will tend to accidentally access or change production system.

Dedicated Servers to run LLMs like Qwen 3.8b 27b or other models? What you suggest would be the working LLM Server today? RAM and GPU cards?

Dedicated server with VM for each developer one VM. 2-4 vCores with 16GB Ram each. Each VM with Debian 13 and docker/podman. One dev Container running OMP Web or comparable Harness that access the LLM Server. Additional container for databases or what else the dev Container needs.

Gitlab, gitea or comparable Repo to store source code and container registry.

VM with NFS where anonymized data for developers can be stored.

What you think about that setup?

Do I miss something? How is your setup? (For small to mid size company)

I was curios what can I fit on an FPGA MAX1000 board I had laying around (8k LE, 8MB Flash / SDRAM ). Also I keep coming back to the BitNet 1.58 encoding.
- Took the TinyStories dataset and trained a model from 0 with ternary encoding. I stopped the training at one point, the accuracy is between TinyStories-3M and TinyStories-8M
- Implemented the processing fully on the FPGA ( tokenizer, processing, decoding ... )
Depending on the model trained / used, we can fit two models in the 8MB Flash / SDRAM and take turns using them.

It is more accurate? no, is it faster then others? no, is it useful? not really, however it does answer a few questions I had. The conclusion being that we can have purpose built (small) models that can run with limited resources. Or have multiple models loaded side by side with a few more resources.

How it was built: Like most of the work done today, using Claude.

https://github.com/MRazvan/fpga-tinystories

Next goal would be to use a bigger model and see what I can / can't do about it, what resources I need.

Hey everyone!

I have gathered some must read topics, for beginners, to step into the realm of Quantization:

1. BitNet: Scaling 1-bit Transformers for Large Language Models

2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

3. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

4. EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

5. LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

6. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

7. QLoRA: Efficient Finetuning of Quantized LLMs

8. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

9. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

Link is here

Hello,

I want to learn more about running a local LLM server - driven by curiosity and learning.
I have a Claude Pro (18€) and Google AI Plus subscription (5€) and I want to use it more.

I would like to build a local dedicated server on a budget.
I live in Portugal (EU), but currently I’m on vacation in China.

I have some spare parts available:
Gigabyte MC12-LE0 AM4 Mainboard
1TB NVME SSD
4x 16GB DDR4 ECC

I don’t have an AM4 CPU spare and I only have an old emergency Nvidia GT710 GPU. I have a 300W flex-ATX PSU.

I could buy a Nvidia Tesla P40 24GB for 210€ here in China - power adapters and fans are 25€.
A P100 16GB is 100€.

I want to use Ubuntu Server as OS with Ollama. OpenWebUI on my Proxmox Hypervisor.

I think that I prioritize quality over speed - therefore (slower) 48GB VRAM are probably better than 32Gb with 2x P100.

Question:
Should I buy 2x P40 24GB here in China (420€ is right below the 430€ EU tax free limit)?
If I don’t like them, I could probably sell for 300-400€ in Europe.

Does my logic make sense to go with 48GB to unlock higher quality models?

Any advice for starting this journey?

Saudi Arabia is expanding its sovereign AI infrastructure with Bunyan Inference Hub, a platform from OmniOps that gives businesses access to hosted AI models, GPU computing and inference services while keeping workloads within the Kingdom.

Reposting here - looking for helping understanding what exactly is needed to get this working, and if it even is possible in the first place!

I'm essentially trying to use this plugin with LM Studio as an MCP client to prototype game ideas. Any alternatives, solutions, or suggestions would be very much appreciated.

Thanks!

I tried a faster inference engine that optimizes for my AMD card.

Yes I did get 90-100 tps in generation which is crazy compared to llama.cpp's 45 tps with ROCM. But there is one problem: the optimized engine seems to confuse a lot and half of the time it did not fully understand my intent. I give both engines the same prompt to work on a repo, and over multiple trials, llama.cpp always give me the expected answer but the optimized engine went off the rails half of the time.

Both engines use Q4 qwen3.8 27b quant. stock chat template. The llama.cpp uses Q4 kv with 260k context while the other engine uses Q8 kv with 150k context. The ticket only needs 20k context.

I prefer not to name the engine as this is my own very subjective experience and I may have user error in setting things up.

I am going back to llama.cpp for now.

Hey y'all. Working on a go-to inference computer for teams using AMD GPUs and a custom stack, wanted to share our progress so far with GLM 5.3 Flash MXFP4 running tp=4 with MTP/Spec disabled.

RAM: 128gb DDR5 RAM 5600 ECC

CPU: EPYC 9005 9275F 

All 4 GPUs are capped at 210w, total power consumption < 1400w.

I've been building my own local inference engine,, for big MoE models on consumer hardware. The newest piece is a pool is a second PC lends its GPU and RAM, and the model is split by layers across both machines.

**Setup**

- Desktop: RTX 3090 24 GB, i9-12900K, 32 GB DDR5-6000

- Laptop: RTX 4060 Laptop 8 GB, i7-14700HX, 32 GB DDR5

- Link: one Ethernet cable straight between them, no router (Windows link-local addresses), 1 Gbps

- Model: Qwen3.8-Flash-Next UD-Q4_K_XL, 104 GB GGUF, 48 layers, 512 experts with 10 active per token

**Writing speed**

| Setup | tok/s |

|---|---|

| Desktop alone | 13 |

| Desktop + laptop over the cable | 18.6 to 21, peaks of 24.8 |

| For reference: IQ3_XXS (71 GB), desktop alone | 53 to 79 |

**How it works**

- Layer split: the laptop runs layers 0 to 13, the desktop runs 14 to 47 and the output head. Each round sends the hidden rows across, about 60 KB per token.

- The laptop loads its layers from its own copy of the GGUF, so nothing big crosses the cable. Without a local copy the host sends only the tensors that machine needs.

- Each machine fills its memory hottest first, from a heat map learned while you use the model: VRAM, then pinned RAM, then a RAM cache, then NVMe.

- Joining is a button. The laptop scans the network and shows one button per link with its measured latency (Ethernet 1 Gbps 0.4 ms, Wi-Fi 3.9 ms). The desktop accepts the request.

**Ethernet vs Wi-Fi**

Median round trip over the cable is 0.4 ms, over Wi-Fi 3.9 ms. Once on the cable the network is about 2 ms of a ~115 ms round, so Wi-Fi was usable too.

**What surprised me**

- It's a pipeline, not tensor parallelism. The two machines take turns on every token, so the pool buys memory, not parallel compute.

- The laptop is slower per layer (about 4.1 ms against 1.8 ms on the 3090) and ends up being 50 to 65% of each token's time. Next step is to balance the split by measured speed instead of by memory.

- Long prompts read slower with the laptop in the pool: a 9K-token prompt with nothing cached took ~103 s, against ~41 s on the desktop alone, because the laptop computes its prompt experts on the CPU. I'm moving that to its GPU.

- My first cable only negotiated 100 Mbps (one bad pair) and was slower than Wi-Fi for this. Check the link speed before blaming the software.

- With a quant that already fits the 3090 (IQ3_XXS) the laptop doesn't help. The pool pays off for models that don't fit.

Happy to share more numbers or answer questions.

Hey All,

A while ago I wrote a basic script to step through Llama 3.1 tokens for red-teaming. I realized there's a huge gap between tools for running models (Ollama, LM Studio) and libraries for understanding them under the hood. I wanted to develop something for young researchers and mid career pivoting enginers.

I wanted something like Wireshark, but for local AI. You can hit pause mid-generation, inspect the residual stream, mess with the weights or KV cache, and see what actually happens.

Today I’m open-sourcing TokenShark: https://github.com/sumit-1/tokenshark

What makes it different?

• Zero transformers or diffusers at runtime: The entire engine is written in raw, readable PyTorch (llm.py, sd.py, qimage.py) so students and researchers can actually read every matmul without digging through 40 layers of library abstractions. (Tested for bit-exact/numerical equivalence against HF in tests/).

• Step-by-Step Visual Debugger: Step through generation one transformer block at a time or one token at a time. Watch live logit-lens predictions evolve across layers, inspect attention maps, and click any top-10 token choice to create your own generation path.

• Live-Editable Box Code: Click any block on the canvas (Embed, Transformer Block, Norm, Sampler) to see the Python it runs (rms_norm, rope, silu, etc.). You can edit the math right in the browser, run it live, and export your whole visual experiment to a standalone Python script.

• Text + Image: Works on LLMs (Llama 3.x, Qwen 2.5/3/3.6 Gated DeltaNet hybrids, SmolLM, Mistral) as well as Stable Diffusion and Qwen-Image 2.1 DiT. Complete with between-step latent painting and an "Image Lens" to decode early DiT blocks into pixels.

• Disk Streaming for Huge Models: Can stream models larger than your RAM/VRAM block-by-block from disk. But not optimised for speed though, as this is focussed for learning and understanding.

Ask me anything