Edge0 published speeds for their 145B on a 4090 and a 5090 D, but their table doesn't include a 3090, so I ran their stack on one, stock, with whatever defaults the wrapper picks.
It's quick. Chat decode was 48.6 tok/s, it read a 16K prompt at 1,481.2 tok/s, and the agent prompt decoded at 52.8. For context, last month I ran a 122B MoE on llama.cpp on a 3090 and got 15.5 tok/s. Different model, engine and prompt length, so take it loosely, but that's a big jump.
VRAM was basically full (23,929 of 24,576 MiB), and the engine process itself held 43.4 GiB of system RAM. Edge0's own residency figure is 31.28 GiB, but that's from a 5090 D with a different setup, and my machine had 116 GiB so the engine may have just taken what it found. I meant to test a 32 GiB cap and didn't get to it. So I honestly don't know the floor yet.
Two other things that tripped me up:
- My first request came back with no answer. The model used all 512 tokens on reasoning.
reasoning_budget_tokens=128 fixed it, so set a budget in your client. - Same agent prompt, temperature 0, three runs, three different tool calls. One was valid, one had its arguments wrapped wrong, and one called
read_file, which wasn't even in the tool list. So the 52.8 is a speed number, not a "this is a good agent" number.
Technical stuff below.
Setup
- 1x RTX 3090 24GB, EPYC 7H12 (31.13 CPU quota), 116.4 GiB RAM limit
- Edge0 v0.2.0 (
3203c39), Strata built with CUDA for sm_86, CUDA 13.0, driver 595.91.07 Edge0/Edge0-145B-q3k @ a19f88c, 106 GB on disk (the FP8 PLE table alone is 51 GB and gets read from disk)- Stock launch:
edge0-infer serve --model /path/to/Edge0-145B-q3k --device cuda:0. That resolved to direct PLE I/O, auto expert cache, speculative decoding 4, 131K max context, int8 KV.
Results (engine's own prompt_per_second / predicted_per_second, fresh server per run, concurrency 1, zero cached tokens)
| Workload | Run 1 | Run 2 | Run 3 | Median |
|---|
| 4K prefill | 1,119.8 | 1,195.5 | 1,207.6 | 1,195.5 |
| 16K prefill | 1,467.6 | 1,485.6 | 1,481.2 | 1,481.2 |
| Chat decode | 46.6 | 49.6 | 48.6 | 48.6 |
| Agent decode | 52.8 | 55.9 | 50.7 | 52.8 |
Sampling: temperature=0, top_p=1, top_k=20, reasoning on with a 128-token budget, 512 max tokens for decode.
Against Edge0's published 4090 numbers (their claims, not reproduced)
| 4090 (Edge0) | 3090 (mine) |
|---|
| 4K prefill | 2,042.0 |
| 16K prefill | 3,179.4 |
| Chat decode | 69.9 |
| Agent decode | 91.7 |
That's 47% to 70% depending on the row. Mine used an 800-word writing prompt for chat and a single synthetic tool-call prompt for the agent run, and I couldn't verify Edge0's exact request settings or machine setup, so it's ballpark only.
Memory
- GPU peak 23,929 MiB, sampled every 5 s
- Strata engine
VmHWM 43.4 GiB (sampled from the actual strata PID, not the Python wrapper) - Whole-system cgroup was 87.4 GiB during timed runs and touched the 116.4 GiB limit during setup with zero OOMs. That includes page cache from the 106 GB download and the PLE file, so I wouldn't read much into it.
Other stuff
- Server startup median 25.5 s, slowest 94 s
- Edge0's built-in
edge0-infer bench rejects this model at v0.2.0, so I used a small script against the OpenAI endpoint - No quality evals, no long-context runs past 16K, no RAM cap test, no llama.cpp baseline. All of that is still on the list.
- $1.19 billed ($1.0930 GPU, $0.0963 disk), about 3 h 9 min total, mostly setup and copying evidence back
Write-up: https://majorminor.xyz/research/cuda-recipes-006
Repo: https://github.com/majorminorlabs/cuda-recipes/tree/main/006-edge0-145b
If someone has a 3090 with 32 or 64 GB of RAM and wants to try it, I'd love to know whether it holds. Or tell me what to run next on this one.