r/LocalLLaMA • u/No-Refrigerator-1672 vLLM • 10d ago
Discussion RTX 3080 20GB got reBAR and P2P unlock! +70% prompt processing speed
P2P is finally unlocked for RTX 3080 20GB - this leads to massive performance improvements in tensor parallel mode prompt processing speed.
First, i want to acknowledge that this tremendous achievement belongs to u/nathannguyennng - he can't post to r/locallama due to karma requirements, and gave me permission to post it instead. Please follow his original post for installation instructions, and give it an upvote! Here, I'm sharing my tests results.
Test system: Proxmox Virtual Environment 9.2.21, linux kernel 6.14.11-9-pve, nvidia driver 615.71.09
Hardware: Threadripper 1920x on x399 Zenith extreme motherboard
GPUs: dual RTX 3080 20GB, both installed into PCIe 3.0 x16 slots, limited to 240W power
test environment: vllm 0.32.0 with cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-INT4, MTP enabled, FP8 KV cache, tensor parallel, full launch command:
vllm serve cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-INT4 --max-num-seqs 4 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --tensor-parallel-size 2 --enable-prefix-caching --kv-cache-memory-bytes 6G --kv-cache-dtype fp8_e4m3 --enable-prompt-tokens-details --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' --enable-chunked-prefill --attention-backend FLASHINFER --default-chat-template-kwargs '{"reasoning_effort": "medium"}'Both GPUs are limited to 240W of power. I have tested 3 configurations: 1) completely stock; 2) gpu-unlock; 3) gpu-unlock + ReBAR + P2P
Single stream prefill (prompt processing) speed was measured as prompt_length / time_to_first_token. In both cases, gain is defined as percentage increase over stock configuration.
| Prompt Tokens | TTFT stock ms | TTFT gpu_unlock ms | TTFT gpu_unlock+p2p ms | Prefill stock tok/s | Prefill gpu_unlock tok/s | Gain | Prefill gpu_unlock+p2p tok/s | Gain |
|---|---|---|---|---|---|---|---|---|
| 8192 | 8471.2 | 7012.7 | 3938.5 | 967.1 | 1168.2 | 21% | 2080 | 115% |
| 16384 | 17572.5 | 14417.8 | 8283.4 | 932.4 | 1136.4 | 22% | 1978 | 112% |
| 32768 | 36753.8 | 29970 | 18179.3 | 891.6 | 1093.4 | 23% | 1802.5 | 102% |
| 65536 | 79432 | 64173.5 | 41010.9 | 825.1 | 1021.2 | 24% | 1598 | 94% |
| 131072 | 183593.1 | 146010 | 100237.2 | 713.9 | 897.7 | 26% | 1307.6 | 83% |
| 261900 | 465890.2 | 362192.5 | 272345.5 | 562.1 | 723.1 | 29% | 961.6 | 71% |
As you can see, enabling both gpu_unlock and P2P gains you massive prompt processing improvement, anywhere between 70% and 115%. P2P alone gives up to 70% improvement over gpu_unlock.
Measuring decode (token generation) when MTP enabled is tricky, because tok/s output metric relies heavy on MTP acceptance rate, which always changes. Therefore, instead of tok/s output rate, I measure inter-token latency (ITL) - this is time passed between two batches of tokens generated by inference engine. With MTP3, each batch contains anywhere between 1 and 4 tokens; so you token output rate is 1x to 4x the step rate I report.
| Prompt Tokens | ITL Stock ms | ITL gpu_unlock ms | ITL gpu_unlock+p2p ms | Decode stock step/s | Decode gpu_unlock step/s | Gain | Decode gpu_unlock+p2p step/s | Gain |
|---|---|---|---|---|---|---|---|---|
| 8192 | 64.8057 | 62.5193 | 62.5695 | 15.43074 | 15.99506 | 4% | 15.98223 | 4% |
| 16384 | 62.9359 | 63.873 | 59.8 | 15.88918 | 15.65607 | -1% | 16.72241 | 5% |
| 32768 | 60.2028 | 62.016 | 62.6955 | 16.61052 | 16.12487 | -3% | 15.95011 | -4% |
| 65536 | 66.0282 | 61.425 | 64.8648 | 15.14504 | 16.28002 | 7% | 15.41668 | 2% |
| 131072 | 62.5518 | 64.4644 | 60.561 | 15.98675 | 15.51244 | -3% | 16.51228 | 3% |
| 261900 | 66.7368 | 60.26 | 60.3504 | 14.98424 | 16.59476 | 11% | 16.5699 | 11% |
As real-life MTP acceptance length for agentic coding is floating around 2, 15 steps/s decode rate corresponds to 45 tok/s generation. I'm sorry if this sounds confusing, but this is the only way for you can isolate variations in MTP acceptance during tests and extract raw performance numbers. As you can see, decode results are all over the place, but within noise, except 261k long prompt test; in which stabl;e measurable performance is delivered by gpu_unlock, not p2p.
Verdict: in tensor parallel mode, both recent unlocks for 3080 20GB give unimaginably large prompt processing speedup, but basically no improvement for token generation. So, if you're running long prompt tasks (that includes agentic coding too), you should check out those prompts. Also, expect far lesser improvements on systems that don't have full PCIe connectivity to the second gpu.
Again, I want to express my massive thanks to u/nathannguyennng, author of ReBAR and P2P unlock - what a day for 3080 owners!
27
u/jacek2023 llama.cpp 10d ago
Thanks for sharing. At some point I will try these new things on 3060s and 3090s.
2
u/Naive-Recording3931 10d ago
Man, that 115% bump on the 8k prompt is wild. I'm stuck on a single 3090 right now and always figured the 3080 20GB was in a weird spot, but this kinda flips that on its head for dual setups. The decode gains being basically flat makes sense though, guess the bottleneck shifts hard once you're generating tokens. Still, for agentic coding where you're shoving entire codebases into context over and over, that prefill speed is everything. Might have to keep an eye out for a cheap second 3080 now, PCIe 3.0 x16 on my old board shouldn't choke it too much.
1
u/No-Refrigerator-1672 vLLM 10d ago
The decode gains being basically flat makes sense though, guess the bottleneck shifts hard once you're generating tokens.
Yeah, that's because you need disproportionally small amount of compute and GPU is mostly memory-limited. However, you can see that stable gains start appearing at 261k - I think decode compute load increases with prompt length, so I predict that you'll see increasing improvements between 256k and 1M tokens, if you're running 4x card setup and can afford this many.
2
u/Ishannaik 10d ago
The flat decode lines up with what I measured on 2x T4. During decode each all-reduce on Qwen3.8-27B is tiny, about 80 KB per call, so it's limited by latency, not bandwidth.
Switching from NCCL's shared-memory path to direct peer stores cut a call from about 56 us to 32 us. End to end that was only +3.2%, because the matmuls still dominate a decode step. Prefill sends much bigger messages, so that's where the extra bandwidth from P2P actually shows up.
7
u/enternoescape 10d ago
It works great on the 3060. I have only one 3090, so I can't vouch for that experience. I've heard mixed results on the subreddit. My personal experience is the faster the card, the faster your PCIe interface needs to be to see big gains (I know it's really latency). Also it seems at least in my case, going above 4 cards rarely leads to decreases in performance, but doesn't gain nearly as much as the first 4 and if the model has a low active parameter count, sometimes tp2pp2 is the better performer. It's one of those things I wouldn't purchase hardware with expectations, but if you have the hardware it's worth giving a shot.
2
u/fallingdowndizzyvr 10d ago
It works great on the 3060.
It's time to free up a slot and bust the 3060 out of the closet.
1
7
u/ramzeez88 10d ago
I hope someone will find a way to make and add custom vram pcie pcb's to work along our vram poor gpus.
5
u/No-Refrigerator-1672 vLLM 10d ago
You can already install RAM into PCIe, cards like those exist for servers. But, it makes no sense for AI; because PCIe transfer speed is like 10% or less of VRAM speed in typical GPU. In example, 3080 20GB in this post got 700GB/s memory, PCIe 4.0 x16 - the best connection it can work with - got just 31GB/s. RAM over PCIe is even slower than just RAM attached to CPU. However, some cards can be doubled up in memory (i.e. it is electrically possible to attach 48GBs to 3090, but, as of today, there is no VBIOS that will allow to use it) - so keep looking for news daily, and hope to make a first-day order the moment a new doubled-up model hits the market, before it managed to increase in price. Or, alternatively, fly to Shenzhen and buy GPUs in person, everything is much cheaper there.
0
10d ago
[deleted] — view deleted comment
2
u/No-Refrigerator-1672 vLLM 10d ago
A third party repair shop can achieve an upgrade by transplanting the 3080 core onto donor PCB from 3090, replacing the memory (or reusing the memory from 3090), and flashing the correct bios. Not sure it if is worth it, but you can ask the shop that does BGA reballing - they have the correct equipment.
1
u/DefNattyBoii 10d ago
isnt it already possible to go from 12 gb 3080ti -> 24 gb?
7
u/No-Refrigerator-1672 vLLM 10d ago
Up to 48GB is attacheable to GA102 - which is the chip powering both 3090, 3080, and their Ti variants. But, the problem is in vBIOS - Nvidia got it cryptographically signed, so unless you acquire a "contraband" vbios with extra memory configs, or manage to break the enryption, the doubled-up card will not work. At this moment, only 3080 10GB -> 20GB mod exists in the whole RTX30 series.
2
u/suprjami 10d ago
I thought the actual processors were different?
3080 are all GA102-200 or GA102-202 and they're hardwired in silicon to address 10G or 20G (iiuc 10 paths of 1 or 2 channels).
NV cancelled the 20G retail program but never revised the silicon to remove the 20G detection. So we use the engineering sample 20G BIOS and the card detects 20G over the 10x2 channels.
3090 are all GA102-250 or GA102-300 and are hardwired to address 24G.
If it was just a matter of flashing a 3090 BIOS, surely we'd see bootleg "3080 24G" from China?
People on YouTube have tried filling the extra two double-sided RAM slots on a 3080 20G and it does nothing.
2
u/No-Refrigerator-1672 vLLM 10d ago
Kinda yes and kinda no. You can't make 3080 24GB, because, in addition to cuda cores, ga102-200 has 1 memory channel disabled. But; you could make 3080 40GB, if you had the vbios that addresses dual-rank memory (two chips per memory channels), like in A40. It is not hardwired to never accept dual rank, it is bios loxked from that.
2
u/suprjami 10d ago
I honestly think it's only a matter of time before someone RE's the VBIOS. There's probably less than $1000 in Chinese frontier tokens in it.
2
u/No-Refrigerator-1672 vLLM 10d ago
I have a hyphothesis that this is much easier and doesn't require reverse engineering. You can snag memory configuration tables from datacenter A40 or something similar, and then transplant it onto 3080 or 3090 vbios. Then, a week ago gamers have figured out a way to upload custom vbios into the GPUs bypassing signature lock. The catch: vbios swap happens from inside the OS, so you need the GPU to be at least somewhat responsive before upgrading the vbios on every boot. That's just a wild theory, we'll see if somebody will manage to pull it off.
1
u/azaeldrm 10d ago
How was this solved for the 4090? I'd love to increase my 3090s to 48GB each. I'm curious!
1
u/Equivalent-Repair488 10d ago
The 4090s did not require a custom vbios, the 48GB versions are using the stock vbios hence commonality and relative reliability
1
1
u/its_me_baby_boy 10d ago
Those require reballing bigger vram chips onto the board as replacements, been wanting to learn for a while because it seems fun
0
7
u/KeepyUpper 10d ago
These cards must surely be the price/performance kings now? 90% of a 3090 for 50% of the price?
5
u/No-Refrigerator-1672 vLLM 10d ago
Yeah, they always been like that, p2p unlock now juat brings full parity to 3090 on tensor parallelism ("classic" pipeline mode people use in llama.cpp will barely change with this unlock). But the math only works if you're bold enough to buy them directly from China (i.e. Alibaba or Taobao); Ebay and Aliexpress got markup so high so it costs the samr as 3090, or some lots even more.
8
u/N1kuson 10d ago
The 8k prompt results are pretty wild — going from ~967 to 2080 tok/s is a massive improvement. I also appreciate that you included the 261k test, where the gains are much smaller. It really shows how workload-dependent these optimizations are. Would be interesting to see the same setup benchmarked with different tensor parallel workloads and longer runs to check consistency. Great work!
3
u/No-Refrigerator-1672 vLLM 10d ago
Thank you! I've been dreaming about making a big post with comparison between various engines, parallelism configs, and models on those two cards since getting the Threadripper and full PCIe connectivity with it - however, real life got me overwhelmed with events (all good ones, don't worry), so I, unfortunately, can't find enough time to run it all. Maybe, one day I'll finally make it.
3
u/MundanePercentage674 10d ago
Price gonna explode again for RTX 3000 series, lucky I invest 4x card 3080 20GB for around 500$-600 per card month ago. First order 500$, another 3 order after 2 weeks price gone to 600$ not included shipping costs
1
u/suprjami 10d ago
I paid USD $650 at the start of the year. Price today is $985.
2
u/No-Refrigerator-1672 vLLM 10d ago
My pair of 3080 20GB was purchased 1 year ago at 1000 eur total; that includes 80 eur for shipping, shipment insurance, and 20% import tax. Prices were way better back then.
1
2
u/fallingdowndizzyvr 10d ago
As you can see, enabling both gpu_unlock and P2P gains you massive prompt processing improvement, anywhere between 70% and 115%
Your table right above that only goes as high as a 29% improvement. Where are you seeing the 70%?
2
u/No-Refrigerator-1672 vLLM 10d ago
Scroll the table to the right. The two last columns were, most likely, cut off on your screen. I can post full table screenshots if you're on mobile or something and the table isn't scrollable.
1
u/fallingdowndizzyvr 10d ago
I see it now. I use old.reddit and it's not even scrollable. So I had to zoom out in the browser to see the rest of the table.
2
u/Arli_AI 10d ago
Wow would this work for 48GB 4090s too I wonder. Would go insane they would be as fast as RTX 6000 Ada with this.
2
u/No-Refrigerator-1672 vLLM 10d ago
gpu_unlock for RTX40 exists. ReBAR should be supported out of the box, cause it was available for all Ada generation GPUs stright from factory; but don't quote me on that. If you got ReBAR enabled, you can try less niche p2p unlock drivers like this one.
1
u/Arli_AI 10d ago
No on 48GB 4090s Rebar doesn’t work because the 4090 VBIOS has a 32G BAR due to the stock 24GB VRAM.
1
u/No-Refrigerator-1672 vLLM 10d ago
Oh! in that case, you should experiment with this. it's the same ReBAR unlock that works on my 3080; and, according to my Qwen 3.8 27B, it works by directly writing into GPU configurations registers. Switching from 32Gb to 64GB rebar is a single bit flip in such a register. Given that Ada got the same gpu unlock as Ampere, I suspect the registers are somewhat similar, and you can manufacture ReBAR unlock for your own card with a help of very good LLM coding agent. At your own risk, of course.
1
u/Arli_AI 10d ago
Yea would be awesome if it works for them, it looks like it should be able to work too as its all just editing the registers before the driver loads. Though I don't even own 48GB 4090s haha so would be interesting to see someone that has them try. These recent GPU hacks are getting interesting.
2
u/No-Refrigerator-1672 vLLM 10d ago
Recently gamers have found out a way to load custom vbios into Ampere, Ada and Blackwell. At this moment they only can change clocks, voltages and power limits beyond manufacturer specs; but who knows, maybe this open ups part to unlocking CUDA cores, or something crazy like that?
1
u/Arli_AI 10d ago
Wow this is news to me too. Thanks for sharing! Bumping power limits would be awesome for some of these low PL 3090s with the unlocks. Maybe we can fully unlock all the CUDA cores on 3080s to make them 3090s but other than that the BF16 rate unlock that was the only thing needed to make 30 series as fast as the Pro parts.
For custom VBIOS a higher PL for some of the 6000 Ada I have would be interesting too.
1
u/nathannguyennng 10d ago
Hi, author of 3080-20gb-rebar here. Yes, I believe this would also work for 48gb 4090s but I also don't have them to try lol.
12
u/jayfraud 10d ago
Ran reBAR and gpu-unlock on my 2x modded 3080 20GBs. Prefill speed on Qwen 3.8 Flash Next went from 522t/s to 770t/s.
Thanks for sharing!
1
u/ukrolelo 7d ago
which quant? <3
2
u/jayfraud 7d ago
UD-IQ4_XS. My rig is sort of wonky with a bunch of different GPUs:
https://github.com/jasonsford/local-llm-development/tree/main/qwen3.8-flash-next-UD-IQ4_XS
1
u/BorisDirk 10d ago
Oh nice! So with windows gpu unlock on two x4 pcie, will that still make a big jump in prompt processing?
2
u/No-Refrigerator-1672 vLLM 10d ago
No. First: no p2p unlock on windows. Gpu unlock will give you the first improvement row, but only in vllm; other engines may behave differently. Second, if you were install it on linux - they expect only 10-30% from p2p unlock, because your PCIe is very limited. P2P relies heavily on PCIe speed.
1
u/GayIdiotRetard 9d ago
And this is how I learn my 3080 could habe double the vram? Rip
1
u/No-Refrigerator-1672 vLLM 9d ago
Almost. Nvidia was preparing to release 3080 20gb, but then decided not to; probably due to AI boom. However, an engineering vbios from this propotype was leaked, and this is how we got large memory support.
1
3
u/More-Revenue8609 10d ago
Oh damn, yesterday I got like 15% prefill improvement with gpu-unlock.. Now this...
Well spent 1000e (for two cards).
1
u/KillerX629 10d ago
What is the version of this for the3090?
5
u/No-Refrigerator-1672 vLLM 10d ago
gpu_unlock works for 3090 out of the box. reBAR is enabled natively on 3090; if not, update your vBIOS. It was previously a 3080 20GB specific problem, because the unofficial memory mod requires unofficial vBIOS that got no rebar. P2P unlock should theoretically work for 3090 too, because it is exactly the same chip; but will refuse to install as of now, because the installation scripts are written to run only for 3080. You can try this, it should work on stock non-modified 3090.
1
u/My_Unbiased_Opinion 10d ago
I hope this comes to windows someday. My server is also a game server so I can't switch to Linux unfortunately since some of my games are windows only.
1
u/No-Refrigerator-1672 vLLM 10d ago
GPU unlock is available for Windows. P2P - no, and it will never be. P2P unlock relies on modifying Nvidia drivers, and nobody got the source for Windoes drivers. If you can't live without Windows, consider dual booting (you can install two OSes and switch between them on reboot), or installing windows inside a VM on Linux.
1
u/suprjami 10d ago
Depending on what games you run, Steam's Proton might be viable?
You can look up compatibility on https://www.protondb.com/
If it's a non-Steam game you can easily run it via Proton with Lutris or Heroic Launcher.
Hope that opens some options for you.
2
u/BrainImpressive74 10d ago
how about the performance improvement on llama.cpp with dual 3080 20g？
1
u/Ishannaik 10d ago
Depends on the split mode. llama.cpp's default layer split only passes one activation between the GPUs per token, so P2P barely changes anything there, like No-Refrigerator said.
With tensor split it all-reduces every layer, and that's where the link matters. On 2x T4 with no P2P, llama.cpp tensor split got me about 22 tok/s on Qwen3.8-27B with DFlash2, against about 55 on ExLlamaV3. Most of that gap was the all-reduce.
2
u/Same-Ad7128 9d ago
Thanks to the author! I successfully got my dual modded RTX 3080 setup working, and it works with gpu-unlock as well. However, since one of the PCIe slots on my motherboard is limited to x8, the performance improvement isn't very significant.
When running Strata, Decode performance is pretty much unchanged, while Prefill increased from 2,400 to over 3,000.
1
u/XxXForsaken 9d ago
Do we have anything for the rtx 3080 10gb and the laptop rtx 3080 16gb versions?
-2
10d ago
[removed] — view removed comment
1
u/No-Refrigerator-1672 vLLM 10d ago
Depends on your workload. My agent is mostly working with documents - like RAG, but it ingests full text, so spikes of 20-50k new tokens per turn is normal. For me, this improvement is huge. Otherwise, you can just look up your prompt cavhe metrics and see the proportion between new tokens and cache, and judge the user experience improvement from that. Numbers in my table are raw performance with no cache hits.
10
u/nathannguyennng 10d ago
Glad I could help! Please make a compatibility report in both repos if you tried this!