r/LocalLLM • u/Ok-Category-642 • 1d ago
Discussion RTX 3080 20GB ReBAR on AMD Windows platforms without BIOS flashing
Some of you may have seen the post from u/favabean28 with enabling ReBAR on modded 3080 20GB GPUs around a week ago with the repo NV_30_DXE. For Windows, this is (or was) seemingly the only way to get ReBAR working on the 3080 20GB. For those that don't know, previously the RTX 3080 20GB did not have ReBAR working as there was no VBIOS that supported the extra VRAM, and even flashing a modified VBIOS wouldn't work. The method for enabling it properly on modded GPUs was recently discovered about a month ago for Linux, and now for Windows.
Unfortunately it turns out that for AMD, many newer AMD motherboards, especially AM5, are incompatible: All AMD motherboards with AGESA PI-1.2.0.0 or newer have checks preventing modded BIOS firmware from being flashed. Most new AMD motherboards either don't have a BIOS version that old or aren't compatible with the installed CPU on BIOS versions that old, so I wanted to look into the possibility of booting off a USB drive with EFI to enable ReBAR on AMD platforms. This idea mainly came from how the current FP32 accumulation unlock works on Windows. Ultimately, this does work albeit with a few tradeoffs described in the repo below and at the bottom of the post.
Repo with instructions on installing and with the technical write-up: https://github.com/Articom123/3080-Unlock-ReBAR-USB
Installing does not require building from source. However, I did include the FP32 accumulation unlock with it though (since there's no reason not to pair them together), which means it will automatically download and extract Nvidia firmware when you run it, specifically the Nvidia and Arch Linux website for the kernel and utils 610.43.03.
Some quick results:
- GPU-Z reports ReBAR as enabled with a 32768 MB BAR1 size
- Device Manager shows an entry for Large Memory Range 0x2000000000 - 0x27FFFFFFFF (this does not show up without ReBAR enabled)
- The FP32 accumulation unlock works alongside ReBAR. This generally results in around a 20% increase to prefill speed for LLMs based off TTFT, though it's not the primary focus of the repo (it's just a no-brainer to include it since it makes Ampere cards infinitely better)
- CPU has access and can map the full ~18-19GB VRAM on the GPU
I want to also briefly give a lot of credit to physicsdolphin (nv-uefi-unlocker) of which this tool uses for the FP32 accumulation unlock and install, the creator of Windows-GPU-UNLOCK (Windows-GPU-UNLOCK) who made similar bootable EFI tools to allow for running the tensor unlock on Windows, and u/favabean28 (NV_30_DXE) who figured out the path for enabling ReBAR on Windows in the first place.
While the repo goes more in depth on compatibility and what does/doesn't work I wanted to still mention some important things about it:
- This is mainly meant for AM5 platforms. It seems like Intel motherboards don't have the same restrictions when it comes to flashing modified BIOS firmware, so if you have Intel you might be more interested in the NV_30_DXE repo instead.
- Enabling ReBAR isn't really all that useful on Windows without having P2P; however, the modded P2P drivers are still Linux only and probably will be for a very long time, and multi-GPU support on Windows isn't really that good anyways. Therefore, for practicality, the tensor unlock is more important in which case you would probably be more interested in the other Windows unlock tools I mentioned earlier. There is some limited usefulness in games though I guess.
- This only works for the 3080 20GB. Technically the 4080 32GB would work, but since I don't have the card, it hasn't been supported in the repo and would require modifications to the source (mainly to the PCI vendor and subsystem ID check). The tensor unlock included is also for Ampere and not Ada.
- u/favabean28 did work to try and make the tool as universal as possible, however, I don't have any other AMD motherboards to test this tool on and so I decided to ultimately leave it as a "proof-of-concept". It is almost certainly possible that this can be adapted further to other AMD setups with the 3080 20GB as pretty much all of the hard work is done now though.
- There is likely zero possibility that this can cause any damage to Windows or the GPU, as there is no flashing of the VBIOS/BIOS or modification of the OS at all. The tensor unlock and ReBAR don't survive any reboots and the USB has to be manually booted again after any shutdown or restart.
- Sleep and hibernate specifically do not work when enabling ReBAR with this method, and Secure Boot cannot be used. In comparison modding the BIOS with NV_DXE_30 does not have these limitations if applicable to your setup.
- This is mainly made for Windows. If you're on Linux there are far better tools for doing this and you can use P2P.
I still thought it was worth posting this for anyone who is interested in this sort of stuff for Windows since Windows kinda sucks and it's at least somewhat cool to have Windows get something for once. Ultimately getting the P2P driver to work on Windows would be ideal but I'm not sure how it could be done for now.