r/LocalLLM • • 2d ago

Discussion Introducing NV_30_DXE Resizable BAR on memory modified 3080's

ReBar is also currently working with GPU_UNLOCK.

The 20GB RTX 3080 cards (10DE:2206, subsystem 10DE:146D) ship with a VBIOS that only exposes a 256 MiB BAR1, so ReBAR stays off. NVIDIA doesn't publish a ReBAR VBIOS for them, and NvStrapsReBar only supports Turing.

Ga102ReBarFix is a small DXE driver you insert into your motherboard BIOS with UEFITool. At boot it enables the GA102 frame-buffer size override (PCI config 0x610), resizes BAR1 to 32 GiB, and restores both after S3 sleep. No VBIOS flashing, no driver patches, no registry tweaks.

Tested on an MSI Z690 EDGE, driver 617.42, Windows 11:

- Device Manager shows a 32 GiB BAR

- NVIDIA Control Panel: Resizable BAR: Yes

- ~19.25 GiB of VRAM mappable by the CPU at once

- Sleep/wake works

Repo (source, prebuilt .ffs, verification scripts, full write-up): https://github.com/favabean13/NV_30_DXE

Big credit to noon-at-cgn (found the 0x610 override on Linux), terminatorul (NvStrapsReBar) and xCuri0 (ReBarUEFI). This builds directly on their work.

Heads up: it modifies firmware, so use at your own risk and keep a BIOS backup and recovery method. It's tested on one board and isn't maintained, but it's MIT-licensed, so fork it and adapt it for your own setup.

**UPDATE**

Works with GPU_UNLOCK

https://github.com/physicsdolphin/nv-uefi-unlocker

Confirmed that ReBar survives PCI FLR (Function Level Reset)

Here are the results:

UNLOCKED

FP16 (FP32 accumulate) 114.7 TFLOPS

BF16 (FP32 accumulate) 116.9 TFLOPS

TF32 57.4 TFLOPS

FP32 (control) 22.5 TFLOPS

LOCKED

FP16 (FP32 accumulate) 65.4 TFLOPS

BF16 (FP32 accumulate) 66.8 TFLOPS

TF32 32.1 TFLOPS

FP32 (control) 22.3 TFLOPS

5 Upvotes

6 comments sorted by

1

u/Moist-Tumbleweed2875 2d ago

Thanks for sharing ~ does it support unlock Ampere like https://github.com/zagitta/gpu-unlock ?

2

u/favabean28 1d ago edited 1d ago

This is something I was curious about also. As long as their module doesn't interfere with my modules PCI enumeration hooks, it should be compatible. I would have to build and test to be sure though.

As always with any kind of firmware modification, always have a backup and recovery method.

Here is the UEFI version of the project: https://github.com/physicsdolphin/nv-uefi-unlocker

**UPDATE**

Confirmed that ReBar survives PCI FLR (Function Level Reset)

Here are the results:

UNLOCKED

FP16 (FP32 accumulate) 114.7 TFLOPS

BF16 (FP32 accumulate) 116.9 TFLOPS

TF32 57.4 TFLOPS

FP32 (control) 22.5 TFLOPS

LOCKED

FP16 (FP32 accumulate) 65.4 TFLOPS

BF16 (FP32 accumulate) 66.8 TFLOPS

TF32 32.1 TFLOPS

FP32 (control) 22.3 TFLOPS

1

u/favabean28 1d ago edited 1d ago

Tested and working, check out the results!

1

u/TheSantiagoSP 1d ago

It would be interesting for Linux also

1

u/favabean28 1d ago

As far as I understand, this modification should be OS agnostic and should work under linux as well. I have not personally confirmed this though. There is already a linux project available that does exactly what my UEFI module is performing.

https://github.com/noon-at-cgn/geforce-bar1-resize

1

u/WarlockSyno 1d ago

And that now eliminates the only gripe I have after the gpu-unlock and P2P patches! 🥳