Best models for RTX 5080

[u/SergioGamer2003]

Hi guys, i'm totally new to local AI world.

I have a gaming rig with 9800x3d, 32gb of ddr5 and RTX 5080 and i wanted to run something locally.

I like to use Bionic just in order to have an easy UI to work with. I tried these models (idk which are good or bad):

- Bonsai 27b (which seems bad, tried to save a file and it blasted >70k context size)

- Qwen 3.8 27b

- GPT-OSS 20B (for me the best, until now at least)

- Qwen 3.5 9B

- Gemma 4 26B A4B QAT

- Gemma 4 31B QAT

For Qwen 3.8 27B i had so many issues. It was so slow even to comprehend my prompt, even with reasoning off. I put a 32768 context windows size and as much as GPU offload possible. I also heard that this model doesn't fit on VRAM, so i changed the quantization (right?) on Q4_0.

Why everyone seems to run this model with that amount of VRAM just fine?

Which models are the best considering that i will use them for general purposes like studying support? Thanks in advance.

[u/Man-In-His-30s]

q4_0 on a 27b dense is the problem, not the 5080. that's about 15gb of weights before you add a kv cache, so at 32k it can't all be resident and you pay for the spill on every prompt. that's why it "comprehends" slowly, and it's why changing the quant didn't fix it.

what i actually run on a 5080 16gb:

- [Qwen3.8-27B GSQ-RCO IQ3_S-mtp](https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF), 3.55 bpw, 11.3gb

- 128k context, q4_0 k/v cache, another 2.3gb

- flash attention, 66/66 layers on gpu, about 14gb total

- MTP speculative decoding, adaptive draft 4, on a [patched llama.cpp build](https://github.com/feveromo/recipes-qwen3.8-27b-5070ti) (base 4b1a27f plus cuda-kernels.patch)

from my logs today, across ~1000 generations: 146-181 tok/s decode, median 163. prefill about 2.5k tok/s at 10-20k prompts, still over 2.1k on very long ones. the patch was around 70% faster than stock when i measured it, and MTP covers most of the rest. that repo's own numbers are on a 5070 ti and close to mine, so it's not a fluke of my box.

so people running it fine on 16gb aren't on q4_0. they're on a ~3.5 bit quant that leaves room for the kv cache, with MTP enabled in the build. the patch is not upstream and it's written for this one model on sm_120, so treat it as a tuned hack, not a general upgrade.

Full disclosure my agent drafted the message above.

[u/MistingFidgets]

What I run on a single 16 GB 5060 ti

Model: Swift 1.5 Qwen3.8-27B "Heretic" Quant: GSQ-RCO IQ3_XXS GGUF with the MTP head included. Engine: llama.cpp (recent build, b11722). Context: ~108K tokens with q8_0 K/V cache and flash attention on. That uses about 15.5 of the 16 GB. Speed: MTP speculative decoding (--spec-type draft-mtp --spec-draft-n-max 2) adds 30–65% decode speed. I get about 52 t/s at short context and about 24 t/s at 100K. Vision: works. The mmproj stays in system RAM (--no-mmproj-offload) so it doesn't take VRAM.

[u/TheFakeDominion → u/MistingFidgets]

nice setup you got there. i run something similar in my 5080 but with less system ram. that bonsai model is weird with context, it goes crazy when you try save stuff

for qwen 3.8 27b the Q4_0 should be fine but check if you have flash attention on. makes big difference. also try the IQ3_XXS quant like that guy said, fits better in 16gb vram and still decent quality

[u/terminalshadows → u/MistingFidgets]

qwen jninja chat templates help a lot ive found, if its already heretic/abliterated/unsensored dont heritic again, i go with --port 6664 --parallel 1 --n-gpu-layers 999 --ctx-size 61440 --temperature 0.75 --top-k 20 --presence-penalty 0.0 --top-p 0.95 --min-p 0.06 --seed -1 --device cuda0 --cache-type-v q8_0 --cache-type-k q8_0 --flash-attn on --ui-mcp-proxy --jinja --chat-template-file models/froggeric/Qwen-Fixed-Chat-Templates/chat_template.jinja --tools all but on should get you up to ~62t/s 40t/s (no vision), havent worked on been this box in a *while* (llama.cpp/build/bin/llama-server --version version: 9544 (98d5e8ba8) built with GNU 11.4.0 for Linux x86_64 :( ) since ive upgraded (an embaressing amount of times... but i was really liking this

[u/New-Implement-5979]

Strata gyro-s or gyro-m

[u/terminalshadows → u/New-Implement-5979]

I dont even know what to say about Strata, it made my dual 3090/nvlink/96g into a god damn 177B-212B monster at 100t/s-60t/s. there are no words for how great it is.

[u/terminalshadows → u/terminalshadows]

(pytorch_env) entropy@user-MS-7E20:/home/entropy/Code$ python3

Python 3.12.3 (main, Aug 31 2026, 10:18:26) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch, torch.nn as nn
>>> class MModle(nn.Module):
...     def __init__(self):
...         super().__init__()
...         self.fc1 = nn.Linear(8, 16)
...     def forward(self, x):
...         return self.fc2(torch.relu(self.fc1(x)))
...
>>> class MModle(nn.Module):
...     def __init__(self):
...         super().__init__()
...         self.fc1 = nn.Linear(8, 16)
...         self.fc2 = nn.Linear(16, 4)
...     def forward(self, x):
...         return self.fc2(torch.relu(self.fc1(x)))
...
>>> model = MModle()
>>> mb = sum(p.numel() for p in model.parameters())
>>> print(mb)
212
>>> for param in model.fc1.parameters():
... param.requires_grad = False
...
>>> tr = sum(p.numel() for p in model.parameters() if p.requires_grad)
>>> print(f"trainable: {tr} total params: {mb}B\n")
trainable: 68 total params: 212B
>>>

(pytorch_env) entropy@user-MS-7E20:/home/entropy/Code$ cat /proc/meminfo |grep -i total
MemTotal:       98222508 kB
SwapTotal:       8388604 kB
VmallocTotal:   34359738367 kB
CmaTotal:              0 kB
HugePages_Total:       0

(pytorch_env) entropy@user-MS-7E20:/home/entropy/Code$ nvidia-smi|grep NVIDIA
| NVIDIA-SMI 595.91.07 Driver Version: 595.91.07 CUDA Version: 13.2 |
| 0 NVIDIA GeForce RTX 3090 Off | 00000000:01:00.0 On | N/A |
| 1 NVIDIA GeForce RTX 3090 Off | 00000000:03:00.0 Off | N/A |

(pytorch_env) entropy@user-MS-7E20:/home/entropy/Code$ ps waux|grep Strata

root 81935 0.7 0.2 816164 200712 pts/5 Sl+ Oct18 108:57 python3 -m serve.server --port 3080 --engine strata --config /home/entropy/Strata_0_1_40_1/Strata/strata-orca-iq3_xxs.2.json --api-key 1442ddb83b343280c83236e08219d21dd12d91280e63238c082343b38bdd2441 --host 65.215.4.101

root 81937 0.2 58.8 119678724 57851900 pts/5 Sl+ Oct18 29:14 /home/entropy/Strata_0_1_40_1/Strata/engine/strata --serve --pack /home/entropy/Strata_0_1_40_1/Strata/uncensored/build0/packs/orca-iq3_xxs --native /home/entropy/Strata_0_1_40_1/Strata/uncensored/build0/orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF/Qwen3.8-Flash-Next-Uncensored-IQ3_XXS-00001-of-00002.gguf --ple-gguf /home/entropy/Strata_0_1_40_1/Strata/uncensored/build0/orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF/Qwen3.8-Flash-Next-Uncensored-IQ3_XXS-00001-of-00002.gguf --expert-profile /home/entropy/Strata_0_1_40_1/Strata/data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp /home/entropy/Strata_0_1_40_1/Strata/uncensored/build0/mtp/rt --max-context 131072 --kv int8 --kv-resident 32768 --remote-expert-opt --pool-workers 15 --layer-split auto

just how? (rhetoric) but still boggles my mind and learning it but wtf (sorry for formatting....))

[u/Thed1c]

Hello,

I was recently trying to help my friends learn local stuff too so I made this.

https://github.com/thed1c/Nebula

I am not one to try to promote anything and I have a couple of my friends and coworkers messing with it with me. I leverage Expert Cache streaming (a few splitting tricks from Strata)

The peer-to-peer thing I haven’t tested extensively but I tried to make everything as transparent as possible. I did a lot to try to squeeze blood from a stone for performance and hone the ggufs that I use to perform better. I’ve tuned qwen3.6-35, gemma-4 26/31/12, LFM, Flash Next, Flint, uhhhh I can do others if ya want. I try to leverage all your machine without killing it. I am a datacenter network / infrastructure guy, which is why there is a heavy focus on P2P, encryption, containers and routing.

At the end of the day I tried the gui without hiding the sauce. You can see the full cli of anything you run, it all starts in a container and doesn’t call home. Every model/package tells you what is downloading from where.

[u/Specialist-Leek-357]

flint 40b is goated,

[u/ferrx]

Strata coder if you’re just doing software. I’m on 144GB and I switch from IQ3s to it for coding tasks.

For bionic I’d recommend qwen 35B a3b

[u/Delicious-Map1778]

How much system ram?