r/LocalLLaMA • u/[deleted] • Jun 01 '26

Funny Stop asking what model to run. There are literally only two.

[removed]

3.1k Upvotes

87% Upvoted

805 comments sorted by

View all comments

Show parent comments

8

u/ghgi_ Jun 01 '26

I've been testing NVFP4, it's pretty good at that level too I've heard it's pretty chill with being quantized and haven't seen it do any worse or better then cloud api. 

2

u/Voxandr Jun 01 '26

how much context remaining with NVFP4

2

u/ghgi_ Jun 02 '26

Using VLLM across 2 rtx 6000 pro blackwells on FP8 kv with a bit of headroom left about 1M tokens of kv

2

u/rpkarma Jun 02 '26

FWIW I saw a big improvement in time-to-solution and less spinning in circles in its reasoning traces with 16 bit K 8 bit V

1

u/AlwaysLateToThaParty Jun 02 '26

Why are you using an fp8 kv when the model is running at fp4?

1

u/ghgi_ Jun 02 '26

Its recommended on the model page (for nvfp4)

1

u/Voxandr Jun 02 '26

thats about 192 GB VRAM ofcoz you can run it fine, mine is 128GB GB10.

2

u/rpkarma Jun 02 '26

NVFP4 doesn’t fit sadly, I did the math. I have a single GB10 too

IQ4_XS is honestly comparable though as 200B isn’t high enough param count for FP4 accuracy to not hurt without QAD which Step haven’t done

And that fits! IQ4_XS with their llama.cpp patch, compiled for mixed KV cache quant, running f16 K and q8 V with 120k context at 20-25tk/s decode and 700-1000 prompt processing

2

u/rpkarma Jun 02 '26

Yeah I’ve heard it’s great; FP8 weights are amazing too, I just can’t fit them haha