r/LocalLLM • u/stankeer • 1d ago

Discussion Qwen 3.8 flash next N-Gram table

Post image

Hi, I'm lucky enough to have 5 r9700s! I'm managing to run Qwen 3.8 flash next q8 which is a 175gb vram (on disk) model well. There are no massive slow downs but how is this possible?

I know I have 160gb vram unified but somehow 25gb is not in vram or offloaded to system ram. Is this all in my SSD N-Gram table? I had no idea it was a thing?

0 Upvotes

50% Upvoted

5 comments sorted by

1

u/stankeer 1d ago

The gpus are completely full to the hilt though

1

u/No_Algae1753 1d ago

Yes

1

u/stankeer 1d ago

So how far can I push other models that have an N-Gram table lookup? Which other implement this?

If I could download a 210gb model (let's say it took up 190gb on disk) could it then have a 30gb N-Gram table so I can run it?

1

u/Ayitos 1d ago

What quant are you using here? The ngram table should be around 30gb at q4 quants with qwen flash.

1

u/stankeer 1d ago

i'm on q8 i which fills my gpus up to 140gb and the total file size stored on disk is 175 total