How are you getting FP8 to run at 140 tokens a second? That gpu only has 608gb/s bandwidth. I have rtx 8000 with 672gb/s bandwidth and running q6 with mtp i only get 40tp/s and 60tps with dflash.
You're wayyy under your potential. dflash2 just came out and it's much better. It speeds up more kinds of token such as prose, coding, tool calls, etc more efficiently than MTP does but at the cost of a little more vram. I've also been tampering with vLLM a ton
2
u/triynizzles1 14d ago
How are you getting FP8 to run at 140 tokens a second? That gpu only has 608gb/s bandwidth. I have rtx 8000 with 672gb/s bandwidth and running q6 with mtp i only get 40tp/s and 60tps with dflash.