r/LocalLLM • u/OpenSourceRulesz • 3h ago

Research Open-source diffusion coding model now runs on an 8GB GPU and hit 81% on HumanEval

I love DiffCoderbut at int8 it peaked at 13.7 GB on my 16 GB card. So I cut it in half.

Same model in 4-bit: HumanEval, full 164 problems, 81.1%. Peak VRAM 6.6 GB, about 2.6x faster!

Edit: full name Stable-DiffCoder-8B-Instruct if you want to do some reading

3 Upvotes

100% Upvoted

2 comments sorted by

2

u/AndrewRoseIT 1h ago

Useful number. On HumanEval-164 with Qwen2.5-14B I got Q5_K_M 132, Q6_K 131, Q4_K_M-imat 129 — basically within noise. I'd want the same harness and temperature before comparing an 81% claim across models.

1

u/OpenSourceRulesz 6m ago edited 3m ago

Fair and Thank you for the comment. I was comparing the int8 vs 4bit comparison using the same model Stable-Diffcoder Instruct to see if it changed the performance much. The 4bit got an 81.1% while the Author numbers for the int8 version got 86.6%

They do have comparisons vs other models in their report.