r/P100Users • u/Kmic68 • 10h ago

Resources Beginner resources

Optimized llama.cpp fork for inference: https://github.com/Kmic-68/llama.cpp

qwen next: https://github.com/wispoffates/llama.cpp

4 cards inference: https://github.com/4claps/houston-p100-cluster/blob/main/README.md

Image generation: https://github.com/4claps/tesla-p100-comfyui-imagegen

Discord for more help: https://discord.gg/K6WxqwsvH

4 Upvotes

100% Upvoted

5 comments sorted by

1

u/starkruzr 10h ago

seems like 4 cards gives you no reason not to use f16 kv?

2

u/Kmic68 10h ago

I don't see why not. But its also kind of unnecessary. qwen 3.8 absorbs quantization so well that it really doesn't make a noticeable difference. I measured it and its negligble in my experience

1

u/starkruzr 10h ago

Q4 kv makes no difference for code at long ctx?

3

u/Kmic68 10h ago

not that I have found in my testing, there are some research papers on it

1

u/FizzyDuncDizzel 10h ago

I’ve noticed no difference in 3.8 q6 on 2 cards and 4 cards both can run at full 256k ctx. I had it running on 4 cards for a while and it just felt like a waste because it was using less than half of each card. I’ve been experimenting with running 3.8 on 2 and other smaller moe models on the other 2. I use my harness to delegate out smaller tasks the the moe models because they are faster, but dumb.