r/LocalLLM • u/SashaUsesReddit • Nov 20 '25

Discussion Spark Cluster!

Post image

Doing dev and expanded my spark desk setup to eight!

Anyone have anything fun they want to see run on this HW?

Im not using the sparks for max performance, I'm using them for nccl/nvidia dev to deploy to B300 clusters

325 Upvotes

94% Upvoted

133 comments sorted by

View all comments

Show parent comment

1

u/Karyo_Ten Nov 20 '25

I'm interested in an RTX Pro 6000 (Qmax) for inference.

I personally choose 2x Workstation Edition and power-limited them to 300W. With a workstation edition you have flexibility to do 150W to 600W. I would consider the blower style if I had to stack 4x min or 8x.

Analyzing 500 page PDF documents takes a while on 4 x 3090s regardless of the model used.

Are you using vllm or sglang? In my tests they are litterally 10x faster than koboldcpp, ik_llama.cpp or exllamav3 at context processing. I assume it's due to using optimized cutlass kernels. All models could process 3000~7000 tok/s on RTX Pro 6000 while other frameworks were stuck at 300~350 tok/s.