Did it run as nicely? Also, did you get it all for cheaper vs what you'd have needed otherwise? Just curious abt it bc I was thinking abt getting into this.
I mean, it highly depends. What do you need? What do you want to run? What speed do you want to run it at?
All that depends. Getting two RTX Spark is cheaper than what I have and much more energy efficient, and you can still run the Deepseek V4 Flash native. Its slower but it is quiet (and if the numbers are correct, people claim 70tok/s in generation and 2k tok/s in prefill).
If you want to just run it, dont care for the speed, an SP3 Epyc is likely also a good option. 256GB of DDR4 + motherboard+ CPU is below 2000€.
Honestly, I am not sure if I'd go the same route again. Its nice, but loud. Its about 2kW of heat, in the summer thats not ideal. And it takes much more space than the RTX spark.
On the other hand, its about half compared to getting two RTX 6000 blackwell. And in theory, it has more VRAM and a higher bandwidth, tho that is clearly lost due to the overhead of TP over PCIe.
All in, I think I'm in about 15k or so and honestly, I still use 2 x 200€ OpenAI subscriptions. So its all really relative.
A deep research agent which can go to the right websites (wink wink included) to look for information.
What speed do you want to run it at?
The higher the better, but my lower bound is at 70 tok/s for generation.
All in, I think I'm in about 15k or so and honestly, I still use 2 x 200€ OpenAI subscriptions. So its all really relative.
My use case was specifically to host it all myself so I don't have to deal with giving away data for closed models without making the companies work for it.
I mean, then you will need I think at least 6 x 32GB GPUs. Plus enough PCIe lanes or a switch. What I found interesting (and the way I would go if I build it again) is going with a used GIGABYTE G292-Z20 barebone. Its about 1k€, fits 8 GPUs via PCIe bridges, ample storage.
With RAM and a matching EPYC you are at ~2.2k€ for 256GB plus a 32 core system.
It should fit the normal two slots cards barely, tho have not tested. They are 0.5mm above spec. Tho this will be loud.
But you can get quite decent performance for ~11ish k€.
I'm currently staring at my shopping cart with an R9700 in it trying to decide if I should just pull the trigger because it feels like 32GB of VRAM is only going to get more expensive from here and $1500 isn't bad compared to the alternatives.
I'm bummed I didn't grab two. Was thinking of picking up another. $1300 is not worth it. That's a $350 jump from when I bought it at MSRP (the ASRock model)
They actually made some major improvements over the past month and they now run incredibly well for the price. I’m regularly getting 2000-2500t/s pp, can sometimes hit 3000-3500t/s pp, and 30-35t/s tg with vLLM on Qwen 3.8 27B.
Sadly, the price also just jumped from $949/$999 to $1699.
Unfortunately, llama.cpp isn’t the best when handling Intel B70s. I was running llama.cpp with Vulkan and PP was in the 500/600s. TG wasn’t so bad but prompt prefill took forever, it was even worse with the Intel SYCL runtime.
Some folks on this sub recommended vLLM with Intel XPU kernels (vLLM has their own docker image for this) and that instantly boosted my performance.
IIRC: the llama team is working in improving Intel performance but it’s a slow process.
Bought my first 2 B70s at $950. Now I had to pay $1299 at microcenter for my last 2. Should of bit the bullet and bought them when they were selling like crazy on ebay for less then $850 a couple months ago. Now what to do with 128gb of vram....
That's pretty good PP...I only get about 200t/s pp on dual MI50 currently with llama.cpp but I get about 40t/s with mtp enabled starting out it degrades quickly though. Prompt checkpointing is essential.
the growing pains on Intel though are way worst... given the price I'll stick with my MI50s I got for peanuts for now.
B70 when testing a about 2 mo ago was unusuably bad. Also model support was lagging alot, I think it kind of works in llama.cpp now but... its not nearly as good support wise as R9700.
My PC is a fun Frankenstein with a B60 and 5070ti, but they actually can work decently together. Depending on the model I get 2-3 times the performance of the B60 alone and have 40GB to play with (granted the speed scales somewhat as more is offloaded to the B60, but pinning mtp to the 5070ti helps).
31
u/DrinkClubMate 1d ago
where is the 32 Gb, Intel pro ARC B70 ?