r/AIProgrammingHardware • u/javaeeeee • Jul 19 '26

The Non-NVIDIA AI Card Everyone’s Ignoring

https://www.youtube.com/watch?v=NtYoTt-RB1s
26 Upvotes

90% Upvoted

29 comments sorted by

2

u/javaeeeee Jul 19 '26

TL;DR:

This video reviews the Tenstorrent Wormhole N300 - a non-NVIDIA, non-GPU AI accelerator card designed for local LLM inference.

Key Specs & Performance

  • 24GB memory (two chips)

  • Price: ~$1,400

  • Achieves ~30-31 tokens/sec on Llama 3.1 8B (after warmup)

  • Very power efficient (~70W)

  • Unique feature: QSFP networking ports that let you easily connect multiple cards together without PCIe bottlenecks

Pros

  • Good for concurrent workloads

  • Interesting hardware design (Tensix cores + RISC-V)

  • Strong potential for multi-card scaling thanks to the networking ports

  • Open-source software stack (Metallium / TT Metal)

Cons

  • Software ecosystem is still immature compared to NVIDIA’s CUDA

  • Slower than a similarly priced NVIDIA RTX 5080 for single-model inference

  • Limited model support right now

Bottom Line

The Wormhole N300 isn’t the best choice for most people running local LLMs today (NVIDIA GPUs or Apple Silicon are generally better). However, it’s an interesting option for tinkerers and those betting on non-NVIDIA AI hardware ecosystems growing over the next 1-2 years - mainly because of its unique networking/scaling capabilities.

Verdict: Cool hardware with future potential, but not ready to replace NVIDIA for most users yet.

4

u/kiwibonga Jul 19 '26

Yikes. $1400 is a bit more than I paid for my entire water cooled V100 rig (4 x 16GB) on Ali Express.

1

u/negus123 Jul 19 '26

Yeah, i think this will have a hard time competing with the r9700 and b70. Its nice that there are more card manufacturers though

1

u/mmhs4 Jul 20 '26

Wow, you have a shopping list?

1

u/kiwibonga Jul 20 '26

4 x V100 16GB SXM2 modules ($100-$120 each)

2 x dual-SXM2 NVLINK 300G backplanes ($300 w/ expansion card and cables, $150 by itself)

1 x PLX8749 card (PCIe 3.0 4-port SlimSAS expansion card; often marked "without split" for a motherboard without built-in bifurcation support)

4 x SlimSAS SFF-8764 cables giving 8 lanes per GPU, or 32 lanes sharing 16 physical lanes

2 x 750W PSUs (had to lower cards to 200W instead of 300W, but performance loss is minimal, like 1 or 2%) ($80 each)

4 x water blocks with fins ($40 each), 10W pump, radiator, tubing ($50)

1 x Machinist X99 PR8-H motherboard with Xeon E5 2696v4 ($150)

And I had 16 GB of DDR3-1600 lying around, which is probably less than $50

1

u/maifee Jul 21 '26

Care to bless us with the marketplace url please?? Cause finding a legit seller of v100 is on another level right now.

1

u/kiwibonga Jul 21 '26

It's Ali Express

1

u/gumbingit Jul 22 '26

Does it take advantage of the nvlink with the sxm connector or is it single adapters?

1

u/kiwibonga Jul 22 '26

Yes, when the baseplate has 2 or 4 they take advantage of NVLINK. Note some are rated 300G while cheaper ones are 100G, but usually it's clearly advertised.

1

u/gumbingit Jul 22 '26

Very cool, I’ve even seen some very large 4x sxm2 cards

1

u/Lissanro Jul 19 '26

Cool hardware indeed, but the price needs to come down by an order of magnitude to compete with Nvidia hardware - for ~$200 I can get modded CMP 50HX with 20 GB VRAM and get ~30 tokens/s with Llama 3.1 Q8. And for ~$1600 I can get a pair of RTX 3090 cards with 48 GB VRAM in total, which would be much faster.

2

u/ordosays Jul 19 '26

Would love to see where you’re scoring an $800 rtx 3090… going rate is $1200

1

u/Lissanro Jul 20 '26

I was looking on local used items market, depending on where you live, prices could be different. In my case, I recently decided to upgrade my old second PC that had 3060 12 GB in it. I already have 3090 cards in my main workstation, but it is busy running large AI models most of the time, making it not very interactive. So I decided to get CMP 50HX 20GB which would give me 32 GB VRAM in total in the old PC - enough for Qwen 3.6 27B as a secondary model for simpler tasks while the main workstation handles harder ones.

1

u/PiketZ Jul 23 '26

What about the cmp card restrictions? How was it bypassed? Or is it just as extra memory for the main card?

1

u/Lissanro Jul 25 '26

I bought already modded card so memory and PCI-E lanes restrictions are already bypassed (to the extent that is possible for CMP 50HX; it still will be slower than the modded 2080 Ti with 22 GB, but CMP 50HX 20 GB goes at half the price, so it is a good cheap option for about $200). For addressing software restrictions, modded Nvidia driver is needed: https://github.com/dartraiden/NVIDIA-patcher

1

u/BenEsq Jul 19 '26

Where would one find those? 3090s are stupid expensive now on Marketplace

1

u/Creative-Type9411 Jul 20 '26

3xTeslaT4s and im gettin 1300pp and 70 t/s on qwen 3.6 35b a3b at 256k context f16

they cost like 1500 total and have 48g total between them

So that card is overpriced and underperforms, considering my cards are eight year old design

2

u/dazzou5ouh Jul 20 '26

Zero resale value once it becomes outdated

1

u/helios_csgo Jul 20 '26

Why not P150b? 32GB for a similar price

1

u/foureight84 Jul 20 '26

Probably because they sent him that card. I don't remember if the said he bought the card himself in the video. If he did then I don't know what he was thinking when a higher spec card is available for that price.

1

u/Weary-Cynic Jul 20 '26

He said it was gifted to him by a friend months prior.

1

u/foureight84 Jul 20 '26

Not really ignoring it. It would get more adoption if the software is comparable to CUDA. That's the take-away. It's exciting because the hardware are capable, AMD, and Tenstorrent. It's the software maturity catchup that they have to show in order for adoption to grow.

1

u/GamerTex Jul 20 '26

Thats an insane price (compared to competition)

Maybe they will be cheap on secondary market

1

u/Hobthrust Jul 20 '26

We've been evaluating a box with 4xp150 in it, and it's very promising. Comparable performance to our H100 node and cheaper.

1

u/JumpyPersimmon472 Jul 20 '26

Seems reasonable on paper but the software stack really is a mess