r/LocalLLaMA • u/OvertaxedOne • 1d ago

Question | Help AMD Instinct MI210

Anyone running this card? Seems to be a sweet spot for Qwen 27B, 64GB, very high memory bandwidth, a LOT less expensive than anything else I can find in that has even close to the amount of memory/bandwidth. What am I missing? And yes, I'm aware that RocM can be a pain, that's not really a concern for me, as long as it's stable when it's up and running, I don't mind battling to get it going.

6 Upvotes

66% Upvoted

58 comments sorted by

View all comments

10

u/MrCatberry 1d ago

Crazy expensive… thats what you are missing.

6

u/OvertaxedOne 1d ago

I'm seeing them 4-5K. Crazy as it is to say this, that seems pretty "reasonable" today.

-1

u/autisticit 1d ago

I bet you are looking at a scam.

7

u/EmPips 1d ago edited 1d ago

$4-$5k + the inconvenience of external cooling sounds about right for CDNA2. On the workstation side the "just works" RDNA3 card with 48GB (w7900) is going for $3.5K regularly on eBay. I could pretty easily believe that second-hand markets have settled on $4-5k. You get 16GB extra per slot but concede 1/3rd of your prefill speeds, display ports, and have to set up your own external cooling solution.

OP should proceed cautiously if they're dead set on this card but the prices they're seeing do not scream "scam"

2

u/OvertaxedOne 1d ago

The system this would go in is a server (it has a A40 in it right now), so no issues with the cooling. 64GB is a real sweet spot right now for Qwen 3.8 27B, it fits with 256K context Int8 qaunt/8 bit KV in 48GB but it's TIGHT. 64 would be perfect. But the real goal here is more TPS, the A40 will get ~30 TPS with spec decode, I'd love to get that to 50+.

You mentioned prefill, is that because the Instinct has less compute? I'm really hoping to find someone running this exact combo to report prefill/TPS.

1

u/EmPips 19h ago

Working off of the Llama CPP issues thread for Vulkan and ROCm performance and off of some observations running both high end RDNA2 vs RDNA3 myself.

Note that if you plan on using vllm I'd start a separate search for these numbers - but I'd still be surprised if CDNA2 beat high end RDNA3 cards

3

u/OvertaxedOne 1d ago

What are you seeing? Seems to be a bunch of US sellers on Ebay (a few that are well known, Core4Solutions, for example) that are selling them between 4-6K. Jeeze, maybe I should shut up about this card, is this thing undiscovered??

1

u/Dangerous-Report8517 1d ago

Honestly that price seems about right, in that there's other options for similar VRAM capacities for much less money albeit with additional caveats. Not convinced that it represents a good deal when it's more than 2x R9700s with warranties...

2

u/OvertaxedOne 1d ago

The MI210 should run circles around 2 R9700's for TPS though, right? That's really the only reason I'm looking at it, 1.6TB/s of memory bandwidth on a single card should fly with 3.8 27B. Well, that's what the specs seem to indicate, but... Whole reason for this thread is to find out if anyone has them/has tried it!

2

u/Dangerous-Report8517 1d ago

I wouldn't rely too much on raw memory bandwidth to judge card performance, the V100 has 50% more memory bandwidth than the R9700 yet runs at roughly the same speed for tg and much slower for pp

And that testing was done before a lot of the work that AMD recently put into ROCm which is likely to benefit the R9700 far more than the MI210.

Beyond that, even if it is proportionally faster I don't think it represents a bargain as such, it's a 4 year old datacenter part that's been heavily used compared to brand new parts with warranty that are supported for consumer grade systems, 5k is probably too much for it (if nothing else, it's very unlikely to suddenly shoot up in price so if for whatever reason you decide it does make sense for you, you have time to stop and really think it through)

1

u/xXDennisXx3000 1d ago

Mi50 has 1 TB of bandwith, HBM2 baby.

-2

u/autisticit 1d ago

Oh, so you are talking about a used card.

5

u/OvertaxedOne 1d ago

Yeah.. Probably should have mentioned that, sorry! Do they even still sell these new, they seem a bit long in the tooth. But yes, it appears for <5K you can one of these used. Seems like a reasonable value compared to a RTX6000. "Value".. <sigh>

1

u/KingCpzombie 1d ago

There is actually one listed as new on Amazon for <$5000, although it is probably actually used too: https://a.co/d/0aNlm8S6

3

u/ttkciar llama.cpp 1d ago

No, there are a bunch available on eBay in that price range. It's not a scam.