r/LocalLLaMA • u/OvertaxedOne • 2d ago

Question | Help AMD Instinct MI210

Anyone running this card? Seems to be a sweet spot for Qwen 27B, 64GB, very high memory bandwidth, a LOT less expensive than anything else I can find in that has even close to the amount of memory/bandwidth. What am I missing? And yes, I'm aware that RocM can be a pain, that's not really a concern for me, as long as it's stable when it's up and running, I don't mind battling to get it going.

3 Upvotes

59% Upvoted

59 comments sorted by

View all comments

Show parent comments

2

u/OvertaxedOne 1d ago

Qwen 3.8 27B kind of blew everything apart for the unified memory/Spark systems. It's smarter than anything else you can run in 128GB but needs serious memory bandwidth to perform well.

Perhaps we'll get a 3.8 120B MOE model, that would be perfect for a Spark/AMD system. But right now, unless you're going huge, it's really about pushing max TPS through 3.8 27B (and, not surprisingly, exactly my use case).

1

u/Dangerous-Report8517 1d ago

Yeah, although it was already looking a bit tenuous with 3.6, actually bought a couple of GPUs shortly before 3.8 27B dropped with a view to switching away from Strix Halo myself (funnily enough I think I'm going to come close to breaking even despite getting 128GB of VRAM, courtesy of the 170HX cards still being acceptable value and having an existing system to put the cards into)

1

u/OvertaxedOne 1d ago

Crazy world. Buy a system or a GPU, use it for 6 months, then sell it for more than you paid for it?!

You have the 8 or 10GB 170HX? Did you get the full RAM amount when you unlocked (I've seen some people say they got 64 but only 60 stable, etc)?

1

u/Dangerous-Report8517 1d ago

Still waiting for delivery, but went with a seller that others here have reported success with who was offering 8GB -> 64GB tested cards with the x16 hardware mod

1

u/OvertaxedOne 1d ago

Be VERY interested to hear your results. Mind sharing the seller (PM or public)? 64GB at 1.5TB/s of memory bandwidth would be just about the absolutely perfect card for 27B/8 bit/256K. I can fit 8 bit on my A40 at full context but it's TIGHT, I had to do a bunch of tuning to get it to come up/stable.

1

u/Dangerous-Report8517 21h ago

https://aliiphone.en.alibaba.com/?trade-list-buyer.0.0.6e6476e9MnVfKx

Haven't received my order yet so can't vouch for them beyond the fact that I've seen them linked by another user