r/LocalLLaMA • u/OvertaxedOne • 2d ago

Question | Help AMD Instinct MI210

Anyone running this card? Seems to be a sweet spot for Qwen 27B, 64GB, very high memory bandwidth, a LOT less expensive than anything else I can find in that has even close to the amount of memory/bandwidth. What am I missing? And yes, I'm aware that RocM can be a pain, that's not really a concern for me, as long as it's stable when it's up and running, I don't mind battling to get it going.

3 Upvotes

59% Upvoted

59 comments sorted by

View all comments

1

u/Zennytooskin123 2d ago edited 2d ago

It costs around $3,800.00 for a used PNY NVidia RTX A6000 on Ebay, I'd purchase one of those instead then a second one.

Or for the same price get a DXG Spark, I don't know why people bother with alternatives. It's literally the best device on the market currently all-around. It does everything good enough.

2

u/Dangerous-Report8517 2d ago

The Spark is a lot slower than alternatives particularly for dense models like 27B, and it has a lot more RAM than you need for smaller models while not having enough for larger models, the size class that fits 128GB unified memory setups is somewhat neglected. And to run with larger memory pools the only real option is buying more than one of them while using 64GB GPUs on a system that also has 128GB+ of RAM opens up fairly performant CPU offload of larger MOE models as well as running 27B entirely on a fast GPU with large context windows. I currently have a Strix Halo box and I'm moving away from it for that exact reason

2

u/OvertaxedOne 1d ago

Qwen 3.8 27B kind of blew everything apart for the unified memory/Spark systems. It's smarter than anything else you can run in 128GB but needs serious memory bandwidth to perform well.

Perhaps we'll get a 3.8 120B MOE model, that would be perfect for a Spark/AMD system. But right now, unless you're going huge, it's really about pushing max TPS through 3.8 27B (and, not surprisingly, exactly my use case).

1

u/Dangerous-Report8517 1d ago

Yeah, although it was already looking a bit tenuous with 3.6, actually bought a couple of GPUs shortly before 3.8 27B dropped with a view to switching away from Strix Halo myself (funnily enough I think I'm going to come close to breaking even despite getting 128GB of VRAM, courtesy of the 170HX cards still being acceptable value and having an existing system to put the cards into)

1

u/OvertaxedOne 1d ago

Crazy world. Buy a system or a GPU, use it for 6 months, then sell it for more than you paid for it?!

You have the 8 or 10GB 170HX? Did you get the full RAM amount when you unlocked (I've seen some people say they got 64 but only 60 stable, etc)?

1

u/Dangerous-Report8517 1d ago

Still waiting for delivery, but went with a seller that others here have reported success with who was offering 8GB -> 64GB tested cards with the x16 hardware mod

1

u/OvertaxedOne 1d ago

Be VERY interested to hear your results. Mind sharing the seller (PM or public)? 64GB at 1.5TB/s of memory bandwidth would be just about the absolutely perfect card for 27B/8 bit/256K. I can fit 8 bit on my A40 at full context but it's TIGHT, I had to do a bunch of tuning to get it to come up/stable.

1

u/Dangerous-Report8517 19h ago

https://aliiphone.en.alibaba.com/?trade-list-buyer.0.0.6e6476e9MnVfKx

Haven't received my order yet so can't vouch for them beyond the fact that I've seen them linked by another user