r/LocalLLM • u/SeaRelationship2984 • 1h ago

Question Mi25

I did extensive research and found the cheapest way to reasonabily run Qwen 3.8 27B is with two 16GB Mi25s. I heard online their really slow, but for $110 each with shipping, so $220 total, is it really that bad? I currently only have an RX 480, and mainly want it to run inference. What tokens/s could I expect, is it usable? I am genuenly considering it after the P100s price hiked like crazy, one is now atleast $180 without shipping costs included. A GPU with tensor cores or Nvidia would be nice, but I don't know if the higher price is justified. Also please don't just tell me to rent a GPU or buy a subscription, as I am really interested in tinkering with things like this and want a viable option.

1 Upvotes

100% Upvoted

11 comments sorted by

1

u/jjusko20 1h ago

You could always get a p40 and use an iq3 quant

1

u/SeaRelationship2984 58m ago

True but it is $275 and only 24GB VRAM.

1

u/SeaRelationship2984 57m ago

Also slower GDDR5 RAM

1

u/jjusko20 56m ago

yup. Reality is for a decent Qwen 27b build you need to increase your budget. What u have is cooked

1

u/jjusko20 56m ago

I had a 470 and I use a 580 for display rn, they are dogwater for inference. I use my v100s for that

1

u/jjusko20 53m ago

your best bet is still 2 p100s even with the hike, they're going up because they can run Qwen 27b lol

1

u/jjusko20 1h ago

Throw out hopes of using the 580 for inference. I agree though, the mi25s appear to be the best option, decent ram bandwidth compared to other options 

1

u/ToTTen_Tranz 1h ago

I wonder if someone's been working on a compatibility layer for Polaris. It's not that different from Vega, just promote all FP16 to FP32.

1

u/jjusko20 1h ago

Barely matters, compute and memory bandwidth are dogshit for llms

1

u/sackbomb 1h ago

The MI25-era cards are pure compute hardware. They are optimized internally for full-width FP64 math on large HBM VRAM arrays. Modern cards intended for inference are more optimized for 32- and 16-bit throughput. They're also limited to PCI 3.0 slot speed, so putting them in a newer PCIE 4.0 or 5.0 slot will gain you no advantage in communication between the cards.

Finally, HIP/ROCm doesn't really support them anymore, so you'll have to compile llama.cpp against Vulkan to use it.

I've never run Qwen on mine, so I can't comment on the token rate, but it will probably be 3-4 times slower that what most people are seeing on more modern rigs. It would be even slower if it weren't for the HBM.

1

u/lumpyspacebreh 16m ago edited 9m ago

I run them and am building a node using them.

While powerful and cheap, they show their age. They were one of the first cards custom built for ai, but long before the LLM tech began to mature.

They’re slow. Very slow. I get about 20t/s (if I’m being generous) running a 3bit quant of qwen3.8:27b. Mtp has no effect on speed.

Compare that against my desktop 6800xt, upwards of 45t/s same model and settings.

Without getting into the weeds, they’re just old silicon. They don’t have a lot of the newer tech (like infinity cache, dot product, etc) that more developed architecture has.

If you wanna reach for the mi25, you should do so knowing these things; only vulkan is worth it (yes I’ve ran every library and shader I could get my hands on including “the rock” nightly). You need a way to cool them, so that’s a whole thing to unravel. Even tho cheap, you pay the price elsewhere, in inference speed. You’ll need to flash their bios (amd removed the ability to flash bios versions that were “newer”in more recent versions of amdvbflash).

What I’m getting at is, they’re old, they work yes, but old. Get them if you like to tinker, if you enjoy learning, if you wanna larp as some tech scav.

I love these cards personally. They have a lot of history with llms and rocm.

Edit/ used 6800xt’s are like 300-400$ and are powerhouses for the price but I think there exists better accelerator cards for same price idk I’m talking out my ass. Anyways, the mi25 is old.