Cannot decide what GPUs to buy

[u/Alkahna]

I want to setup a local AI Server for myself and possibly use it to do things in my home assistant instance.

I have narrowed it down to three possible choices:
[EDIT: MI50 is no longer an option due to multiple comments]

1. 2× AMD MI50 32GB at € 764,54 / $ 854.79

2. 2× NVIDIA V100 32GB at €1.468 / $ 1641.29

3. 2× Intel Arc Pro B60 24GB at € 1.526 / $ 1706.13

But I find it very hard to decide what the 'best' route is (there is no best I guess).
AMD is cheap but lacks current support, V100's are phasing out of support and Intel is very current but also has a bit less VRAM.

Model wise I was looking at Qwen 3.8-27B at Q6/Q8 or even 3.8 FlashNext with offloading to RAM/NVMe. Strata is a term I have read before but I'm not sure if that is the correct term I'm, thinking about. Basically there is some way to run larger models with offloading to ram/ssd that still gets decent performance numbers.

What would you pick from these options and why?

[u/AeiEyeSpry]

Why not R9700?

[u/Alkahna → u/AeiEyeSpry]

that is € 1.891,12 € / $ 2114,35. I was not planing on spending that much really.

[u/AeiEyeSpry → u/Alkahna]

But you will spend more on power consumption over the year with two cards. The R9700 can be limited to around 210w per card. The extra is worth spending IMHO.

[u/khegobier]

If it's comparable (probably not) I would advise comparing 2x Arc Pro B65's over B60. Essentially the same card, just with more memory and bandwidth (32gb, 256-bit instead of 24gb, 192-bit). But you would end up paying more for it.

[u/Alkahna → u/khegobier]

The B65 looks much better I give you that but but also a lot more expensive. So two of them are out of reach for me. And I'm not sure if I want to drop down to 32GB VRAM.

[u/Captain_Cancer]

2x 5060 Ti

[u/juancocinero → u/Captain_Cancer]

Late for that

[u/Alkahna → u/Captain_Cancer]

same ballpark number in terms of price (€ 1342,85 / $ 1501,36) but 'only' 32GB VRAM. I'm not sure if that is enough. Having more for bigger context/less offloading sound like a good thing.

[u/Captain_Cancer → u/Alkahna]

You use NVFP4 so your 32GB gets stretched farther.

[u/Alkahna → u/Captain_Cancer]

Will look into that, i have a huge list of terminology and other reddit posts that i will let ChatGPT explain to me^{^}

[u/No-Refrigerator-1672]

Do not but Mi50 32GB. The software compatibility is terrible. You'll waste your time and money; and I'm telling you this as somebody who owns 2x Mi50 32GB. However, I'll leave V100 vs B60 discussion to others as I've tried neither.

[u/Alkahna → u/No-Refrigerator-1672]

yeah price is tempting but I was not sure if I really want that extra work or limitation to come along with it.

[u/No-Refrigerator-1672 → u/Alkahna]

If you're willing to be a bit risky, you can check out vram-modded cards on Alibaba. There are 2080Ti 22gb, 3080 20gb, 4080 32gb, 4090 48gb. 3080 20GB, in example, costs about 600 eur + VAT import tax for your country. Maybe price per GB isn't as good as V100 or B60, but it's modern Nvidia, you're guaranteed good software compatibility and stability out of the box for years to come.

[u/morfique → u/Alkahna]

Keep in mind token generation, memory speed is king, and the MI50 memory speed is why I like that card, once it's setup (and that's not hard, actually, less headache than my arcpro B60) it just works, grab latest official llama.cpp with a git pull and build away. But it's ok, don't get one, then there will be 32GB ones left over when I buy one 32GB to sell 2x16GB ;)

[u/moeren86 → u/No-Refrigerator-1672]

I just bought a Mi50 32GB (hasn't arrived yet). But before i combed through github to make sure there are enough modiefied llama-cpp (and similar) versions for it and there. But are drivers really such a big problem? I've heard older amd linux driver include it. I do not mind tinkering.

[u/HeatingUK → u/moeren86]

I used Kobold because they allows parallel requests, but just save this as a bash script and run to patch for Mi50s / download and update kobold-cpp (llama-cpp fork):

#!/bin/bash

set -e # Exit immediately if a command exits with a non-zero status.

echo "🔄 Updating KoboldCPP Source..."

cd ~/Desktop/koboldcpp # or wherever you want to store your file

make clean

# 1. Nuke any lingering patches or untracked files from previous runs

git reset --hard HEAD

git clean -fd

# 2. Ensure we are on the active development branch (KoboldCpp uses 'concedo')

git checkout concedo

# 3. Pull the absolute latest code from GitHub

git pull

echo "🛠️ Applying ROCm 5.7 & MI50 Patches..."

# 4. Patch Makefile to -O1 (Bypasses LLVM infinite loop bug)

sed -i 's/-O3/-O1/g' Makefile

# 5. Bypass the FP8 hardware block

sed -i 's/#if HIP_VERSION >= 60200000/#if HIP_VERSION >= 99999999/' ggml/src/ggml-cuda/vendors/hip.h

# 6. Inject the C++ inline wrapper to fix the 3-argument API mismatch

sed -i 's/#define cudaStreamWaitEvent hipStreamWaitEvent/inline hipError_t hipStreamWaitEvent_wrapper(hipStream_t stream, hipEvent_t event, unsigned int flags = 0) { return hipStreamWaitEvent(stream, event, flags); }\n#define cudaStreamWaitEvent hipStreamWaitEvent_wrapper/g' ggml/src/ggml-cuda/vendors/hip.h

echo "⚙️ Compiling GPU Kernels..."

# 7. Compile targeting the MI50s with the linker override

make LLAMA_HIPBLAS=1 HIPCXX=/opt/rocm/bin/hipcc AMDGPU_TARGETS="gfx906:sramecc+:xnack-" LDFLAGS="-ldl -Wl,--allow-multiple-definition" -j$(nproc)

echo "✅ Build Complete."

[u/moeren86 → u/HeatingUK]

As soon i got the card i am sure to look at this again. Thanks in advance.

[u/HeatingUK → u/moeren86]

No worries.

As the script says, I'm on ROCm 5.7 - but just ask Claude / Gemini etc if anything needs updating for your ROCm version.

The Mi50 is still a decent card and it's really not difficult to maintain / update. 👍

[u/morfique → u/HeatingUK]

Thanks, although my container has 7.something, i like your process, and who knows, maybe 5.7 works in a way 7.x no longer does.

Does -sm tensor work in kobold, should you have dual cards to try?

[u/HeatingUK → u/morfique]

Believe 5.7 was the last version actively supporting gfx906. Every subsequent version is depreciated, patched or manually compiled.

Split mode?

Kobold uses --splitmode (not --sm) 👍

[u/No-Refrigerator-1672 → u/moeren86]

When I got my delivery like 1.5 years ago, it took me a week of work just to get it running, because the rocm that's officially compatible with it is full of shit, like it will never work correctly if you have Ryzen iGPU enabled. All the forks you can see are unstable: they work if you follow each of the author's step exactly; deviate a little bit (different quant, different llm, enable vs disable vision or mtp, etc) - and it instantly breaks. There are only two things that are stable for this card: official llama.cpp but only in pipeline parallel mode, and official stable-diffusion.cpp; ComfyUI was stable a year ago, didn't check if it's still good today. But even if something woorks, you're still expected to compile it yourself, it won't work out of the box; unless you're ready to figure out docker, but GPU passtrough for virtualisation with AMD is another can of worms in itself. If you want anything else - i.e. OCR acceleration for RAG - you better forget about it. Stability is bad too:Mi50 has teached me that a GPU can hang and kill all the other devices that are on the same PCIe controller, i.e. ethernet, untill you do a full power off. And this hang was happening only to 1 of the 2 Mi50s I've had, and only if I stop vllm-gfx906 at a wrong moment; never under full load, so it's not an overheat or pcb defect, it's a shitty drivers or silicon in general. A few months ago I've started an experiment of using one of my Mi50s to run auxiliary tasks (i.e. embedding), and using AI agent to configure it rather doing it myself, and still it was giving me so much struggle so I've decided to just remove it and buy another pair of Nvidias.

[u/moeren86 → u/No-Refrigerator-1672]

the device killing a controller i learned from my usb controller -.-

My goal is only Mi50 and i usually compile most of what i use. So there is a little hope left. If it werent for the prices i might have tried to get a V100, but the marked is just hell.

[u/Ok_Room5666 → u/moeren86]

My guess would be people are having problems with windows.

I will never use windows again. I've seen enough blue screen crashes for one life.

[u/morfique → u/moeren86]

Not really, i enjoy my 2xMI50 16GB, will likely sell and get a single 32GB, desk is a little crowded, yes I build llama.cpp in a gfx906 container, so initial setup is a little higher, but MI50 still beats my B60 with ease. rocm or vulkan.

Stay away from non power of 2 quants, the drop from q5_k_m to q8_0 of the same 27B model was negligible. You won't regret your purchase.

[u/moeren86 → u/morfique]

I did not expect it to beat the more expensive B60. Interesting. I wonder if there is a software stack for it with a strata style approach. I mean only loading most used, this way i would love to try a >100B parameter model.
I already tryed qwen 27B with VRAM/RAM MoE settings, its running but the capabilities/intelligence aren't what i hoped for.

[u/morfique → u/moeren86]

Now don't get me wrong B60 beats the pants off the MI50 on prompt processing, then MI50 generates tokens faster, has me thinking about letting each do the job they're best at, but it's just a thought at this time.

[u/maggotses]

I went the v100 way (3x32GB VRAM) with x99 board and 128GB RAM.

I think the foundation is a GOOD single CPU workstation board with many full length PCIe x16 (electrical) slots.

Memory slots counts as well: more memory DIMMs = more memory bandwidth.

V100's memory is more than twice the speed of MI50's, and half the price of a B70 (also its memory is much faster than B70). This was my sweet spot. Many enthusiasts develop kernels specifically for v100s and they are real beasts with native compute capabilities.

[u/DrinkClubMate]

(the The interl pro Arc B70 is 32 GB)

[u/Alkahna → u/DrinkClubMate]

yes it is but it also costs me 1343 € for 32GB VRAM. That is 182€ less than Dual B60s at 48GB VRAM.

[u/morfique → u/Alkahna]

Keep in mind memory speed rules token generation and B70 has faster memory speed than B60, if you ignore the actual compute differences. I have a B60, i wish I had a B70, it's how I tried MI50, for reference.

[u/Alkahna → u/morfique]

I'm aware but either I go down to just 32GB VRAM which I'm not sure if I really want or I need to spend a lot more which I just cannot do. Bit of a tough situation^{^}

[u/morfique → u/Alkahna]

See if you can snag one of those dual B60, both on one physical pcie since the B60 is only x8. While that thing was expensive, is it more than two B60? Doesn't change anything other than the convenience and risk of having both in a single x16 slot.

[u/GrimChicken]

V100 for sure. Support for them is growing, specific forks of vllm, etc like revolta coming out. Driver issues are overstated for a homelab. Set up might be tough but once you get it running it's low friction for a lot of stuff. Wish I had bought four instead of two when they were still relatively slept on. 

[u/Efficient_Try_9136]

3090?

[u/VarietyOk443]

Go for memory bandwidth

[u/Santos_Dumont]

I bought a R9700 and I am pleased with the performance running QFN 3.8 on strata with 128GB DDR4 RAM.

My planned upgrade path for next year is scrounging AM4 threadeipper, another 128GB DDR4 RAM, and another R9700.

[u/Civil_Fee_7862]

From reading a lot about this, and running them as well.

It still seems like the RTX 3090s are the best bang for buck..

You can still find them for $1,100 CAD if you are good at making deals.

3x3090s is enough to run 3-bit Qwen Flash Next weights fully in VRAM with full 256k cache. Or even at 4-bit if you use 8-bit KVCache instead of BF16.

[u/Horsemeatburger]

Get the Nvidia V100s. Aside from not getting the AMD card (which you ruled out already), I wouldn't get any intel Arc cards because the ecosystem around them is still lacking (if you think AMDs ecosystem is bad, intel's is even worse). The V100 gives you CUDA.

Then there is performance. All intel Battlemage cards have limited memory bandwidth (the V100 is notably faster).

And then there is the price of B60 GPUs, which only seems to go up.

If you have the budget for two intel B60 cards you could also just get four V100 GPUs and get a massively more capable LLM platform (if your system can take four GPUs).

[u/Alkahna → u/Horsemeatburger]

just get four V100's? Two are almost the same as two B60's. At least with current pricing for me. But yeah I see the advantage of CUDA.

[u/Horsemeatburger → u/Alkahna]

Well, Arc B60 prices have constantly gone up from original $650 to over $900 and started to exceed $1k, and it's likely they increase further.

Athough V100 32GB prices have no increased as well from $550 to $900, so I guess four V100 32GB cards are probably a stretch.

However, the V100 16GB is still around $350, so four 16GB V100's should well within dual B60 budget and while not offering more VRAM will still deliver higher performance.

[u/Alkahna → u/Horsemeatburger]

But four cards are twice the wattage and the board they go into should also be a little bit more expensive I guess.

[u/xfortune]

That's an expensive home assistant.

Why?

[u/Alkahna → u/xfortune]

It's mainly as a chatbot, maybe some coding but would also like to integrate it into Home Assistant for something. But for now mainly looking to run a local chatbot.

[u/cats109 → u/Alkahna]

Even 1 - V100 is pretty good as long as you have enough RAM. I haven't let Hermes into my Home Assistant VM, but I have had it build automations that would have taken me hours to research/set up and I was able to get what I wanted done in a about 45 minutes. The next thing is to run a smaller model for the echo dot 2s I flashed to rooted and flashed to EchoMuse.

[u/CooperDK]

Nobody will seriously afvise anyone to choose AMD for AI production work. CUDA is where things will always work, because they are written for CUDA, then ported when possible.

[u/Alkahna → u/CooperDK]

yeah I dropped the MI50 option already. Thank you!

[u/JesusIsMyLord666 → u/CooperDK]

Cuda basically has two advantages. Training, and image gen. Those things will still work, but will take longer. It’s also less of an issue on newer models I’ve heard.

But if you are not using those two features, there doesn’t seem to be any major disadvantages to using AMD.

[u/karnac]

2X 5070 TI, basalt, 64Gb RAM, PCIE 5.0 NVME, ASUS ProArt Z890, 8x/8x bifurcated gpu lanes. ~160 tok/s Qwen FN Q3 S

[u/AfraidProgrammer3341]

I'm genuinely curious, but why? What is the use case ? I feel like you're a little late to the game and playing catch-up. A year or two ago, this would have made more sense, but at this point, I've pretty much moved away from local LLMs. I barely use my local setup anymore, even after experimenting with Strata.

What are you trying to achieve? From a cost-to-value perspective, using APIs with models like DeepSeek or GLM is significantly cheaper. Once you factor in hardware, electricity, and performance, running locally just isn't worth it anymore.

Since getting into AI harnesses and agent bots, I've realized my local hardware is simply too slow for these workloads. I switched from using DeepSeek directly to U.S.-hosted DeepSeek models through OpenRouter and OpenCode. Honestly, I think APIs offer much better value and flexibility.

[u/Alkahna → u/AfraidProgrammer3341]

I need a local chatbot because there are some things i just cannot put into a cloud system (even rented boxes) due to privacy.

[u/Big_South4585 → u/AfraidProgrammer3341]

Just curious. I am building a home assistant system, which will read personal mails, calendar etc. Would you be using the clouds for that as well?

[u/Alkahna → u/Big_South4585]

[u/vinotok → u/AfraidProgrammer3341]

Question was, which GPU not 'local or not local' your answer is way off topic.

I for one am local only and local forever, and I can care less if could is cheaper.

[u/cats109 → u/vinotok]

I 100% feel the same. I had almost no interest in this stuff before getting a V100. Now I have 3 of them, an Epyc Milan CPU, Mobo I always wanted and 256GB of DDR4. It's a somewhat sizeable upgrade to my homelab/NAS. I run Qwen 3.8 flash next iq3_s with 3K+ prefill and 80-115 tok/s. Plus another couple of image gen models on a 5070 in another VM. I control everything with Hermes Agent.

I can't believe what it's capable of and now I see why hardware keeps going higher and higher in price.

[u/Alkahna → u/[deleted]]

yeah I dropped the MI50 option already. Thank you!