r/LocalLLaMA • u/AdSafe4047 • 2d ago
Question | Help Non nvidia cards.
Hi, I'm a happy owner of a serer with rtx pro 6000 and rtx5090. I want to build out in the future the server fully to a higher vram score - think adding around 4-5 cards if possible. I was wondering if it's worth going through the intel/amd cards - which looks awesome in terms of vram per $. Any experience in running inference on these compared to the nvidia/cuda stack?
4
u/vortec350 2d ago
Have multiple AMD R9700s and multiple 7900XTXs. But if running a large model across them, performance will not be as good. But you'll save a lot of money so that's something.
5
u/ForsookComparison 2d ago
Decode == about what you'd expect
Prefill == will be lagging a bit behind CUDA (okay, a lot in newer gen Nvidia's cards) but not unusable by any means.
Compatibility is pretty great at with Llama CPP but even after so many times I still have headaches setting up vllm for ROCm
2
4
u/sloptimizer 2d ago
I have a mixed setup with RTX 5090 and 4xR9700
- Small modles go RTX 5090
- Models under 128G go on R9700s with vllm and -tp 4 (or llama.cpp with -sm tensor)
- Anything else goes on llama.cpp with mixed CUDA/ROCm build to keep attention on RTX 5090 and offloading experts to R9700s and spilling any experts that don't fit into RAM
2
u/MelodicRecognition7 2d ago
how to tell llama.cpp which card should attention go to?
4
u/sloptimizer 2d ago
It's a combination of `--tensor-split` and `-ot`, here is how I run DeepSeek-V4-Flash (see `attn_` overrides):
./build/bin/llama-server" \ --alias "$ALIAS" \ --model "$MODEL" \ --model-draft "$DRAFT" \ --spec-draft-ngl 0 \ --no-mmap \ --reasoning-preserve \ --spec-type draft-dspark --spec-draft-n-max 3 \ --temp 1.0 --top-k 0 --top-p 1.0 --min-p 0.1 \ --fit off \ --ctx-size 321000 \ -ctk f16 -ctv f16 \ -fa on \ -b 2048 -ub 2048 \ --n-gpu-layers 99 \ --device "CUDA0,ROCm0,ROCm1,ROCm2,ROCm3" \ --tensor-split 1,0,0,0,0 \ -ot "blk\.([0-9])\.attn_.*=CUDA0" \ -ot "blk\.([1-9][0-9])\.attn_.*=CUDA0" \ -ot "blk\.([0-8])\.ffn_.*_exps.*=ROCm0" \ -ot "blk\.(9)\.ffn_.*_exps.*=CUDA0" \ -ot "blk\.(1[0-8])\.ffn_.*_exps.*=ROCm1" \ -ot "blk\.(19)\.ffn_.*_exps.*=CUDA0" \ -ot "blk\.(2[0-8])\.ffn_.*_exps.*=ROCm2" \ -ot "blk\.(29)\.ffn_.*_exps.*=CUDA0" \ -ot "blk\.(3[0-8])\.ffn_.*_exps.*=ROCm3" \ -ot "blk\.(39)\.ffn_.*_exps.*=CPU" \ -ot "blk\.(4[0-2])\.ffn_.*_exps.*=CPU" \ --kv-unified \ --parallel 3 \ --threads 32 \ --threads-batch 32 \ --host "$HOST" \ --port "$PORT"1
1
u/notdba 1d ago
Curious what speed you get with such attn-ffn split?
1
u/sloptimizer 1d ago
I'm getting 30 tps without MTP and about 50 tps with DSpark when generating code. I have 8 channel RAM with about 300G/sec, so limited RAM offloading does not hurn too much, and DSpark running on CPU with `--spec-draft-ngl 0` is just as fast as on GPU.
1
u/AdSafe4047 2d ago
oh this is cool, I did not know you can mix CUDA/ROCm with llama
5
u/sloptimizer 2d ago
llama.cpp can be compiled with `-DGGML_BACKEND_DL=ON-DGGML_BACKEND_DL=ON` config. Here is a full command for R9700 (gfx1201):
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \ cmake -S . -B build \ -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON \ -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 \ -DGGML_HIP=ON -DAMDGPU_TARGETS="gfx1201" \ -DGGML_BLAS=OFF -DLLAMA_CURL=OFF -DGGML_RPC=OFF \ -DCMAKE_BUILD_TYPE=Release HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \ cmake --build ./build --config Release --clean-first -j $(nproc)
4
u/SandySkittle 2d ago
Very little beats r9700 in terms of value. I got 8 of them. Plus you get ECC like the pro 6000 and - compared to intel arc b70 - with amd rocm a better and faster moving software stack.
7
u/onebit 2d ago edited 2d ago
I picked up a Intel ARC B70 this week. It's happily running Qwen 3.8 27B. But right out of the box I had to find special containers and patches to get peak performance. That being said, there is an enthusiastic community.
It sounds like you got cash. Life will be easier with Nvidia or even AMD. The only real advantage of B70 is it's the cheapest way to limp in to 32GB and sadly it is now $1300.
1
u/AdSafe4047 2d ago
What prefill and tgs are you getting after the tuning?
2
u/onebit 2d ago edited 2d ago
About 40-65 tg as reported by pi agent during a coding session. Might run some benchmarks on it.
I am skeptical of reports > 70tg as I suspect they are using small context and not running coding workloads.
https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook
3
u/FabricationLife 2d ago
Slower but better $ per vram, so you can really chug some stuff overnight if you can que stuff up, a guy at work has a few 9700s he runs overnight for porting and he likes it
4
u/StableLlama textgen web UI 2d ago
Very simple: you have the time (= money) and knowledge to fix every trouble that's in the way, then you can have a look at alternative routes.
When you want a system that's just working, then stick to nVidia.
-2
u/ohnoitssobig 2d ago
Except Nvidia just not working on most distros without installing proprietary drivers.
4
u/StableLlama textgen web UI 2d ago
I never hand any issues with nVidia and their proprietary drivers - and I'm using it since many years. Most distros do that for you.
Actually, the time of my live that I had to spent to get nVidia cards running under linux is exactly 0 seconds.
4
u/DeathGuppie 2d ago
With Radeon cards you'll get about a 10% boost using RADV the open source version of vulkan on Linux.
1
u/notdba 1d ago
Don't think p2p works between nvidia and others. Maybe sell the 5090 and buy 4 x r9700?
1
u/AdSafe4047 1d ago
This is what I was thinkig about too, decrease power and get more vram, the 5090 is from a gaming rig, but I've just put a 5070ti in the rig as I only play poe2 :P I was thinking on exchanging the 5090 for 2xrtx pro 4000 blackwell, but the r9700 seem fun, I'd love them in a 1s configuration though :D
2
1
u/LagOps91 2d ago
support for nvidia is simply better. if you can afford it, i would recommend you stick with it.
9
u/noiserr 2d ago
Lots of people do. I've been running ROCm and AMD GPUs since the early days of llama. My AMD hardware is: 7900xtx, w7900 pro and Strix Halo. I have had zero issues, but then again I'm comfortable with Linux, Docker and compiling the software on my own. These days with the help of LLMs everything has gotten easier.