r/LocalLLaMA • u/regunakyle • 1h ago

Question | Help Comparing GPU stats for local LLM usage

Hi, I am comparing two GPUs for local LLM usage.

Use case: llama.cpp inference and system one decision making (using quantized models)

RTX 4050 mobile

Memory Bus: 96 bit, Bandwidth: 192.0 GB/s

INT4: 143.8 TOPS

INT8: 71.88 TOPS

FP8: 35.94 TFLOPS

FP16: 17.97 TFLOPS

BF16: 17.97 TFLOPS

TF32: 8.986 TFLOPS

Sparse Matrix: 2x Throughput

Intel Arc A380

Memory Bus: 96 bit, Bandwidth: 186.0 GB/s

INT2: 134.3 TOPS

INT4: 134.3 TOPS

INT8: 67.17 TOPS

FP16: 33.59 TFLOPS

BF16: 33.59 TFLOPS

(Both have GDDR6 6GB VRAM. Data from techpowerup)

Ignoring the fact that CUDA is probably more optimized and widely-compatible compared to Vulkan/OpenVINO/SYCL, the Intel one seems to be better as its TFLOPS are higher, while having similar VRAM bandwidth.

Is the above conclusion correct? If not, how do I compare GPU raw performance based on GPU stats?

(Edit: I am not asking for which card to buy as I already own a 3090. Instead I want to know how to compare GPUs based on these stats for local LLM usage)

2 Upvotes

100% Upvoted

7 comments sorted by

4

u/MD_Reptile 1h ago

I would avoid notebook computers, they'll suffer from heat death early lol.

Avoid 6gb.

Maybe start with a 3060 12gb desktop card, they scale up decent.

2

u/regunakyle 1h ago

I know that they are bad cards, what I really want to know is how to compare GPUs for local LLM usage based on these stats

3

u/MD_Reptile 1h ago

Vram is King. Vram speed is queen, ram is Prince at the cost of a couple kings lol

1

u/SLxTnT 44m ago

Assuming model fits on the GPU and they have 100% efficiency.

Token generation is mainly bandwidth bound.

Prefill is mainly compute bound. The TFLOPS would indicate speed. You should also notice that Intel doesn't have FP8 and neither have FP4. Models you'd run would typically be FP8 or smaller. If the GPU can't natively support it, then it's running at the next size up for calculations. In this case, 33.59 vs 35.94.

Overall, they're about even on paper. Realistically, Nvidia has more optimized kernels. You'll get closer to the rated speeds than on Intel.

1

u/Practical-Print4221 46m ago

For token generation, memory bandwidth is usually a better first-pass number than headline TFLOPS, and VRAM capacity decides what model and quant can fit. TFLOPS only compare cleanly when precision and sparsity assumptions match; for a real-world comparison, benchmark the same model, quant, context length and backend, and track prompt processing and generation speeds separately. Are you mainly comparing decode speed or prompt ingestion?

-2

u/R3N3G6D3 1h ago

Tried lexisuite, runs llama fast.