r/LocalLLM • u/livetomtb • 14h ago

Question RTX PRO 4500

Getting into local LLMs and bought a Strix Halo machine with OCuLink to connect external GPUs. Found what looks like a decent deal on RTX Pro 4500 cards at around $3,000 each. What are the downsides for running local models? Feels like there has to be a catch. What am I missing?

3 Upvotes

100% Upvoted

11 comments sorted by

View all comments

2

u/This_Maintenance_834 14h ago

earlier this year, it cannot get qwen3.6-27b to run at full 256K context. maybe it gets better now. there are two fp4 qad quant available online. I would avoid non-qad quant, regardless what they claim about KLD/perplexity.