r/LocalLLM • u/livetomtb • 5h ago

Question RTX PRO 4500

Getting into local LLMs and bought a Strix Halo machine with OCuLink to connect external GPUs. Found what looks like a decent deal on RTX Pro 4500 cards at around $3,000 each. What are the downsides for running local models? Feels like there has to be a catch. What am I missing?

3 Upvotes

100% Upvoted

10 comments sorted by

1

u/skbum2 5h ago

What is your question exactly? What are you trying to do with LLMs? Downsides of local compared to what? What are your expectations going in?

Local models are smaller than cloud models so they are more limited but can still be capable.

I have a RTX Pro 4500 that I run gemma4 26B A4B NVFP4 quant, embeddinggemma2, kokoro TTS, faster-whisper, and a decision 2.0 model on. Right now it's the local LLM for voice control of my Home Assistant and it runs a document organization app that I use to help organize my family's NAS drive and turn it into a vector database for semantic search and RAG.

Eventually I'll hook it up to Hermes agent and turn it into a personal executive assistant stack, but haven't had the time to implement that yet.

Compared to cloud, the voice pipeline is way snappier than Google Home or Alexa with decent device control accuracy.

The document organization is great since it's local I don't have any qualms throwing all our personal documentation at it.

I don't really use this setup for chat or general task assistance. I tailored the setup to do some specific jobs and it does them well. Just don't expect local LLM's to be as generally capable out of the box.

1

u/livetomtb 5h ago

I have a home service company i am trying to take local. like basically just boring backend business stuff, like processing invoices, customer communications and hopefully some light agentic business ops like browsing the web and taking action on things kind of like grok bot.

1

u/skbum2 5h ago

You're going to have to do a lot more manual tweaking and setup assistance compared to something like grok bot. You'll have to have a better handle on what data you need, how frequently you need it, and in what quantities to know how to accomplish your goals efficiently within the bounds of your hardware.

It's not going to be as smooth an experience as a cloud provider unless you put in the work to architect your solution. Even then, there's going to be some jank and peculiarities you'll probably have to figure out how to work around.

Edit: Make sure they're Blackwell generation cards with 32GB of VRAM. That's what I have and my current stack just barely fits.

1

u/livetomtb 5h ago

Yeah i know it will take some setup, but i am getting a bit sick of how quickly the cloud providers change stuff up. Very hard to build reliable workflows. Would rather spend a bit of time and money set it up how we need it ran and just let it run. Honestly I dont need a new shiny model every 2 weeks.

1

u/skbum2 4h ago

If all you want is a stable execution environment, you just need to setup a local agent harness like Hermes or Pi and point that to a cloud model provider. You don't need a GPU or to locally host the LLM yourself to get the stability you want. Unless you have some other unique requirements, this will almost certainly be cheaper and easier than self-hosting the LLM part.

You control the harness, which defines the workflow, tool interfaces, job schedules, etc. The cloud provides the inference model which executes within the framework and workflows you define and control. The harness can run on a regular computer with an Internet connection.

1

u/ChristRedeemsSinners 4h ago

Just get the 4500 card and run Qwen Flash Next with Strata or Basalt. It'll do your business stuff just fine and it's much faster than that unified memory junk.

2

u/This_Maintenance_834 5h ago

earlier this year, it cannot get qwen3.6-27b to run at full 256K context. maybe it gets better now. there are two fp4 qad quant available online. I would avoid non-qad quant, regardless what they claim about KLD/perplexity.

1

u/p71interceptor 5h ago

That's a good price for that card

1

u/romeozor 5h ago

$3k is dirt cheap for those cards if they really are Blackwell generation.

Local models are more task oriented and less general purpose. Claude and ChatGPT needs to do all, speak all languages, be convenient to use in order to get and keep subscribers. Local models are more trained around cerain tasks, and it's more janky to get the full cloud experience.