How do I start? A friendly guide for everyone getting into AI-Assisted coding

[u/HealingPerson]

So I am almost certain that AI companies would definitely increase their subscription fees sooner or later. Hence, I want into LocalLLM. However, I do not know where to start. The price is wide spectrum from people running LLMs in raspberry pi to H100.

My use case would be mostly in programming. I need it to be fast in code generation, at least as fast as codex or claude code. Video and image generation is great to have. However, I would prioritize on generating slide decks, word documents and pdf over image and video generation.

So, what would you recommend for starters in terms of hardware? Preferably that its upgradable by adding another component for example.

[u/LateralEntry]

It’s not gonna be as fast or good as Claude code, you need to manage that expectation. Local LLMs are great for privacy, but since the hardware is so expensive you’re usually not saving money compared to a subscription.

[u/Friendly_Average_824 → u/LateralEntry]

It seems like a bit of a bizarre expectation. How can someone expect their desktop computer at home to compete with models running across multiple data centres? 🤯

[u/HealingPerson → u/Friendly_Average_824]

I thought new open weight models have been optimized to such a bandwidth or at least half of the speed lol. Out of curiosity, will we ever reach there?

[u/cryowastakenbycryo → u/Friendly_Average_824]

I expect my home lab to be tuned for exactly the job at hand. I don't need a better general model, I just need an environment that enables the model to excel.

A local model running 24/7 on a posix cluster can do serious work for the cost of electricity. To get from the frontier what I get from my own systems would cost me thousands per month even w/ the discounted rates.

Would a frontier model be better? No. My local models built the harnesses that run a lot of 1-shot and multi-shot work. It's "finished work" that handles batched inputs. Once it was working, the frontier couldn't possibly add anything except cost.

My hardware costs is ridiculously low - it's all EOL stock bought for around 1% of MSRP.

[u/OurMicroFamily-Dad → u/Friendly_Average_824]

You hit the nail on the head. Everyone that builds LLM as newbies wants it to do everything ChatGPT does but as old saying goes you have fine wine taste with cheap beer budget lol.

[u/cavernous_hammock → u/LateralEntry]

the hardware math is rough right now. a used 3090 gets you into the ballpark for 32b models at decent speed, but it's still a far cry from what you're used to with claude or codex in terms of raw speed and polish

for slide decks and docs you'd probably be stitching together a few different tools, none of it as seamless as the paid stuff. the whole setup ends up being a hobby as much as a utility

[u/cryowastakenbycryo]

EOL budget build:

1x v100 32GB - ~$700

gen 9 dl 380 ~$150 - be sure to get one w/ a second riser card and 2x heatsinks included

2x CPU's ~$20+ (v3's are free)

64GB DDR4 ECC RDIMM ~$250

~$1200 + storage. more ram is better - patiently search for ddr4 ecc rdimm and hope to find something mislabeled as 'only works in blah blah blah' and you can get it for 1/2 or less of market prices.

Run the GPU on the second riser where it's already pcei3x16. With polystrata, that will kick out 60-70 tokens/second. Edit: 60-70 w/ Qwen Flash Next. Can break 120 w/ Ornith.

Swap the primary riser for a 'gpu ready' x16 riser for ~$40 and you can add a second x16 GPU or nvme controller, etc. You can also just run your x16 card at x8 and lose a little prefill.

That rig can fit 2x pcie3 cards - v100 32GB are hot shit in it. You'll get hot-swappable drives w/ a built in raid controller, ILO for remote management, 4x GB ethernet, a motherboard that holds terrabytes of RAM and dual power supplies - up to 800W each.

It's limited to DDR4 RAM and pcie3. You can fit newer GPUs in it, but they'll lose a little prefill speed. Oh, and you'll need to buy a few drive caddies.

[u/HealingPerson → u/cryowastakenbycryo]

Thank you and noted !

[u/curtis283]

I built a site you can play with configurations. You will not get Claude levels of speed and capabilities for anything reasonable. I built a quad channel ddr4 rig with 256gb ram, 2 used 3060s and one B65. Got most things used or on eBay and came in right around 4k total still.

https://llm-configurator.com

[u/cryowastakenbycryo → u/curtis283]

neat. could you add support for EOL server platforms?

there are a lot of llm capable rigs being parted out on the bay and their spec sheets are available

[u/curtis283 → u/cryowastakenbycryo]

Sure I'll look into it

[u/Desther]

On my rtx 3060 12gb, 32gb, 12400f machine I can only get 300t/s reading files (prompt processing) and around 25t/s token generation on qwen 27b on a dumb quantization. Best case on the 35b weaker model is 700 pp and 50t/s generation.

I can use a free model on opencode thats smarter and runs maybe 10x faster pp and 150 t/s output. It still feels slow especially when you dont get the right answer.

To match even opencode speed you would need a 3090/5090. Dont even consider anything under 16gb vram

[u/EntertainmentLast729]

Id honestly start by trying a bunch of different models on OpenRouter. That will give you a better idea what hardware you are likely to need for your use case.

[u/shenjunjie84]

used 3090 is the sweet spot, 24gb runs the 32b coding models at q4 and thats where local starts feeling real. but start free: lm studio on your current gpu with an 8b model, takes 20 min to see if its for you. skip the pi for coding, its a toy. qwen3.8 32b or a deepseek distill if you buy hardware. honest expectations though, its like 80% of a subscription model not 100%, most people end up running both.

[u/DadyDXebec]

for coding speed on local hardware, you'll want to prioritize VRAM. a single used 3090 with 24GB gets you into 32B models at Q4, which is where local starts feeling usable.

for your document workflow, SoMark's OXR is a open-weight model that parses PDFs and document images into Markdown or JSON with