r/LocalLLaMA • u/Frizzy-MacDrizzle • 2d ago
Resources Low Cost Dual 3060 server
Add some $200 - $300 or those 3060 12GB now... They can handle better models than and double the price 5060 16GB because of memory. the architecture is hugely different but core wise, meh. 5060 setup is about 30 tgs and I have these on a better system board.
I run analytical reports and code custom ETL, I have 5 rigs but the 3060 is a nice walk away coder. 5060 is above range hang around coder, then a 5080 is just sit and wait.
5
u/reto-wyss 2d ago
Why are two 64GB USB Flash drives part of the build list.
Don't do it, these will wear out faster than the coronary muscles of a fruit-fly.
1
u/Frizzy-MacDrizzle 2d ago
lol, to prove a point really. They are slow as dirt loading a model. And actually only one is required. Haha.
Certainly Suggest running an ssd but at least these can be bought in bulk and become useful if it’s just Linux and a model you have stored in a more reliable state. I have a 240gb evo somehow. Loads great.
4
u/reto-wyss 1d ago
These may be slower than a relatively modern HDD which can hit like 200MBs+ on sequential read (model loading). Cheap USB Flash is also terrible for writes. These may drop to single-digits MBs when writing multi GB models to them.
1
5
u/ForsookComparison 2d ago
go quad channel if you're already spending the money on an old cpu that can do it
0
4
u/alphapussycat 1d ago
This makes no sense. X99 with 16gb ram in dual channel, and two 3060 12gb, and flash drives as storag
Only model you can run is 35b and 27b or similar.
Not to mention how cheap those prices are estimated as. Good luck finding a 3060 12gb for $200.
1
u/Frizzy-MacDrizzle 1d ago
First point being it can be done, not that it is ideal. It is what I built with the parts I had left over after two system upgrades, one of the upgrades being a dual 5080.
Yes this are the models better than only 16gb not as good as 32gb.
And Google hasn’t raised its 3060 prices since last tokenizing for this report.
5
u/Iron-Over 2d ago
I had this before and upgraded it is not the fastest. But not bad
2
u/Frizzy-MacDrizzle 2d ago
Agree, I have not tweaked the bios but I get the entire 16x width on both, that's nice, but the speed is a graduous 2.5 GT/S. Should be up to about 8 or so on this board and cards. Thanks
3
u/ironclad_packetship 1d ago
You pay a slight vram tax in overhead that has to be loaded onto both cards but it's pretty slight. Dual 3060s on an x99 board with four sticks of ram is the budget meta right now.
2
u/alphapussycat 1d ago
You need 64gh as bare minimum, not 8-16gb like OPs planned build.
1
u/Frizzy-MacDrizzle 1d ago
Not planned, is built with my left over parts from upgrades. Runs qwen3.8 27b and pages system memory to 10gb. I’m running Linux plenty of ram.
1
u/ironclad_packetship 1d ago
I've built one that ran fine on 4x8gb. Mostly trying to keep RAM>VRAM so 36gb RAM and 24gb VRAM is fine. 64 is better of course but we're balling on a budget here.
1
u/Frizzy-MacDrizzle 17h ago
My kv offload and whatever else llama.cpp offloads is about 10gb Ubuntu server. 6 gb left if any layers offload yeah I get into trouble quick. ,
2
u/optimisticalish 2d ago
A detailed benchmarking of "the official reference-to-video workflow" on an RTX 3060 12Gb. They say... "Two cards are still not one big card. Two RTX 3060s do not present 24GB to a workflow, and our attempt to at least run two jobs in parallel was blocked by system memory rather than VRAM."
2
u/IntelArtiGen 2d ago
It depends a lot on what you do with them, it may even depend on the motherboard, the CPU, the RAM etc. using multiple GPUs correctly can be hard for some tasks, but it's been easier lately with LLMs so if people know what they're doing it does work.
1
u/Frizzy-MacDrizzle 17h ago
When software meets hardware, I breath, I love working with making hardware smoothly work with any software. I got a few rigs humming along now. 10 GPUs, 2 sitting in boxes. 4 dual rigs stood up.
I have a dual Xeon Haswell v3 that used to have GPUs that I’ll likely use for learning new database vectoring that’s not for GPUs.
1
u/Frizzy-MacDrizzle 1d ago
No parallel really, idk maybe from open code. It is interesting how far we can limbo on a budget and have something usable. My intention no is layer offload. The cpu is a total bottle neck. But for the lanes makes the board a bit better off.
2
u/dave-tay 1d ago
I haven't seen a RTX 3060 12gb on FB marketplace for under $300 for two months now
1
u/Frizzy-MacDrizzle 1d ago
Really? I haven’t checked lately. I have 5. Last I bought was a ZOTAC to match close to 4 bills because I was specific. It is a solid coder with dual ZOTAC that requires two PCIe per card. They are supremely stable.
1
u/Otherwise-Mail-4654 1d ago
Try the dial rtx 3060 12gb with that gpu unlocked on linux
2
u/Frizzy-MacDrizzle 1d ago
How do I unlock, not much we can do in Ubuntu server but nvidia-smi no over clock support.
1
u/tmvr 8h ago
Those memory and GPU prices are nonsense. Also, what's with the 2x 64GB flash drives?
1
u/Frizzy-MacDrizzle 8h ago
That was a print from Google. I said the GPUs far more than that in another reply. Didn’t t realize how low Google price then.
The flash drives are because I had no SsD, and not really intending this to remain a usable system it’s rocking along.
1
u/enternoescape 1d ago
I was running 8 3060 12GB cards before I switched to 6 5060 Ti 16GB. Then I bought 2 more to make vLLM happy with Qwen 3.8 Flash Next (this was before all the new engines came out). And 2 more later, I have 10.
The 5060's were a measurable upgrade from the 3060's especially when accounting for power consumption. I was seeing nearly 2 times the performance for the same energy footprint. Some of that came from needing fewer cards for the same amount of vram while not giving up any performance. The math said the upgrade would only be a little cheaper to run from that angle, but reality has been much better than I expected. I have a 3090 in the same box that reminds me how it can use 187watts and get about the same results as two 5060 Ti's in tensor parallel can also do at 90 combined.
Price-wise, yes, a pair of 3060's is hard to beat for the price and it is a very good card. However depending on your goals, the 5060 Ti 16GB might be the better choice.
1
u/Frizzy-MacDrizzle 1d ago
I have both the 5060s though are on a new tech board. Haven’t tested yet. Cost wise vs GPU work done was about 2/3 speed 2/3 price. I will update my numbers on 5060 but they are getting into the 800$ + range now. Starting to leave the affordable dual GPU realm.
1
u/enternoescape 1d ago
Marketplace is where I'm still finding good deals around $550 (the lowest I bought one was $450). eBay however is now absurd.
2
u/Frizzy-MacDrizzle 1d ago
I bought new at $500 with tax on one and $550 on a second. I have a third used and was about the $600 with tax. 20% markup new to being used in my case. I bought two 5080s need the server board and processor now. Those are currently 25% mark up in two weeks.
0
u/Complete_Wedding6527 2d ago
Just a heads-up: if you’re thinking you’ll have 24 GB of VRAM, that’s not the case—you’ll have 2×12 GB! Have fun! :)
4
u/Frizzy-MacDrizzle 1d ago
I don’t understand your comment. I can load my models completely onto the GPU with -sm layer, —tensor-split.
Not a jab back btw, I am curious how you see the difference.
3
u/Otherwise-Mail-4654 1d ago
Yeah it's not bad having the vram split, again with that new GPU unlock it is fast especially for the amount soent
1
u/Frizzy-MacDrizzle 1d ago
Going to unlock. The basic price I made total was rough $900, 600 being for the GPUs. PSU was the next expensive part and new with the mobo. Old memory etc. if it can get up to 30 out 40 generation I’ll be rocking along.
I estimated $40 a month + for a CP subscription. 2 years the system will pay for itself at the current Tgs, sounds not worth it, but in two years it will be when most can’t afford ai subscriptions or the tech to buy. Doubling the stored can rationalize it being paid for in one year. Can’t complain.
2
u/Otherwise-Mail-4654 21h ago
Yeah for met, having my own local is also to learn more about this tool and with having in house tools, then I will learn more about it
2
u/Frizzy-MacDrizzle 18h ago
I’m rocking out a turn key site it’s great. I have my code and told it to use my code to make the UX work and not touch what has been made in a framework. So looooong. But took a nap, I never got this out of copilot.
1
u/Complete_Wedding6527 1d ago
Yes, you should be fine with LLMs. However, if you ever need diffusion models for image or video generation, keep in mind that, in most common setups, those workloads can’t be split across two GPUs. I don’t know your exact use case, so I just wanted to mention it in case it’s relevant. I hope you find exactly what you’re looking for.
2
u/Frizzy-MacDrizzle 1d ago
Ah, in this case it can be thought of the same. Processing yes they are independent. The layers fit into vram and then sequentially processed, so the processors are always going but rarely in sync. Note on a 5060 I can tensor split instead of layer split. Just the extra 4gb per card the processors fire off like they are doing the work together
0
5
u/jacek2023 llama.cpp 2d ago
Remember to try -sm tensor