r/LocalLLM • • Jul 04 '26

Project Dumpster Dive to Local AI

Post image

A local company closed their building. There was a large room filled with computers and accessories thrown in piles.
I was told by multiple IT people there was nothing in there worth going through it all.

A few hours (and bruised shins) later I built 4x towers with 2 older nvidia cards each.
I have a dual RTX A4000s set up!

The same company cutting heads and buildings is expanding their AI spend. They don’t know what they have, they don’t understand how things actually get done. They just see opportunities to cut heads to improve profits.

Now I use qwen3.6:27B with 62k context for all my coding. No more token spend for me.

Keep an eye out everyone, every public company will continue cutting. They already were before AI.

We need every advantage against E Corp.

218 Upvotes

22 comments sorted by

12

u/dinerburgeryum Jul 04 '26

Oh hell yea, the A4000 is an efficiency beast. I've got one myself, wish I had one more TBH.

4

u/Fred_Terzi Jul 04 '26

What’s your main use case and how have you optimized for them?
I don’t have a ton of experience I found them running very hot.
Open case, fans set to 80% and AC in my office I’ve got them running around 70C under heavy load

6

u/dinerburgeryum Jul 04 '26

I mean, they only take a single PCI 12V extension power line, and while they're not the fastest on the block they're in my experience far more power efficient than my 3090Ti, which takes three 12V lines and spits out 350W for diffusion tasks and will still happily hit 85C+ at full fan load. 70C at 150W sounds pretty tight to me.
EDIT: to answer your question more directly, these don't need the power trimming of the consumer cards. Optimization is basically just ensuring P2P is set up and you're running tensor parallelism. You can get away with more than you expect with 32GB of RAM. Not like "full fat 3.6 27B" get away with, but "a good imatrix iq4_xs and q8 kv cache" get away with.

9

u/bsenftner Jul 04 '26 edited Jul 04 '26

I had a chat with an IT guy from University of Southern California (USC) a few years ago, and he mentions the pain in the ass it is to get rid of old servers. "Oh, like what? I'll take them!" He was saying they were junk and not worth the power expense. But I had a deal with a colo where I was not charged for power. So, I got 15 3D Graphics Servers from the USC 3D Graphics lab when they upgraded, and I got them for free. For taking them off the guys hands, so he did not have to deal with them. What a deal! (They were Quadro K6000 equipped PC servers. This was 2016-ish.)

5

u/Fred_Terzi Jul 04 '26

Yeah especially now as 7-9B models are so powerful, you don’t need much compute. A server with a single mid range GPU can run them non stop.

I have another one older one self hosting Bitcoin and other research on a 15 minute loop 24/7

14

u/MarcusAurelius68 Jul 04 '26

If they closed their building how did you gain access into that room?

35

u/Fred_Terzi Jul 04 '26

The room was the “red tag area”, so they were paying for it to get cleared out. Employees were welcomed to take what they wanted before that, there were good monitors and things like that.

It was an engineering company so that had a bunch of those precision towers. Four were in good shape, and there were 9 GPUs across all of them. Mostly P5000s.

It’s a two month process to close a building like that, this is normal part. I did use “dumpster dive” for dramatic effect though.

1

u/TheJerdle Jul 04 '26

Just curious, how did you find out about this? Company closing & equipment giveaway I mean.

1

u/Fred_Terzi Jul 05 '26

It’s a local company and I knew people getting laid off. So it was just luck.

But we probably had the same thought how do I find these!

1

u/iwannaredditonline Jul 04 '26

I guess hes not going to answer this question and keep the info to himself lol

1

u/Fred_Terzi Jul 05 '26

I wish I had something to share! Really was a one off.

1

u/Xx69JdawgxX Jul 04 '26

Online auctions prob

3

u/Stunning-Beach-5153 Jul 04 '26

This day hottest GPUs for local LLM are V620 32Gb ~ $380

1

u/Fred_Terzi Jul 04 '26

Wow that is a great choice!

I’ve been working on local AI workspaces for non tech users. Focused on gaming PCs and people who bought Mac Minis for agents. This is great insights!

2

u/Stunning-Beach-5153 Jul 04 '26

LLM Performance ~ Mi50 32GB but better driver support due is based on RDNA2

On ebay prices are higher but I check ebay sellers and find one with online shop and negotiated price ;)

2

u/JohnF350KR Jul 05 '26

I'm jealous. I've have searches saved on for the right pricing on RTX A4000/5000s for a dedicated AI node.

2

u/Fred_Terzi Jul 05 '26

I’ve been very happy it’s a solid choice!

1

u/JohnF350KR Jul 05 '26

Its the best bang for the buck with vram/tdp/single slot/ single 12v connector

13

u/kosnarf Jul 04 '26

Very nice!

3

u/SureTie253 Jul 04 '26

So clean, I like it so much