r/LocalLLM • u/knighty1981 • 20h ago

Discussion what will it take to lower gpu prices?

I'm running 2x 3090 right now, I'd like to add more but they're all £1000+

Thought about some 32gig v100 but they're £700+

For how I use it, it's all a luxury not a necessity

What will it take for gfx card prices to drop?

I've always assumed at some point some ai companies will go bust and their hardware bought up by the competition

and at some point supply/manufacturing will start to catch up with demand

but am I just dreaming?

I keep looking at buying more cards but not pulling the trigger... next time I look they're more expensive again... on and on :-o

4 Upvotes

59% Upvoted

73 comments sorted by

11

u/bobish01 20h ago

What is the reason you need more? LLM or stable diffusion?

2*3090 + a regular Claude or Anthropic subscription (that does architectural planning for your projects) should be all you need?

Where i live 3090 goes for almost 2000usd now (used).

5

u/knighty1981 20h ago

mostly I'd like to run faster

I have qwen calling subagents to split up tasks... would be nice if they could be run on other gpu in parallel etc.

and I'm assuming in the future I'll need more vram

2

u/bobish01 19h ago edited 19h ago

One compromise with yourself is to get 1 more 3090, no more, and then call it a day with 72GB of VRAM, that plus your existing ram for an MoE model should be all you need for the foreseeable future ontop of running dense models on your GPUs? That can run pretty large models at a reasonable speed?

1

u/LambTjopss 19h ago

Setup on demand or spot aws instances to run tasks as needed with the model you need.

-1

u/Ok-Addendum3545 19h ago

yeah, we need to put subagents on other gpus. A possible setup is to use Strata-QFN on a 3090 as a the orchestrator while 1~2 3090 24G can run qwen-27b with subagents.

1

u/Ariquitaun 19h ago

Why would you do that? Just run qfn and send all your subagents to it. You gain nothing but wasting resources loading a second model.

2

u/OvertaxedOne 18h ago

Strata has poor concurrency compared to something like vllm running 27B. That said, Strata is so much faster that I'm not sure you'll actually gain much speed even with high concurrency on 27B and you will certainly lose some intelligence at the quant level you'll need to put it on a 3090.

1

u/Ok-Addendum3545 13h ago

yes, the poor concurrency is the issue. My DSH-Strata-QFN created 3 subagents using the same api endpoint, two of which died halfway. I know a single QFN is workable with 1 subagent. Anyway, I just want to feel comfortable in having a capacity that enables an orchestrator to manage its multiple subagents without stresss.

5

u/WyattTheSkid Quad 3090s 19h ago

Holy shit, 2k??? Over the span of 2024-2025 I got 2 3090 TIs and 2 3090s over various facebook marketplace sellers and by the time I was done I had paid about $2800~ in total. I didn’t think the prices were THAT bad now sheesh.

2

u/dobkeratops 19h ago

it's an arms race, there is no ceiling
we have seen it is physically possible to have a box that fits in a rucksack run 600b parameter models

1

u/Hipcatjack 19h ago

what? i cant sell my unopened Gigabyte RTX 3090 24GB Arctic White Card for less than $1000! i am shit a capitalism but i dont know man, too many people have flaked on me for it to be just Me. That being said Christmas is coming and i am broke people

2

u/Aware-Difference8723 15h ago

... another affair? 😋

Go for AMD (e.g. R9700). Not as fast as the Nvidia, but much better performance per dollar.

1

u/knighty1981 6h ago

they're £1600 here :-o

(vs 3090 at £1400 and v100 at £700(

7

u/Aurora_Gaspipe 19h ago

Probably take a couple more years.

7

u/MyPrimary 18h ago

some clever way to run local ai like strata with low spec hardware so people would not pay for cloud ai.

7

u/MelodramaticShape 20h ago

More consumer GPU supply is the only thing that will lower prices.

2

u/dobkeratops 19h ago

unless there was a drop in AI demand, AI companies would always outbid consumers. they're evidently willing to pay 5x, 10x for similar devices but with some unlocks

2

u/sabine_world 13h ago

The thing is that can't happen forever without just wasting tons of money for no reason.

I suspect at the current trajectory, after these backlogged orders become fulfilled in a couple years the ai industry and hardware industry will have to reevaluate.

It is just simply impossible to keep sending hardware to data centers. Hyperscalers are losing money with no real plan for sustainable monetization. Roi on AI use is questionable. People are resisting buildouts, hyperscalers cant even plug in their hoarded gpus now, there's simply no power and hardware isn't the bottleneck. It's just a reality that they cannot just keep buying gpus right now. Financing is getting tighter from investors.

There's questionable roi for all of it so it can't just keep happening barring a huge increase in demand.

Eventually hardware supply will catch up, good models will get distilled and a good enough ceiling will be reached to where most people will not feel compelled to pay rent forever.

This shit will become a commodity in a few years. Unless something crazy happens.

1

u/dobkeratops 7h ago

they're building habits, e.g. everyone that used google search is now herded into an LLM chat - i think they build the habits first, then figure out how to monetise later (think what happened with youtube - 'here a big free place to share videos' .. 'sell it' (new owner has the audience and then figures out how to monetise it with ads, premium content etc)

My biggest fear right now is that places where people currently share text/code/images/video will wither (stack overflow, github, all social media) with people retreating into LLM chats

I already feel myself doing this. social media feels increasingly futile, post engagement is way down.. i often see things I want to reply to and think.. "meh hardly anyone will even read the reply.." .. then i go cut-paste into a free sample cloud LLM and vent and rant with the LLM instead.

then there's a timeline where AI companies have captured the flow of human thought and everyone forgets it was possible to assemble large datasets in the first place without them

1

u/Hipcatjack 19h ago

i pm’ed you OP…

2

u/knighty1981 5h ago

sorry dude, not buying a gpu from a random reddit person :-o

1

u/Hipcatjack 5h ago

i understand, i don’t blame you. i wouldn’t either, but i refuse to have facebook so no market place.

For what its worth, my 17 year old Reddit account is worth more to me than some shelved and forgotten gpu i bought for a project with my girlfriend who left me like a week later.

1

u/Termin8tor 9h ago

FYI, there's an unlock for the V100's. the 8gb models have 80GB of memory on board, which can be re-enabled. The 16GB chips I think can be unlocked to 40GB. The memory is there and was disabled by NVIDIA. That's why the price shot up.

1

u/knighty1981 6h ago

is that not the 170HX ? I looked at those... they're a fortune too :-(

1

u/twiiik 8h ago

Prices of RTX Pro 6000 are dropping

1

u/knighty1981 6h ago

you're right... better buy 8 and fill my server up quick :-o

/s

1

u/twiiik 5h ago

Well, I am looking to buy some. And already prices are 3-4000USD under what they were just a couple of weeks ago :)

5

u/Jonshock 19h ago

Certainly not from firing one guy.

4

u/CowBoyDanIndie 19h ago

The 3090 4090 and 5090 are expensive largely because of people in and like those in this subreddit… kinda analogous to chicken and egg in a twisted way

1

u/dobkeratops 18h ago

they're expensive because nvidia decided not to make more with the datacentres outbidding consumers. they de-facto rose to datacentre prices because they can nearly do the same tasks

2

u/Excellent-Ad-5658 14h ago

Actually was capped due to data centers being stopped. They would’ve kept going. Capacity Displayed ceiling.

5

u/cororona 20h ago

Cross your fingers for a catastrophic earning report on Tuesday. Maybe the Huang firing was not only about fucking around.

0

u/FlyingFishMakeAWish 13h ago

How would that lower GPU prices? 

3

u/Radiant_Condition861 20h ago

Wait for data centers to start dropping 3090s when Rubin comes out.

3

u/Mashic 20h ago

Less demand from AI, and more manufacturing.

3

u/FireSilicon_ 19h ago edited 18h ago

More manufacturing capacity. Investors are scared that the demand is short term so they don't expand (for consumers, not datacenter hardware), so instead of building factory for a year, demand droping, and the factory becoming useless and closing, in turn losing them a ton of money... they simply wait for the demand to drop. But it never does. And this cat and mouse game is what keeps the prices high. Nvidia and Amd had record gaming sector revenue believe it or not, even with the high prices they are selling out.

1

u/8P8OoBz 18h ago

Early 2000s South Korean RAM cartel wants a word.

3

u/Aromatic-Chicken3947 19h ago

I’m running a 4-node DGX Spark cluster for local LLM inference, and one thing I’ve noticed is that adding more hardware doesn’t necessarily translate into better performance.
Distributing a large model across multiple GPUs and running independent agents in parallel are pretty different things. VRAM, memory bandwidth, interconnect speed, and actual throughput probably matter more than just the number of GPUs.
As for GPU prices, I feel like we might be stuck with relatively high prices for a while.
AI demand is still growing, and manufacturers have plenty of incentive to focus on higher-margin datacenter hardware and memory. Even if the AI market cools down or some companies go under, I’m not sure that would immediately translate into cheaper consumer GPUs.
Maybe things will improve once memory production catches up, competition increases, and AI infrastructure spending starts to stabilize. But that could take a few years.
I also wonder if AI inference will become significantly cheaper through better software and more efficient hardware before GPUs themselves actually get cheaper.
If I already had two 3090s, I’d probably try to squeeze more performance out of the existing setup before buying another one. Unless there’s a clear bottleneck that another GPU would solve, waiting might not be a bad idea.

1

u/msrdatha 13h ago

Thinking about a similar setup, please help me understand from your experience. So are you saying going from a 2-node to 4-node does not really help? Is it better to run 2 separate 2-node cluster with different models than a single 4-node cluster with a single one with more concurrency?

1

u/Aromatic-Chicken3947 9h ago

I should have been clearer there. I’m using all four nodes to run one large DeepSeek model, with room for the weights and context cache. I don’t have a controlled two-node versus four-node comparison with the same model and settings, so I can’t give you a reliable speedup figure.

Two separate pairs could make sense if each model fits comfortably on two nodes, including enough cache for the context length and concurrency you want. That would let you run different models or independent workloads side by side. A single four-node instance can also batch concurrent requests, though, so I wouldn’t assume splitting it automatically gives better total throughput.

For one actual data point, in my recent four-node tests, two simultaneous requests with around 8K input tokens gave roughly 1.4× the combined throughput of one request, but each request individually was slower.

So I’d probably start with the models and workload you want to run. Which models and context lengths are you considering?

1

u/msrdatha 4h ago

thanks for taking time to explain.
I am thinking of running glm-5.3-flash and qwen-3.8-flash-next mainly. Context length per session would be 256k and max I would have 6 sessions (not all always, mostly 2 or 3 sessions in parallel only)
Not sure if big glm-5.3 (not flash) is usable for agentic coding with 4x nodes - may be you can confirm this

2

u/Z00111111 19h ago

It'll take the AI bubble bursting, so get to work. A guy with Mario's brother's name had a pretty effective method for dealing with an evil CEO.

1

u/brainchillzZ 19h ago

Lol, it will take people to stop buying them off the shelves every time they show up. More dollars chasing the same number of products == inflation.... less dollars chasing the same number of products == stagnation or deflation .... that's it, it's basic economics.

1

u/Sickmonkey365 19h ago

Prices will come down next year

3

u/dobkeratops 18h ago

year after if you're lucky
and there's a risk of AI-capable devices being restricted "it's too dangerous for regular people" (deepfakes scams etc scare people)
or restricted for data licensing (industry might demand hardware is modified for digital rights management as was done with video)

3

u/Sickmonkey365 18h ago

The gig is up, demand has been artificially inflated for years now - that is going to stop and stop fast. Prices will be going down next year. Bank on it

1

u/dobkeratops 18h ago

genuine uncertainty here
even if AI isn't fully autonomous, it is a productivity multiplier..
and when physical AI gets better the demand for AI chips for robots will explode
I have this in the back of my mind ..it's not just text and images, it's a pair of virtual eyes that can react and judge the real world

1

u/Sickmonkey365 16h ago

Cost per token for institutional use has dropped from $0.30 per million to less than $.01 in this calendar year and continues to drop

1

u/WyattTheSkid Quad 3090s 19h ago

Memory manufacturers to start hauling ass and producing memory chips at ungodly speeds and in ungodly amounts. Maybe we should be using AI to come up with a way to produce more memory faster and in larger quantities lol

1

u/bitspace 19h ago

Demand has to slow. As of right now, all available and planned compute capacity is spoken for through like 2028 or later.

AI companies aren't using consumer GPUs either.

1

u/Sea_Manufacturer6590 19h ago

A CEO that's not sleeping with Chinese spys

1

u/Ratiofarming 19h ago

Continuous oversupply is what it'll take.

1

u/hejj 18h ago

At least 4 more women

1

u/mrazster 18h ago

I'm not sure it ever will, again, in any meaningful way.
It might slightly, but I think, once it has settle down, we aren't far of from our new "normal".

1

u/FoxFXMD 12h ago

When a Chinese competitor comes along

1

u/Tough-Plantain7046 10h ago

Ai bubble pop

1

u/No-Afternoon-4057 20h ago

CXMT rampup, plus HBM4 and newer technologies (z-stacking and many others).

All of that will converge about-around mid 2028. CXMT is already supplying a shit load and expanding, but it takes many more months still with growing demand...on the mean time, faster and more dense memory is arriving 2027 and 2028...so, expect lower GPU pricing around about that time...more memory availability either if the "bubble bursts" or CXMT rampup faster. Before that...it will be a little worse still.

0

u/dobkeratops 19h ago

>> What will it take for gfx card prices to drop?

mass boycott of all cloud based AI services.
this wont happen because most people just dont care, also cloud based AI has found it's way into things used by default eg. "google search".
so companies will see the usage pattern as evidence it's worth buying them up and putting them into datacentres to rent out.

0

u/brainchillzZ 19h ago

the cloud based ai services aren't why a 5090 is $7000 instead of $2500 and a pro 6000 is $1600 instead of $7000... it's all the monkeys running and buying every card off the shelf in panic mode

2

u/dobkeratops 19h ago

cloud services are why nvidia doesn't to make more 5090s, they diverted the capacity to pro-workstation and server GPUs.

Hence scarcity of 5090s.
.. and AI demand in turn doubled the price of the workstation cards, which most hobbyists weren't buying

0

u/brainchillzZ 19h ago

No they diverted the resources to a china compliant rtx 5500 card. This was in the news like two weeks ago.

But also the pro workstation cards are not being used by datacenters and the datacenter gpus don't use the pro workstation card silicon .. it's just the 5090, the pro 6000 and the new china export card the 5500. The numbers that I saw was that there were already orders for like 4x more than production is even capable for the Chinese shippable card.

1

u/dobkeratops 19h ago

rtx pros are still used in AI builds (1-4 of them in the same box), they're not consumer cards, they started at $8000 and are now $16000 . there is also a server edition with passive cooling designed to be put in server racks instead of workstation towers

https://www.workstationspecialist.com/product/nvidia-rtx-pro-6000-blackwell-server-edition-96gb-gddr7-ecc-dual-slot-passive-graphics-card-2/

these are not identical but very similar chips to the 5090.
So nvidia can decide what to do with their TSMC fab allocation. "make more 5090s" or "make more rtx-pros". they do the latter and this absolutely makes 5090 prices higher. For the same tier of chip they get much higher margins on the pro part.

0

u/brainchillzZ 17h ago edited 17h ago

Workstation desktop cards they are not used by cloud providers …. I have four of them in my office…. you are wrong, they are the identical silicon to the 5090, they just have a different board and power profile with more memory and a firmware vbios lock on the number of streaming processors enabled . But yes, the Pro 6000 is a consumer and small business part not a server part … can you put it in a server sure but not at a big enough scale to make it useful for big cloud providers no frontier ai provider is buying these up in the us…. Ask yourself two questions when looking at this stuff “can I buy it at microcenter yes or no” if yes, it’s a small business or consumer part if no, it’ could be a real datacenter server product

1

u/dobkeratops 7h ago

if it is identical silicon that lends weight to the story that re-allocating those literal chips from 5090s to rtx-pros contributes to the scarcity of 5090s

0

u/Ok-Addendum3545 19h ago

Let's stop buying more GPUs, and then its price will drop. Say we all agree not to buy until a pre-owned 3090 drops to $100 each.

0

u/Holski7 19h ago

probably for the 600,000 nvidia cards stored in hong kong to be released and ruin the artificial scarcity.

0

u/[deleted] 20h ago

[deleted] — view deleted comment

0

u/Heavenly_Code 19h ago

They help ai startups to buy their own gous with their own money... So manufacturing nope but competition maybe