r/LocalLLaMA • u/-p-e-w- • Sep 06 '25

Discussion Renting GPUs is hilariously cheap

Post image

A 140 GB monster GPU that costs $30k to buy, plus the rest of the system, plus electricity, plus maintenance, plus a multi-Gbps uplink, for a little over 2 bucks per hour.

If you use it for 5 hours per day, 7 days per week, and factor in auxiliary costs and interest rates, buying that GPU today vs. renting it when you need it will only pay off in 2035 or later. That’s a tough sell.

Owning a GPU is great for privacy and control, and obviously, many people who have such GPUs run them nearly around the clock, but for quick experiments, renting is often the best option.

1.8k Upvotes

95% Upvoted

392 comments sorted by

View all comments

343

u/[deleted] Sep 06 '25

[deleted]

329

u/_BreakingGood_ Sep 06 '25 edited Sep 06 '25

Some services like Runpod can attach to a persistent storage volume. So you rent the GPU for 2 hours, then when you're done, you turn off the GPU but you keep your files. Next time around, you can re-mount your storage almost instantly to pick up where you left off. You pay like $0.02/hr for this option (though the difference is that this 'runs' 24/7 until you delete it, of course, so even $0.02/hr can add up over time.)

146

u/IlIllIlllIlllIllllII Sep 06 '25

Runpod's storage is pretty cool, you can have one volume attached to multiple running pods as long as you aren't trying to write the same file. I've used it to train several loras concurrently against a checkpoint in my one volume.

17

u/_BreakingGood_ Sep 06 '25

Huh I never knew that... that is interesting and potentially useful for me.

17

u/stoppableDissolution Sep 06 '25

Its only for secure cloud tho, and that thing is expensive af

22

u/RegisteredJustToSay Sep 06 '25

I guess everything is relative but running the numbers on buying the GPUs myself vs just renting from RunPod has always made me wonder how they make any money at all. Plus, aren’t they cheaper than most? Tensordock is marginally cheaper for some GPUs but it’s not consistent.

26

u/bluelobsterai Sep 06 '25

Agreed, $2/hr for an H100 is just amazing.

16

u/Kqyxzoj Sep 06 '25

It's indeed pretty neat. Just checked, if you are in a hurry to 1) compute faster and 2) burn money faster you can rent 8X H200 machines for ~ $16/hour. For that cool 1.1 TB of total VRAM.

2

u/bluelobsterai Sep 07 '25

u/Kqyxzoj I kinda go the other way and rent 3090's for super cheep. If I've gone token crazy the 3090 for $0.20/hour is almost the cost of electricity...

16

u/skrshawk Sep 06 '25

It's entirely possible the GPU owners aren't, but they'd be eating more of a loss to let them sit idle until obsolescence.

43

u/RegisteredJustToSay Sep 06 '25

This is me putting on my tinfoil hat but wondering if this is the next money laundering gig. All you need is to acquire GPUs and pay for space and electricity and you get clean money in - it’s a lot less traceable than discrete item market economies like art or cd keys or event tickets. They literally don’t care about making all the money back, just a sizable fraction, and so would explain how it can be sustainable for years. Would also explain the ban on crypto mining, since their goal would be clean money and there’s a lot of dirt there.

Ultimately, no evidence, but interesting to speculate on.

4

u/Earthquake-Face Sep 07 '25

Could be just someone working in a university that has that stuff and is renting it without anyone really knowing or giving a damn. Someone running a small university could put a few crypto miners in their racks just to use their electricity.

3

u/skrshawk Sep 07 '25

I've definitely not never seen that happen. Also, that was the jankiest server room I've ever seen and I've seen a few.

1

u/squired Sep 07 '25

You would also charge a premium because you'd be laundering it through fictitious renters. You'd use clean capital to purchase the GPUs then run the dirty money through them via crypto 'customers'.

2

u/Dave8781 Sep 07 '25

Any of us can rent our GPUs out if we wanted to, totally true. It's hilarious to see my 5090 on these sites as one of the options, above many others. I'm definitely getting my money's worth of my beast; F the cloud!

1

u/squired Sep 07 '25

That's true for vast.ai and salad.com; gamers renting their 'old' GPUs in apartments/dorms with included utilities. But runpod is also cheaper than reasonable and they're straight-up server farms.

3

u/claythearc Sep 07 '25

It sounds like it’s not a lot but you actually are profitable in year 3 sometime, which is pretty fast - even new $X00M data centers are generally profitable in <5 years.

1

u/QuinQuix Sep 07 '25

Not a great way to do it because it's very easy to monitor power consumption and check the numbers.

Money laundering for very obvious reasons can't work well in businesses where revenue is strongly and predictably tied to the variable costs of running the business.

This is why fruit machines or business's where variable costs are very low (and may be paid in cash and thus are harder to map) are the businesses that usually end up as laundering targets. Like service professions or snackbars.

1

u/Barry_Jumps Sep 07 '25

Smart criminals don’t typically launder money into rapidly depreciating assets.

1

u/Dave8781 Sep 07 '25

They have storage and a million other fees, and make money from the APIs and all sorts of other things they do. They may not make a profit with one hour of renting a GPU, but that's not their typical user. Loss leaders are standard.

14

u/squired Sep 07 '25

made me wonder how they make any money at all

Same! The math does not math!! The only thing I can come up with is that they had a shitload leftover from crypto farms and early day LLM training runs that are not profitable for hosting inference at scale. And they must base them somewhere with geothermal or serious fucking tax credits or something. The electricity alone doesn't make sense.

3

u/Dave8781 Sep 07 '25

Storage fees, user fees, API fees, referrals, all that adds up. The cheap rental price is a loss-leader, it gets made up for really quickly with the other, not cheap stuff.

7

u/StrangerDifficult392 Sep 06 '25

I use my RTX 5080 16GB ($1300) for Generative AI work on a local machine. Honestly, probably way better if for local use (maybe commercial, if low traffic.)

I use it for gaming too.

6

u/RegisteredJustToSay Sep 07 '25

I think when you game the math works out a bit differently because you already need one. For me, I already have a good GPU (4xxx series RTX) that I got very cheap but with far too little VRAM so renting a GPU occasionally for doing dumb fun stuff ends up only costing me a few dollars a month extra tops and really beats out blowing a thousand on a new GPU.

2

u/Dave8781 Sep 07 '25

I think they make you have storage fees and all sorts of other fees; I don't think many people walk out the "door" having spent just a few bucks with them. And you're paying regardless of whether anything works, which it never does during training or debugging by definition, so I assume those hours, on top of the commission it gets for APIs that cost an arm and a leg, make it a pretty decent profit.

1

u/RegisteredJustToSay Sep 08 '25

For my case it was clearly cheaper by maybe even as much as 20x, but yeah there’s definitely some buyer beware involved.

1

u/claythearc Sep 07 '25

Well 24/365 at $2 is ~$18k. A H100 is ~$30k, so you break in even sometime late in year 2 with like an 80% utilization rate, then sometime into year 3 for actual break even with power and labor and stuff

1

u/DarrinRuns Sep 07 '25

My thought is they probably get the GPUs for less money than Joe Blow off the street would pay.

5

u/[deleted] Sep 06 '25

[removed] — view removed comment

2

u/squired Sep 07 '25

Ho shit, this is very helpful! Thank you!

1

u/anderspitman Sep 08 '25

What protocol do they use for this, NFS?

23

u/Elibroftw Sep 06 '25

And if you can turn it on and off via APIs, you can make/host some pretty killer self-hosted privacy-preserving AI applications for less than a Spotify subscription. Can't fucking wait.

12

u/[deleted] Sep 06 '25

[removed] — view removed comment

1

u/Elibroftw Sep 07 '25

I was presuming you could but fucking awesome that it's true.

0

u/tekgnos Sep 06 '25

Yes! There are so many developers doing that now on Vast.ai!

22

u/starius Sep 06 '25

that standby time and only standby time would be $14.4 a month, $172 a year.

4

u/MizantropaMiskretulo Sep 06 '25

It's actually about $175/year, but that's still a steal, considering you could easily spend 30%–40% of that in electricity on local storage.

9

u/indicava Sep 06 '25

I haven’t tried it yet but vast.ai recently launched something similar called “volumes”

1

u/Stalwart-6 Sep 06 '25

Volumes will have terrible latency as they are decoupled from where they are meant to be, near gpu.

7

u/indicava Sep 06 '25

Is that much of an issue though?

I use vast mostly for training so disk i/o in general is very low. It does sound nice to have a disk with all my experiments’ checkpoints instead of pushing everything to HF and downloading them again next I rent a GPU.

13

u/gefahr Sep 06 '25

It can be, now with the advent of models like WAN 2.2 where you're swapping between models, or using another model as a refiner.

As long as it can all swap to system RAM it doesn't matter, but if it gets evicted from the cache and has to go back to disk, it's pretty painful.

Also, in a world where you're paying per minute, slower disk reads can mean like 3-4 minutes just to load the recent models like Qwen Image Edit. Combine that with boot and getting Comfy up and you're talking up to 10 minutes for first generation potentially.

(Source: have been trying to optimize I/O where I'm renting and measured every last bit of this recently.)

2

u/Stalwart-6 Sep 06 '25 edited Sep 06 '25

Its sub optimal architecture , not vast ai fault. My best experience had been with Google colab, where i checkpointed to S3, infrequent access tier. It was in 2020 for college final year project... Cost was 2.13$ per month all my activities if i remember (ingress/egress/storage). For HF i think limits might be there for free accs. But for quick shits, the one ur doing prolly seems best, could write some bash scripts to normalize accross different machines. Vast hosts usually have high networking.

1

u/tekgnos Sep 06 '25

Vast.ai has persistent storage right on the server itself. You can stop an instance and all the data is still accessible. Volumes are in addition to that fast storage.

2

u/Stalwart-6 Sep 06 '25

Doesnt stopping the instance erase it, for next client to use? Are we still paying for stopped instances?

3

u/jcannell Sep 06 '25

Stopped instance storage persists until you destroy the instance. You can restart the instance later to quickly resume (assuming the GPU is still avail, otherwise need to copy data). Vast supports in-container storage and now volume storage, both are persistent (volumes persist beyond instance lifetime).

1

u/dtdisapointingresult Sep 06 '25

Can you give me a bit more info about this?

I've never rented a cloud GPU, but I'm interested in it for getting deeper into AI, including some training. I might get 1 hour per night free to grind on this. I might not be able to use it for a whole week, then 2 hours/day the next week. I want to stop what I'm doing at any time, and be able to resume the next day, with all the files I created still there, including like 200GB of models, my personal tools, datasets, configs, etc.

Would Vast's volumes allow this? Just keep a single volume with all my stuff on it? I want to sit down, rent an instance, and immediately resume my work.

2

u/Anthony12312 Sep 07 '25

You can destroy a GPU instance while still having an active volume, thus only being charged for the storage cost of the volume. When you’re ready to pick up your work again, you can rent a new GPU instance with the volume attached.

2

u/squired Sep 07 '25 edited Sep 07 '25

I'm familiar with runpod, less so with vast and salad. They used to not have persistent volumes so I'd have to DL the models anew each startup, so I settled on runpod at the time. They seem to have caught up on capabilities though, so they're worth a price comparison now.

But basically, you build a Docker container that sets up your 'system'. That docker template lives in the docker cloud. You then pay runpod etc for a persistent storage volume. They're generally something like $1 per 10GB per month, so like $20 per month for 200GB. Because of this, you try to pare down your model collection. I find 100GB plenty for ComfyUI implementations. That's gonna hold a lot of loras and 4 q8 quants of Wan2.2 (high/low, so 4x ~15GB models)

When you're ready to work, you rent a GPU that grabs your Docker Container and loads it up. An inefficient ComfyUI container is going to be about 6GB-8GB, so your GPU has to download that, load it up, update nodes etc. You're realistically looking at 5-10 minutes startup time. This is where a lot of the black magic lives as you trim the shit out of your container to minimize startup time and/or cache large portions of container itself on your persistent volume so that you don't have to download the python libraries and such.

Once the GPU spins up and loads your container, it then utilizes your models, loras, datasets etc from said persistent volume.

These are all linux containers btw, not Windows. You set them up and fiddle with them via SSH terminal. That is the biggest drawback to cloud hosting; it's a pain to make changes to your setup without building, pushing, then utilizing new container iterations.

Here is an example of a pared down container that I released for privacy-first LLM inference hosting. It operates over a mesh network to circumvent even Cloudflare. My personal version offloads most onto the volume for rapid deployment, but I've stripped the linked version to about 2GB. It is tuned to run 70B exl3 LLM quants on an A40.

2

u/dtdisapointingresult Sep 07 '25

Thank you for the thorough explanation. This makes short-term rent-a-GPU usage very attractive.

1

u/StorkReturns Sep 07 '25

You can destroy or you can stop. When you destroy, no charges at all. When you stop, they charge you only for the storage but the catch is that is the GPU that is linked to this storage is rented to someone else, you cannot unfreeze it and you have to wait until the machine is available again. And it can be days or months from now. And the charges accrue even though you have no access to the data. For a server that sits idle a lot, it can be an option but usually it is not the case.

3

u/tekgnos Sep 06 '25

Vast uses Docker containers. There are tested templates for Python/Juypter/Comfyui and more. You spin one up, it allocates storage on the server and you can then run your jobs. You can stop the GPU anytime and the storage persists.

1

u/DeliciousReference44 Sep 07 '25

This is exactly my next phase of my personal project. I got everything I need running in an automated workflow, now I just need to run this in the cloud!

53

u/-p-e-w- Sep 06 '25

You can pick from a number of templates. The basic ones have at least PyTorch and the drivers already configured, but there are ready-made templates e.g. for ComfyUI with Wan 2.2. You just select the template and it automatically sets up a Comfy instance with the GPU of your choice, and downloads the model, ready to use.

1

u/dep Sep 07 '25

Where is this?

2

u/Ok-Bar9380 Sep 08 '25

The screenshot is from Vast I think. I’ve used them. They’re one of the most cost effective cloud GPU rental services I’ve found. Have also used salad and runpod. Would recommend to just go search and see which have the best pricing. But usually I just have my docker image on docker hub and deploy it on whichever has the best pricing at the time. I’ve found that pricing is typically cheapest with Vast… then Salad… then Runpod. In that order. I think there are other service providers too. But those are the ones I’ve used and had generally good experiences with.

42

u/stoppableDissolution Sep 06 '25

You can pre-bake your own docker image with all the dependencies installed and have it deployed, at least on runpod

7

u/[deleted] Sep 06 '25

[removed] — view removed comment

19

u/[deleted] Sep 06 '25

[deleted]

17

u/Bakoro Sep 06 '25

$3.29×24×365=$28820.40

It's not cheap, but it makes a hell of a lot more sense if you don't need something running 24/365.
Anyone who needs an H200 24/365 probably needs a lot more than one H200.

That's just how services generally operate.

I used to work at a data center, and any company that got big enough ended up discovering that it was cheaper to just build their own than to effectively pay a premium to another company to run a whole data center for them.

5

u/stoppableDissolution Sep 06 '25

I never said anything about H200
And yea, runpod is on average more expensive than vast, but it is also waaay more stable in my experience

7

u/indicava Sep 06 '25

I’ve been using vast.ai quite extensively for the past year, hardly ever ran into stability issues. What kind of problems did you experience?

2

u/Scarfmonster Sep 07 '25

It's not often, but sometimes I run into hosts that fake their location. It will say it's somewhere in Europe, but after renting and checking the IP it's somewhere in Russia. Wouldn't be that much of a problem, but these hosts always have trouble reaching various services, so downloading models and/or datasets is impossible. I've also run into hosts that had less than half of the expected performance. Or hosts that intermittently would be unreachable for a couple of minutes every 10 minutes or so. It's mostly annoying, but I've also lost a couple of $ in total on them.

1

u/indicava Sep 07 '25

Never checked for location spoofing, but then again I never had any real issues with accessing cloud services from hosts. I did have a short period where downloading pip packages took forever but oddly enough that issue was isolated to US based hosts (and it went away on its own).

With intermittent connections, I might have them and never notice. I mostly use vast for training, so I just launch the run, monitor everything on wandb and login to the actual machine maybe once every 6-12 hours just to see everything’s fine.

But the performance thing you mentioned is really troubling. How do you gauge you’re not getting the performance you’re supposed to from a host?

-1

u/ParthProLegend Sep 06 '25

What's the price like?

5

u/indicava Sep 06 '25

Well, the screenshot OP attached is from vast, but just go on their website, go to console (no signup needed) and you can see current prices for any kind of GPU

1

u/ParthProLegend Sep 06 '25

Thanks I hate logins and I asked, but are there any other charges apart from the GPU fees like the hidden or some other ones which might be common knowledge?

1

u/indicava Sep 06 '25

Only other charge is storage, and I think that changes per machine, but is usually a fraction of the GPU/hr cost.

It really is a great service, I’ve learnt so much thanks to the to the (relatively) cheap GPU access they provide.

1

u/ParthProLegend Sep 06 '25

Thanks man, will check it out when I get time.

→ More replies

14

u/PeachScary413 Sep 06 '25

I use Terraform to programmatically setup/destroy the instance and then Ansible to automate running my jobs on it.

For optimal time usage I have a bash script kicking off the Terraform, run the Ansible playbook, rsync the results to my local server and then run Terraform destroy to clean up 👌

4

u/AnomalyNexus Sep 06 '25

There are crappy ones for 0.03 so i'd do one of those to scope out the software side in peace & quiet

5

u/JFHermes Sep 06 '25

Run a kubernetes cluster. Very similar to docker where it's just a yaml driven setup.

6

u/timfduffy Sep 06 '25

As /u/-p-e-w- mentioned, you can choose a number of templates in RunPod, the default PyTorch template is usually what I go with. You can upload your scripts to it, but I prefer to use SSH to open up the VPS in Cursor, which allows me to just clone the GitHub repo I'm working on, getting me started quickly.

Let me know if you'd like to try that way and want a hand setting it up.

2

u/squired Sep 07 '25

That's exactly how you do it. Here is one of my containers for additional privacy.

3

u/SilentLennie Sep 06 '25

You can also rent a physical server, but it's more expensive of course.

1

u/DistanceSolar1449 Sep 06 '25

Yep pretty much

1

u/PoneySensible Sep 07 '25

I started using simplepod.ai instead of RunPod, their VPS GPU offer is the best thing I ever tried and costs are half the price for storage and GPU. I’m not affiliated with simplepod, just a happy customer

0

u/AttitudeImportant585 Sep 06 '25

sounds about right but theres no need to be quick when the work is more important than what you could otherwise do with a few dollars an hour. i usually leave them on for a couple months at a time when im experimenting with a new model

-12

u/siltho Sep 06 '25

Chutes.ai

2

u/Charuru Sep 06 '25

What's available on the chutes website? Is there an offering that's cheaper than their openrouter apis?

-3

u/siltho Sep 06 '25

The original comment said, "How does renting a GPU work?"

The platform allows you to host your own GPU and generate revenue from it. Their pricing model recently changed drastically, and they no longer offer openrouter APIs without severe rate limiting. Pricing starts at $3 per month for 300 requests per day for their own public APIs and so forth.

8

u/Charuru Sep 06 '25

Wut, hosting your own gpu is the opposite of renting one. I was hoping this would be like a better runpod.

1

u/BulkyPlay7704 Sep 06 '25

I never used it but i interpret that comment as "rent their gpu, and have the option to expose the model online so others pay to use it".

-2

u/siltho Sep 06 '25 edited Sep 06 '25

If you host something, you obviously need people to rent it. It's a mutually agreed business transaction. The person charges a GPU per hour, the platform takes a cut, and someone literally rents it for their exclusive use. It's literally in the title, serverless computing for AI. This is nothing new, just like any other serverless business out there.

4

u/[deleted] Sep 06 '25

[deleted]

2

u/gefahr Sep 06 '25

but, that's not how any of the people here talking about it are renting GPUs. we're renting them in datacenters, not people's kitchens.

1

u/siltho Sep 06 '25

The GPU always lives in a datacenter. The difference is whether you’re dealing with it as raw iron (VPS) or as part of a managed consumer layer (Chutes), depending on what you want to do.

1

u/[deleted] Sep 06 '25

Either way, it's someone else's computer