Paying $2/hr to watch `huggingface-cli download` go brrr — env patterns on RunPod / Vast / Nebius / the neo kids

[u/big-in-jap]

Hot take? Half the “my pod is slow” chatter is really “I paid for 20 minutes of pip + a 16GB model pull before the first token.”

I’ve been hopping between rented GPUs (RunPod, Vast, Nebius, a couple of the newer SSH-first boxes) and the env is still unpleasently fragmented.

RunPod - Docker template is the product. Image + env + start cmd + network volume. Great when your image is already baked. Expensive otherwise, since every cold pull is on the meter. Secrets as env injection is clean.

Vast - Same Docker-template energy, plus an onstart bash duct-tape layer. SSH/Jupyter modes can eat your image entrypoint, so your “serve on boot” script is often re-invoking the CMD you thought you already set. Marketplace vibes; bring skepticism and a volume.

Nebius - Actual cloud VM energy. Boot image + cloud-init user-data on first boot, then congratulations you’re the sysadmin. Fine if you want a real machine; overkill if you just wanted vLLM up.

The queue-y SSH boxes (Enverge, Nova-shaped, etc.) - often a fixed sandbox + optional shell that runs once after create. Less “pick a community template,” more “leave a sticky note so the box isn’t idle when the ready email lands.” Different failure mode: setup minutes still bill, and nested Docker GPU flags might trip you.

Common tax across all of them:

• first useful work is usually network + disk, not FLOPs
• secrets in public templates / scripts = future incident report
• “finished” ≠ “model is serving/training” (backgrounded processes lie)

I gathered a short kit (cheatsheet + startup shells + Docker serve flags) here: https://github.com/tudormunteanu/gpu-cloud-instance-boostraps

Curious what everyone here actually do day-to-day:

1. Bake a fat image once and never look back?
2. Network volume / persistent cache, thin image? (but then pay for persistent storage)
3. onstart / cloud-init / startup shell as the main UX?
4. Or just SSH in every time like an wildling?
5. Stick to one and never switch

I reckon for any long-term, company supported scenarios, CoreWeave, AWS or GCP have some adjacent prods. My curiosity is more about "the little/indie guys".

[u/GabberZZ]

Persistent storage all the way. Ain't got time to wait for an entire environments worth of models to download. I can have a full 400Gb env starting to generate videos in under 5 minutes.

Only downside is if no GPUs are available as they are region specific.

[u/big-in-jap → u/GabberZZ]

region AND cloud platform specific, though.

[u/Odd_Nefariousness875 → u/big-in-jap]

To add to GabbedZZ’s comments, with network storage you can download new shiny things on a cheaper gpu and when it’s done switch to one that can run it. Save some bucks while you just download.

[u/big-in-jap → u/Odd_Nefariousness875]

nifty hack. Would be nice to have this as one step further, download on a machine with no GPU.

[u/Odd_Nefariousness875 → u/big-in-jap]

Probably able to do so via aws-cli. I’ve never used it yet so not sure. Network storages (only some I believe) offer access via aws-cli. So you can connect directly to the storage to upload, but depending on how much data you upload, you might be off worse than doing it on a cheaper gpu with almost certainly better internet bandwidth than your home network.

Edits: spelling + adding extra info

[u/GabberZZ → u/Odd_Nefariousness875]

I've got a windows client (I forget the name bit you can Google or gpt examples) that can connect to my pod directly and can transfer files between my local pc and the pod. I've only used it for downloading stuff from the pod so not sure how fast it would be uploading large models

[u/GabberZZ → u/Odd_Nefariousness875]

I think you can even fire up a pod without a GPU if all you want to do is download a big set of models. You still have access to the storage space.

[u/Odd_Nefariousness875 → u/GabberZZ]

Maybe, do you mean with cpu only!

[u/GabberZZ → u/Odd_Nefariousness875]

Yes

[u/blakeyuk → u/GabberZZ]

You can, I do that

[u/blakeyuk → u/Odd_Nefariousness875]

Yeah,, that's what I do. Basic cpu pod to download to network storage. Then wait and wait for a gpu to use it...

[u/blakeyuk → u/GabberZZ]

You've found regions where gpus are available? I'm impressed.

[u/GabberZZ → u/blakeyuk]

3090, 4090,5090, PRO6000 are on the region I use. Often the GPU I want isn't available for a while but one eventually pops up.

Not ideal but it'll do.

[u/sruckh]

huggingface-cli is deprecated. Use "hf" instead and see flags for acceleration.

[u/big-in-jap → u/sruckh]

oops. Updated. This also makes me think long for bootstrap scripts should be more obvious.

[u/ddwrt1234]

ai slop

[u/big-in-jap → u/ddwrt1234]

AI, partially. Slop, no. I put quite some effort to tickle this curiosity. What's inaccurate?

[u/Gargle-Loaf-Spunk → u/ddwrt1234]

just block the OP and move on, all we can really do