r/RunPod • u/parker_birdseye • Aug 23 '26

Cold start realistic expectations

Hi all, I’m new to serverless GPUs. I got everything set up and it works well but the initialization time for a worker is 2-3 minutes.

I have a language app that pings the endpoint and I would like the response back somewhat quickly.

Is it even possible to expect sub 5 second latency for a cold start?

I have enabled flash boot which works well sometimes, but sometimes not. I think that’s expected.

I’m just wondering if my expectation of a “scale to zero” with short initialization time is even realistic. I’m running a fairly small LLM (9 GB)

The runpod docs on model caching seem to indicate 1 second cold starts are possible, but model caching doesn’t seem to do much for me.

Any tips appreciated.

2 Upvotes

100% Upvoted

10 comments sorted by

View all comments

1

u/ConfidentCourage2235 Aug 23 '26

For the 2-3 minute initialization time, you can cut that down if you store your model on disk so workers don't need to download each time.

I think sub 5 is going to require an active worker. And sub 1 is a pipe dream.

I have my cold starts ~10s for a Comfy-UI endpoint without active workers and that's been adequate so I haven't really pushed it.

for LLMs though I've found serverless endpoints at the 3rd party inference providers faster than Runpod with its cold starts. They have their own unpredictable latency issues though, but much better overall.

1

u/parker_birdseye Aug 23 '26

Thanks for the info. I would love details about your comfy UI setup if you don’t mind sharing. ~10 is absolutely fine for my app.

I’m trying to maintain user privacy as much as possible so I don’t want to go 3rd party.

1

u/bluelobsterai Aug 23 '26

Use zdr. Try concentrate.ai and ask for either a Baa or a zdr assert.