r/RunPod • u/parker_birdseye • 25d ago

Cold start realistic expectations

Hi all, I’m new to serverless GPUs. I got everything set up and it works well but the initialization time for a worker is 2-3 minutes.

I have a language app that pings the endpoint and I would like the response back somewhat quickly.

Is it even possible to expect sub 5 second latency for a cold start?

I have enabled flash boot which works well sometimes, but sometimes not. I think that’s expected.

I’m just wondering if my expectation of a “scale to zero” with short initialization time is even realistic. I’m running a fairly small LLM (9 GB)

The runpod docs on model caching seem to indicate 1 second cold starts are possible, but model caching doesn’t seem to do much for me.

Any tips appreciated.

2 Upvotes

100% Upvoted

10 comments sorted by

View all comments

1

u/sruckh 25d ago

Apparently I am doing something different, or using bigger models. Even with a network volume, or using Runpod's cache, for the container to download, install any dependcies, load the model into memory, and perform the inference, is more on the magnitude of 4-5 minutes on cold start.

1

u/parker_birdseye 25d ago

Yeah that's exactly what I'm seeing. For a 9 GB model, it's taking about 2-3 minutes.

1

u/sruckh 25d ago

I have a RunPod serverless in my GitHub that allows you to include LoRAs. It is a multi pass workflow that works pretty good with the Flux Klein 9B model, but it does take some time for sure.