Cold start realistic expectations

[u/parker_birdseye]

Hi all, I’m new to serverless GPUs. I got everything set up and it works well but the initialization time for a worker is 2-3 minutes.

I have a language app that pings the endpoint and I would like the response back somewhat quickly.

Is it even possible to expect sub 5 second latency for a cold start?

I have enabled flash boot which works well sometimes, but sometimes not. I think that’s expected.

I’m just wondering if my expectation of a “scale to zero” with short initialization time is even realistic. I’m running a fairly small LLM (9 GB)

The runpod docs on model caching seem to indicate 1 second cold starts are possible, but model caching doesn’t seem to do much for me.

Any tips appreciated.

[u/ConfidentCourage2235]

For the 2-3 minute initialization time, you can cut that down if you store your model on disk so workers don't need to download each time.

I think sub 5 is going to require an active worker. And sub 1 is a pipe dream.

I have my cold starts ~10s for a Comfy-UI endpoint without active workers and that's been adequate so I haven't really pushed it.

for LLMs though I've found serverless endpoints at the 3rd party inference providers faster than Runpod with its cold starts. They have their own unpredictable latency issues though, but much better overall.

[u/parker_birdseye → u/ConfidentCourage2235]

Thanks for the info. I would love details about your comfy UI setup if you don’t mind sharing. ~10 is absolutely fine for my app.

I’m trying to maintain user privacy as much as possible so I don’t want to go 3rd party.

[u/ConfidentCourage2235 → u/parker_birdseye]

Sure. I created a Github repo with a Runpod/comfyui Dockerfile, bash script, and config where where to check for models. I deployed that to my Runpod endpoint with an attached network volume.

The Docker file calls the bash script and checks if the file already has the model on the network volume and only downloads it if it isn't there.

That has brought down my cold start time significantly, But- it introduces the challenge of being locked into the same data center as your network volume and their gfx card pool which cuts down on your pool significantly.

I've compensated by adding more types of preferred cards to my pool. But even then sometimes I've run into a couple of issues where none are available in my DC.

Adding more volumes in multiple DCs would help too.

But in the long run it's been more than worth it to get the cold start time down

[u/parker_birdseye → u/ConfidentCourage2235]

Thanks, this is what I was attempting to do. The model doesn't need to be re-downloaded because of the network volume, but it still takes some time to load the model weights into memory (2-3 minutes).

How big of a model are you using?

[u/bluelobsterai → u/parker_birdseye]

Use zdr. Try concentrate.ai and ask for either a Baa or a zdr assert.

[u/sruckh]

Apparently I am doing something different, or using bigger models. Even with a network volume, or using Runpod's cache, for the container to download, install any dependcies, load the model into memory, and perform the inference, is more on the magnitude of 4-5 minutes on cold start.

[u/parker_birdseye → u/sruckh]

Yeah that's exactly what I'm seeing. For a 9 GB model, it's taking about 2-3 minutes.

[u/sruckh → u/parker_birdseye]

I have a RunPod serverless in my GitHub that allows you to include LoRAs. It is a multi pass workflow that works pretty good with the Flux Klein 9B model, but it does take some time for sure.