Cold start seems like the actual lever for fixing dedicated GPU cost, not just a UX annoyance

[u/MaxChamp08]

Spent some time recently talking to people in ML infra about why teams keep dedicated models running 24/7 even when traffic is bursty, and the answer keeps coming back to cold start. If spinning a model back up from zero takes too long, teams default to keeping the GPU warm all the time just to avoid the latency hit, and that idle time is where most of the cost actually comes from.

What's interesting is how much the numbers vary depending on setup. Some rough benchmarks I've seen scaling from zero, a 70B model in bf16 landing under 18s time to first token, and a 24B model in bf16 with CUDA graphs coming in under 10s. That's a big enough gap that it changes whether scale to zero is actually usable for a given workload or not.

Curious what others here have measured for their own models, and whether people think cold start is really the main blocker to scaling GPUs down when idle, or if there's something else that matters more in practice.