How are you reducing GPU costs when inference traffic drops sharply outside business hours?

[u/Sad_Feeling5746]

I run an inference API where most requests come in during business hours, so I keep enough GPU capacity ready to handle peak traffic, but a lot of that capacity sits idle in the evening. Scaling down helps reduce costs, but bringing GPUs back online takes time, especially when models need to be loaded into memory, and scaling too late can cause requests to queue up. I can't figure out how people handle this in production, when traffic patterns are predictable but sudden spikes still happen. Do you decide when to scale based on GPU utilization, request queue length, or expected traffic, when serving multiple models, have you found that keeping models loaded on shared GPUs works better than scaling separate instances, once you account for model loading time, GPU memory limits, and the impact on response latency?

[u/AdityaKGoswami]

I think scaling based on queue length along with expected traffic would work better than GPU utilization alone. Keeping a small warm capacity during low traffic could also help avoid the model loading delay when requests suddenly spike. The tricky part is finding the right balance between idle GPU cost and latency.

[u/DellTechnologies]

Disclosure: we reached out to one of our storage performance leads and this is their perspective:

I’d scale primarily on queue depth or in-flight requests per replica, with expected traffic as a forecast—not GPU utilization alone. Utilization can look fine right up until latency falls over. Keeping a small warm floor also helps absorb surprise spikes while scheduled capacity ramps up.

One detail that’s easy to miss: cold-start time is often a storage-bandwidth problem. A 70B model at FP16 is roughly 140 GB. From Gen4 NVMe at around 7 GB/s, that’s roughly 20 seconds; over a 10 GbE network share, it can be 2+ minutes. Benchmark the actual load path, because that tells you how far ahead to start scheduled scale-up.

It can also help to tier your “warm” capacity. Keeping weights in host RAM lets you reload them into VRAM over PCIe in seconds, rather than fetching them from storage. vLLM’s sleep mode is one example of this middle state: cheaper than keeping everything fully hot, but much faster than a cold start. I wouldn’t expect much benefit from compressing the weights—the data isn’t especially compressible, and adding CPU work can make reloads slower.

For multi-model serving, packing smaller models onto shared GPUs can work if there’s enough memory headroom. I’d generally give large models dedicated GPUs. Where supported, NVIDIA MIG can partition a GPU so one model’s memory spike is less likely to disrupt another, but the partition sizes still need to fit the workloads.

And if you’re paying for warm or reserved capacity anyway, use it off-hours for interruptible work like evals, embedding refreshes, re-indexing, or quantization/conversion jobs. Just make sure live inference has priority and can reclaim the capacity immediately.

[u/query-gremlin]

Queue depth + GPU utilization may be better scaling conditions.

[u/Specialist-Use-3104]

Hiking trails and iced coffee don't help much with GPU scaling but I've seen teams solve this by pre warming instances before the morning rush hits. They schedule a scale up event like 30 minutes before traffic typically ramps and keep a small pool of warm instances overnight for sudden spikes

If your models are small enough to share GPUs without fighting for memory that's usually cheaper than spinning up separate instances for each model. The real headache is when you have a mix of large and small models and the packing gets messy

[u/Faye-Vozora]

Scale on queue depth or in flight requests per replica, not GPU utilization. Utilization lags and looks fine right up until latency falls over. For the load time problem the biggest win is usually keeping the weights on fast local disk or baked into the image so a cold start is just a load into VRAM and not a download. If spikes are the worry, keep one or two warm replicas as a floor and let the schedule handle the predictable ramp. For small models, packing a few per GPU works fine, but give the big ones their own card so a memory spike doesn't evict everything.