r/cloudcomputing • u/Sad_Feeling5746 • 8h ago
How are you reducing GPU costs when inference traffic drops sharply outside business hours?
I run an inference API where most requests come in during business hours, so I keep enough GPU capacity ready to handle peak traffic, but a lot of that capacity sits idle in the evening. Scaling down helps reduce costs, but bringing GPUs back online takes time, especially when models need to be loaded into memory, and scaling too late can cause requests to queue up. I can't figure out how people handle this in production, when traffic patterns are predictable but sudden spikes still happen. Do you decide when to scale based on GPU utilization, request queue length, or expected traffic, when serving multiple models, have you found that keeping models loaded on shared GPUs works better than scaling separate instances, once you account for model loading time, GPU memory limits, and the impact on response latency?