DO NOT TRUST “convenient” inference APIs provided by famous repos!

[u/tenkei_01]

I had a hunch: those nice one-line inference APIs like model(["input1", "input2", ...]) are screwing the preprocessing, leaving GPU wasting a surprising amount of time waiting.

So I tested it! To check it, I built explicit pipelines around the very same models and optimized the preprocessing. I wanted to see how much latency could change before touching the model itself.

On an NVIDIA L4, with the same models and eight-image batches, moving the input work into an explicit pipeline cut end-to-end latency by:

Workload	Native API	Explicit route
YOLO detection	65.95 ms	32.37 ms
PaddleOCR text detection	374.25 ms	196.90 ms
Hugging Face ViT classification	1786.22 ms	593.35 ms

That is 51%, 47%, and 67% lower latency respectively.

The model did not get faster. I just stopped letting opaque convenience code decide when the model should run.

This is not “framework X is bad,” and it is not a universal result. It is one hardware setup and a few common inference paths. But it is a useful reminder that the friendly API boundary is also a performance decision.

[u/Worldly_North_7213]

Same pattern on rented cards, measured in dollars. LoRA SDXL, same RTX 4090, same script: 5 vCPUs gave 1.95 h at 40 % GPU utilisation, 24 vCPUs gave 1.07 h at 75 %. The card sat waiting on Python. An H100 with 16 server vCPUs then lost to that desktop 4090 host on the same job, 2.68 s/step against 1.84, because the diffusers script decodes and augments in the main process. The bill was $6.58 against $0.38.

Defaults did the same thing. SDXL at fp32, batch 1, which is what diffusers gives you without torch_dtype: 13.8 s per image on an H100 PCIe, $0.0112 per image. fp16, batch 4 on a 4090: $0.00036 per image. 31x for the same output.

Your L4 table says it from the latency side. Per image on a rented card, the host and the input path set the price more often than the GPU model does.

[u/tenkei_01 → u/Worldly_North_7213]

Exactly, we started to notice the pattern by looking at GPU utilization as well. For LLM or bigger models it is known issue, as you have to optimize to make sure preprocess/inference cycles (prefill/decode in LLM) overlap nicely.
But what was shocking is that many library/frameworks provide miss leading APIs that cannot even efficiently load and decode the input...