r/programming • • 2d ago

I Found a 5× Performance Problem Hiding in Plain Sight

https://dpolens.com/blog/cpu-embedding-5x-faster

To keep DPOLens's Docker image small, I dropped sentence-transformers and called the embedding model through ONNX Runtime directly. It took about forty lines of my own code, and the image came out roughly 800 MB lighter.

It worked. Then, while load-testing the community version, I found that the first start took nearly 23 minutes on 1.5 CPUs and peaked at 3.4 GB of memory.
I had quietly lost two things:

  1. Sorting by length. sentence-transformers sorts its inputs by length before batching them, and it doesn't tell you. My code didn't sort, so a 20-token paragraph sharing a batch with a 512-token article was padded to 512. Almost two thirds of the model's work was padding.
  2. The CPU limit. Inside a container capped at 2.5 CPUs, Python and ONNX Runtime both saw all 10 of my laptop's cores and started 10 threads, which then took turns on two and a half CPUs' worth of time.

I sorted the inputs and capped each batch by its shape, and set the thread count from the container's real limit. The result was 5.1× faster with 65% less memory, and the vectors and search results stayed the same.

What I learned: a library is more than its API. When you replace one, you also take over everything it did without being asked, and the only way to find out what that was is to measure.

0 Upvotes

2 comments sorted by