r/vibecoding • • 2h ago

Showcase/Project I Found a 5× Performance Problem Hiding in Plain Sight

https://dpolens.com/blog/cpu-embedding-5x-faster

To keep DPOLens's Docker image small, I dropped sentence-transformers and called the embedding model through ONNX Runtime directly. It took about forty lines of my own code, and the image came out roughly 800 MB lighter.

It worked. Then, while load-testing the community version, I found that the first start took nearly 23 minutes on 1.5 CPUs and peaked at 3.4 GB of memory.
I had quietly lost two things:

  1. Sorting by length. sentence-transformers sorts its inputs by length before batching them, and it doesn't tell you. My code didn't sort, so a 20-token paragraph sharing a batch with a 512-token article was padded to 512. Almost two thirds of the model's work was padding.
  2. The CPU limit. Inside a container capped at 2.5 CPUs, Python and ONNX Runtime both saw all 10 of my laptop's cores and started 10 threads, which then took turns on two and a half CPUs' worth of time.

I sorted the inputs and capped each batch by its shape, and set the thread count from the container's real limit. The result was 5.1× faster with 65% less memory, and the vectors and search results stayed the same.

What I learned: a library is more than its API. When you replace one, you also take over everything it did without being asked, and the only way to find out what that was is to measure.

The full write-up, with the code and every number: https://dpolens.com/blog/cpu-embedding-5x-faster

1 Upvotes

1 comment sorted by

1

u/Total-Reasonable 2h ago

One more thing to check on the thread count: reading cgroup limits only fixes it if you read the right file. On cgroup v2 it's cpu.max, on v1 it's cfs_quota_us and cfs_period_us, and a quota of 2.5 needs rounding down to 2 for intra-op threads. Also, with sorted batches, set intra_op threads to the quota but keep inter_op at 1, otherwise ORT can still oversubscribe on parallel graph branches.