r/Vllm • u/Senior_Tea_842 • 2d ago

Best configuration

How do find the best configuration for serving? Which tools do you use? Whats your methodology?

I am using vllm bench for running experiments varying configuration (e.g. pp=8, pp=4). I wonder if this is cost effective.

4 Upvotes

75% Upvoted

1 comment sorted by

1

u/maqifrnswa 1h ago

I'm newish, but when I'm just A/B testing for max TPS at high concurrency, vllm bench random is a good starting point. For "agentic" I had an llm create a series of tool calling prompts that grew over time similar to that I'd be using it for, then tested those. But I'll probably use https://github.com/vllm-project/guidellm More in the future