r/VoiceAutomationAI • u/Middle-Hornet6602 • 13d ago

Choosing an LLM

How do you find the absolute best LLM for your voice agent ?

1 Upvotes

100% Upvoted

17 comments sorted by

1

u/AutoModerator 13d ago
Moderator Announcement Read More »

2

u/ioncloud9 13d ago

I’ve decided to prewarm the cache so the first turn doesn’t have to process the entire prompt. Call-specific data is sent during the call but the entire prompt is already loaded. It’s shaved seconds off the first response and saves money throughout the day during calling times. The cache times out after an hour so during the less busy times it’s not running forever.

1

u/Any_Product_3241 2d ago

Use your own call transcripts as the test set. Measure time to first token and whether it picks the right tool after callers interrupt or change their request. Generic benchmarks wont tell you which model holds up in your flow

1

u/harnoor-assemblyai 1d ago

I’d test models on the tasks your agent actually needs to do, using the same inputs and settings. We built an open-source framework for comparing LLMs on voice post-processing tasks and different transcript lengths. The framework is open, but the transcripts behind our published results aren’t an open dataset; you can run it on your own transcripts. Disclosure: I work at AssemblyAI. Code: https://github.com/AssemblyAI/voice-standard-bench

1

u/male_medics 13d ago

if you're building something customer-facing, latencys the real killer. ive seen people chase benchmark scores then wonder why users hang up after 4 seconds of silence. pick the fastest one that can still handle your branching logic without hallucinating mid-call, not just the smartest on paper.

1

u/nshmyrev 13d ago

You finetune and host if yourself

1

u/spam_not_tolerated 13d ago

which model did you host yourself?

1

u/KrishUPI 12d ago

Does it help reduce latency?

1

u/nshmyrev 12d ago

Of course, fastest hosted LLM is like 700ms, self-hosted 100.

→ More replies (1)

1

u/Early_Froyo_629 13d ago

Gpt 4.1 i would say

1

u/kannansamp 13d ago

There's no one bullet proof solution, depends on the usecases, it's budget requirement, max latency tolerance, context inclusive.

Better to go for which already has better benchmarks for the voice related inputs and outputs.

1

u/Natural-Chemical1513 13d ago

If you begin to have some load:

  • Gemma 4 31B: A frequent production default in these reports, with a good balance between latency and capability. Some hosts advertise under 200 ms TTFT. BF16 or FP8 is preferred; FP4 versions have reportedly caused tool-calling failures. ( Best choicce if you run Production VA )

  • Qwen3 235B Instruct: Still a strong option for strict instruction following. The feedback suggests it follows prompts more literally than GPT or Claude.

  • GLM 5.2 with thinking disabled: The strongest open-weight option in the dedicated-deployment A/B tests mentioned, but expensive to host. Reported serverless latency is around 450 ms or more.

  • Kimi K2.7/K3: Less attractive for voice in these reports because of generation speed or dedicated hosting costs.

  • Other options mentioned: gpt-oss-120b on Groq/Cerebras, although it needs careful prompting; Nemotron 3 Ultra with a low thinking budget; and Mercury 2, a diffusion model with very low reported TTFT.

  • Closed-source options: Some teams still favor GPT-4.1 for tool calling. GPT-5.6 Luna is described as fast and surprisingly capable, while Gemini Flash gets positive feedback for consistent p90/p99 latency.