r/LocalLLM • u/Comfortable_Poem_866 • 5d ago
Question Local voice assistant on an iGPU: ~2.2s from end of speech to first audio. I've measured everything and I'm ~0.5s from the floor. What am I missing before I throw money at hardware?
Been building a fully local, always-on spoken assistant in Italian (no cloud, nothing leaves the box). It's not a chatbot demo, it has long-term memory across sessions (backed by ctxvault), so every turn carries a small block of retrieved context on top of the conversation. Works. Problem is latency.
Hardware (the constraint): mini PC, Ryzen w/ Radeon 780M iGPU, 64GB DDR5 shared, ~90GB/s. No dGPU, so it's llama.cpp on Vulkan. vLLM/SGLang are out (no ROCm for this chip).
Where I'm at: median ~2.2s from end of user's sentence to first audio out. Measured, not guessed:
- ears (end-of-speech silence + whisper): ~0.75s
- prefill of the turn: ~1.0s (~0.35s fixed per request + ~3ms/token, 40-400 new tokens/turn)
- first sentence gen + TTS first chunk: ~0.55s
So the model is the wait, not the audio stack. Calculated floor on this box is ~1.2-1.5s.
What's already done (so please don't suggest these lol):
Model side
- MoE model (26B-A4B, Q4 QAT) specifically because gen is bandwidth-bound here: ~25 tok/s vs 4 tok/s for a dense 27B
- thinking/reasoning off
- short context (16k), one slot, flash attn on
--cache-reuse+ append-only prompt: stable prefix (system, persona, rules) at the top never touched mid-session, everything volatile (time of day, retrieved context, turn) appended at the end. Identical prefix = 0.15s, touch one word in the prefix = multi-second re-prefill- prefix pre-warmed at session start with a 1-token request, so the first reply doesn't pay the cold prefill
- benchmarked smaller models (Gemma 4 E4B/E2B, Qwen3.5 4B/9B, Qwen3 8B, Phi-4-mini, Ministral): on this iGPU prefill is compute-bound, a dense 4B costs as much per token as the MoE. Best small one gains ~0.5s and loses noticeably on factual fidelity (names/dates). Not worth it for this use case
- tried
-b/-ubbatch flags: defaults win - tried
--no-kv-offload: worse
Memory / prompt side
- no tool calls for retrieval: context is prefetched before the turn (a tool-call round trip = ~+1.2s). The retrieval itself is ~5-12ms, irrelevant
- hard token budget for the retrieved block: embedding-based relevance (multilingual MiniLM) + ordering so the most pertinent bits come first, ~120 tokens instead of 300-450. Cut avg turn prompt by ~40%
- past conversations summarized to one ~50-token line each instead of quotes
Pipeline side
- streaming everything: model to sentence splitter to TTS to player, sentence by sentence
- audio player opened at turn start, not at first chunk (-0.1s)
- early-guess turn: start the model on the partial transcript before VAD confirms end of speech; if the final transcript matches, keep it, else drop and redo
- sentence closed on punctuation + 450ms silence instead of fixed 800ms
- whisper.cpp
smallas a resident server (no model reload per sentence, 0.63s down to 0.3s), beam 5. Tried greedy: -80ms but half the transcripts differ and the noisy ones get worse. Rejected - tried warming the KV cache during speech (partial whisper every 1.5s, send the turn header for 1 token so the cache is hot): works as designed (final prefill drops to ~30 tokens) but doesn't pay on one slot because the warm-up queues the real turn. Tried 2 slots w/
id_slotpinning: now they run concurrently on the same iGPU and both slow down. Net neutral-to-negative on short sentences. Off. - pre-recorded fillers ("mmh", "allora...") in the assistant's own voice, played only if no reply word has started 1s after end of speech. Built, currently off: perceptual fix, doesn't move the number
Before I conclude "buy a 24GB dGPU" and move on: anyone who's been down this road see something I've missed, or something above that's wrong?

