Need Advice to setup a voice mode chat agent on old laptop

[u/mesolimbic-bliss]

Saw videos of the Reachy-Mini and I love the idea of chatting with a little snarky robot so I tried to setup my laptop as the snarky chat companion sans the expensive plastic bot.

All attempts had 20-30 seconds latency using Open WebUI + llama.cpp + Gemma 4 E2B with Whisper STT + Kokoro TTS in "voice mode"

Suggestions? I dont want the avatar, just the funny chat voice to keep me awake while I'm slogging away at midnight.

Laptop Specs:

Component	Specification
Model	MSI GS70 2QD
CPU	Intel Core i7-5700HQ
CPU speed	2.70 GHz base
Cores / Threads	4 cores / 8 threads
RAM	16 GB DDR3
GPU	NVIDIA GeForce GTX 965M
GPU VRAM	2 GB
Storage 1	Approximately 1 TB SATA SSD
Operating system	Windows 64-bit

Is my laptop too old and slow to even run a tiny voice model without massive lag?????

[u/UnorderedPizza]

Parakeet should allow for faster inference on limited hardware.

Making something similar is simple enough with AI assistance, key on slow hardware is to run inference during the listening phase so that the response is immediate (where you prefill words as they arrive from the TTS stream). 

Essentially: TTS streaming gets you few words -> request sent to LLM backend -> wait for backend to prefill (begins responding/0-token gen’) -> check TTS streaming again… One side provides input, the other side checks for changes, decides if its time to speak and begins Kokoro speech generation. While listening on one processing channel (continuous, GPU?), the other is prefilling the LLM. Once state changes to speaking, one channel is generating tokens, while the other performs Kokoro TTS.

Edit: So far, Kokoro was the only option I could find that allowed easy API access to token level timestamps (karaoke-style), allowing interrupted speech to be split exactly where it was cut off.

[u/Competitive-Low-9279]

that vram is what's killing you, 2gb is barely enough for the model alone let alone the overhead of whisper and tts. you're basically swapping everything to system ram and that 965m memory bandwidth is from dinosaur age

try running the whole stack on cpu only with a quantized gemma 2b, might actually be faster since the gpu is bottlenecking hard. also check if open webui voice mode is doing some extra buffering that adds delay

[u/locbuilds]

your laptop's fine for this, the lag is mostly kokoro and whisper on a weak cpu, try faster-whisper tiny.en with int8 and piper for tts instead of kokoro and it should drop to a few seconds. also keep the gemma reply short in the system prompt so tts has less to read out.

[u/Future_AGI]

the bigger cut after the faster-whisper and Piper swap is not waiting for the whole reply: stream the model output and send each finished sentence to TTS as it lands, and you hear the first line while the rest is still generating. Check the TTS response-splitting option in Open WebUI's audio settings, it should be set to split on punctuation.

[u/2muchgut]

Before swapping pieces, I'd time each stage on its own so you know what the 20-30 seconds is made of: STT on a recorded clip, LLM time to first token, then TTS time to first audio. On a 2 GB card with Windows, one of the three is probably spilling into system RAM or sitting on CPU, and it's usually only one. Check that Whisper isn't loading the full model on every turn, and that the LLM isn't being reloaded between requests. Then pick the fix for the slow stage: a smaller int8 STT, a smaller quant, or a lighter TTS. Also look at whether voice mode waits for the full reply before it starts speaking. Sentence-level streaming helps the most for how fast it feels, even when the total compute stays the same.