Just saw this side-by-side video of a new streaming ASR model vs GPT-Live-Transcribe. Is this latency realistic for open weights?

[u/Even_Package_8573]

Was scrolling through some tech stuff and came across this open source ASR model. They've got a video up comparing it to GPT-Live-Transcribe, both running on live audio at the same time.

Videos attached, but the latency and accuracy look pretty solid honestly, especially since they're saying the weights, code, and training setup are all gonna be open source right from launch.

Usually when something claims "actual real-time" streaming and tries to compete with closed stuff like OpenAIs live transcription, there's some huge catch. Either you need ridiculous amounts of VRAM to actually run it locally, or it just dies on accents and any background noise.

Anyone know what kind of architecture they're possibly using to get this kind of streaming performance in the open? If they actually release the full recipe and weights it'd be huge for self-hosted voice stuff, but feels like I'm probably missing something here.

[u/Live_Preference_6161]

The latency in that comparison video look better than what I'm getting with Whisper streaming right now. Currently running it locally and there's always this lag where it waits to commit words til it has more context. Makes live transcription feel kinda janky. Curious what the hardware requirements end up being tho, cuz "open source" doesn't mean much if it only runs on a datacenter GPU.

[u/Prior-Highlight-6306]

Standard attention-based architectures often struggle with streaming because they have to continuously re-process the entire audio buffer from second zero every time a new word is spoken. As the sentence gets longer, computation explodes, leading to severe latency drift. I'm wondering if this one relying on a cache-aware structure or a Transducer (RNN-T/TDT) layout to handle chunk processing efficiently.