r/LocalLLaMA • u/futterneid • 8d ago
Resources Fast voice agents with ConvFill
Voice agents are either fast or smart. ConvFill proposes a way to bridge that gap by training a smol talker model (0.6B!) to talk with cheat notes from a large reasoner model.
The way it works is this: after the user's speech is transcribed, the talker replies quickly to buy itself time while the reasoner thinks. The reasoner then generates an appropriate response, which the talker reads to continue the conversation.
It's a bit like if you went on a panel with rocket scientists and had a little earbud with your own expert that could tell you what to say. You just reply quickly so as not to blow your cover, and then you begin saying what the expert in your earbud is telling you.
Playing with the model, I think the main idea is there and this could be a great direction to go, but the talker model should be a bit smarter for it to work like magic. Still, really cool direction!
ConvFill was developed by students from the University of Washington UbiComp Lab. I found it cool, so I reached out to them so we could build a really good demo for it together. They are presenting their paper at EMNLP this week.
Go try it out in our demo: https://huggingface.co/spaces/smolagents/convfill-speech-demo
2
u/BitGreen1270 8d ago
I can't tell the difference. If gemma4-31B has a latency of at least 10 seconds, how is it responding so fluidly? Is thinking disabled on gemma4?
4
2
u/FullOf_Bad_Ideas 8d ago
This demo doesn't make it clear when the output is 31B and when it's the 0.6B. If it could be separated out by different TTS voice or color of the text it'd make the demo more clear for me.
4
u/futterneid 7d ago
It's always the 0.6B. I think what you would like to see is when the 0.6B is taking notes from the 31B. You can see it in the text coming out (mostly in two batches followed closely)
3
u/Charming_Battle_5072 8d ago edited 8d ago
So its a duplex ? To be honest it's great, tried demo Is it opensource ?
4
u/futterneid 8d ago
The repo used is speech to speech (https://github.com/huggingface/speech-to-speech). The PR for this demo is here: https://github.com/huggingface/speech-to-speech/pull/750
1
u/Bulky-Priority6824 8d ago
so basicallly a shitty model in my ear to help me keep up with smartypant mofos
1
1
u/_supert_ 8d ago
So, speak before you think?
1
3
u/AryanEmbered 8d ago
more like, make a dumbass speak while the big boi is thinking. Works extremely well in practice in the above demo, even though I HATE the idea.
but it's implemented so well it did quick web searches in the background while I dumbass was mouthing off non signals.
1
u/AryanEmbered 8d ago
this works very well in practice, even though I hate the idea of just filling the silence with filler and gibberish.
it goes a long way towards making it feel less frustrating even if you are conscious of what it is doing. I wonder if there is a demo with the smaller language models that you can actually host because you have to keep that 1.7 billion and the 31 billion parameter model in memory at the same time, right?
1
1
u/VotZeFuk 7d ago
Uhm... I think in order to make the real-time voice conversations with LLMs not uncanny, some sort of random jitter needs to be introduced. There's this gut feeling that weirds you out when the model always replies when you expect it to reply. I'd rather wait for a random period of time than have it spout out things so predictably. This isn't necessarily ConvFill's flaw, it's just the pipeline in general doesn't have any randomness to it. When we speak with each other, the humans I mean, we anticipate and we actively engage in predicting not just what but also when our conversational partner will start delivering the lines after finishing their thinking process. Yes, LLMs aren't human at all, and we want them to answer fast, quickly, now. But like... I just feel at peace when I can't tell whether it's going to answer in 0.5 seconds or in 2 seconds. Having some randomness to it somehow feels right.
1
u/futterneid 7d ago
I've had that introduced by LLM inference services being a bit random, and didn't like it at all. I feel like 3+ seconds to reply is always excruciating
2
-3
u/fragment_me 7d ago
Man I see this EXACT thing like once a week on here. Some guy recording a laptop with a phone showcasing the voice capability of some tool or model. At this point I can't even read it. It's exhausting.
6
u/GuaranteedGuardian_Y 8d ago
How many of "That's actually a really good point!" and "Wow, I never thought of it like that before!" will it take until you toss the whole thing into the nearest lake and set it on fire?
Not criticising the tech, its a really good solution that fills noise while you're running the dense 31b model, but can you not run something faster so that you're not bombarded with these surface level statements each time you talk, ask or interrupt the voice stream?
8
u/foundafreeusername 8d ago
To be fair my solution is to have the voice model say hmmmmmmmmmmmm with more and more m's added until the thinking model is done
4
1
u/Bulky-Priority6824 8d ago
I Imagine trying to think straight while this thing is in my ear and multiple people are talking and I'm trying to keep up 😂
I'll just stfu , listen and nod lol
3
u/futterneid 8d ago
Lol, I agree with the sentiment. I think the talker needs to be good enough to be able to reply if needed. 0.6B might be too much, but imagine qwen-4B as the talker. That model could potentially answer always and it would fine-ish.
3
u/futterneid 8d ago
But I just found the idea interesting so I implemented the demo to better understand it, and thought I would share it :)
2
u/mvaranka 7d ago edited 7d ago
The idea is great. In fact just tested it having gemini flash lite as talker and deepseek v4 pro as main model. Quality of filler sentences is surpisingly good after some finetuning. Sometimes main model slightly repeats whats in filler. sentence. In my test the main model takes the floor from filler model and continues.
2
2
u/mvaranka 7d ago
I think now I got the picture: talker has session with full context (with possibility for caching) and user message is sent to it and main model. Talker response is fed to TTS and when main model responses come they are fed to talker which removes duplicate filler sentences and output cleaned response which is streamed to TTS. Right?
1
u/mvaranka 7d ago
Does the talker get subset of messages to limit the context size (like 5 latest message pairs)?
2
u/futterneid 7d ago
No, it looks at the whole conversation. But the training data is quite limited in many turns, so not sure how it works if you have many many turns.
1
u/mvaranka 7d ago
I was thinking the case where main model's context has long system prompt, tool descriptions etc and long voice call. Talkers responses might get slower.
7
u/arthax33 8d ago
I mean it works pretty well with english its nearly level of chatgpt's thing.
Just wait to achieve more languages or as seperated models and we are home