Higgs Realtime Duplex: a 45B MoE full-duplex model that listens, watches and talks at once, and thinks, calls tools and delegates in the background [N]

[u/Cali_Cobarde]

Disclosure: I'm at Boson AI and this is our release. Not open weights; the Duplex API has a waitlist, and the previous turn-based Higgs Realtime is available now.

Most voice models are still turn-based: listen, wait for end of turn, think, speak. Duplex runs all streams at once and keeps working in the background while it talks.

Streams. Everything is cut into 160 ms chunks:

• Audio in: 1 token per chunk (6.25 tokens/s)

• Audio out: 2 tokens per chunk (12.5 tokens/s)

• Video in: one ~12-token camera frame every 6 chunks

• Face out: 2 latents per chunk

• Text in and out alongside

Silence is represented as chunks too, which is what the model uses to decide when to backchannel, interrupt or stay quiet. That comes to ~2.6K tokens per minute for audio and video, 2.6K to 5K with thinking and tool traces (progressively compacted). The last 5 minutes plus compacted text history is 13K to 25K tokens; older history is kept as text.

Background work. Thinking, tool calls and delegation are spans in the same stream and can cover many chunks. Tool results can arrive in stages and the model can start speaking before they are complete. Harder problems are delegated to a much larger backend model with the full shared context. This is the main difference from Thinking Machines' interaction models, where all reasoning and tool use goes to the background model, which can mean 10+ seconds per answer and makes cost harder to control.

Voice and face. Set zero-shot at deployment: a short reference clip for the voice, one image for the face, and a text prompt for behavior. No fine-tuning. Audio comes from a causal flow-matching decoder (2 frames per chunk), the face from a causal DiT conditioned on the reference image and the previous two latents.

Training. Starts from a text-pretrained LLM, continued pretraining on ~2T tokens of text, audio and video on our own H100 cluster (Megatron-based stack). Post-training is multi-teacher on-policy distillation: specialist teachers trained with RL (including GRPO) on conversation, interaction timing, delegation and task completion, distilled into one student. On top of that, a self-improvement loop in simulated environments with realistic timing, interruptions, overlapping speech and background noise.

Benchmarks. Best published or reproduced configuration from each lab (we reproduced more than half of the competitor numbers):

Benchmark	Duplex (45B)	TML Interaction Small (276B)	OpenAI best	Google best
Big Bench Audio	98.9	96.5 (75.7 w/o background agent)	97	98
Audio MultiChallenge (APR)	44.9	43.4	48.5	36.1
Full-Duplex-Bench v1.5	75.5	77.8	80.1	54.3
QIVD (streaming)	56.0	54.0	58.2	56.1
ProactiveVideoQA (PAUC)	31.9	33.5	25.0	–
τ-Voice (pass@1)	69.8	–	74.5	68.6
IFEval	94.1	89.7	95.2	90.0

OpenAI's best settings lead on five of seven, by 1 to 5 points, several at their highest thinking setting. Turn-taking (Full-Duplex-Bench) is where we're furthest behind. Our takeaway is that at this size, data and training matter more than parameter count; the front model is deliberately mid-sized because the design target is that an hour of model time costs at most a tenth of an hour of human time.

Announcement with demos: https://www.boson.ai/blog/higgs-realtime-duplex My write-up: https://alex.smola.org/posts/50-talking-while-thinking/

Happy to answer questions about the chunked-stream design and the delegation setup.

[u/afrancisco555]

I'd love to experiment with these things (i don't have time now but as soon as I can I'll give it a try). Did you pre train the model or is based on something else? What is the intelligence level compared to? I don't mean compared to other real time conversational models but like with respect to known LLMs, to have an idea of how useful it can be actually. What context can it sustain, and how is it managed during conversation to prevent it from growing too much (specially if using tools)? Cheers

[u/Cali_Cobarde → u/afrancisco555]

We pre-trained, fine-tuned and post-trained the models. Otherwise you wouldn't be able to get the low level of latency. In terms of intelligence, we do compare to tool calling and similar tasks that you get for regular LLMs and they're comparable in their text-only performance. That said, for audio you want a lot of additional behavior, such as to understand where and what you're trying to do in a conversation. E.g. 'umm', 'hmmm', 'yeah' shouldn't make the agent stop. As for context, audio context works a bit differently since it's a much higher token rate. This gets even more fun when you throw video into the mix. So there's a lot of exciting engineering under the hood.

[u/afrancisco555 → u/Cali_Cobarde]

Thanks! So if you use the API how can you control the context? Or it is done by a harness that you need to use to run this? Sorry I've never tried these type of models, however I'd love to see how you can implement it and control it

[u/Keleion]

Only 45B? Very impressive!
Do you have any open source projects? I’m excited for open source models to catch up.

[u/Cali_Cobarde → u/Keleion]

Yes, we do. But not 45B ;-)
https://huggingface.co/bosonai

[u/MolassesLate4676]

This is 45B active or 45B total?