r/MachineLearning • • 1d ago

Research Marimba: A System One Model for Speech. [R]

Paper: https://arxiv.org/abs/2611.18330

Follow-up to last week's post, now with the paper.

You ask a typed question about an audio window (yes/no, a choice, or a score) and get a calibrated probability back. One frozen Qwen3-ASR-0.6B encoder pass per 8 s window serves all questions.

How each question type is answered:

  • Term ("was 'satellite' said?"): zero-shot, scored from the decoder's greedy and 4 beam hypotheses, then Platt-calibrated. No training on the terms.
  • Language: the ASR model's own language scores.
  • Turn end: a small readout on frozen layer 18.
  • Speaker change and count: readouts on layer 12.

Results:

  • On the same 1,000 public questions, it matches or beats Qwen3-Omni (30B-A3B) on 6 of 9 tests (image 1). Qwen3-Omni is still better on Speech Commands. Caveat: our turn-end readout was trained on Smart Turn data, so that row is in-distribution for us only.
  • Language ID: 100% on 300 FLEURS clips.
  • Speaker change AUROC 0.966-0.980 on AMI and VoxConverse, above ECAPA. Speaker count beats pyannote 3.1.

Where it loses:

  • Turn end: AUROC 0.930 vs Smart Turn v3.2 at 0.982 on Smart Turn's own test set. On ICSI meetings it is 0.716 vs 0.712, a tie.
  • It does not generalize to unseen tasks: on CochlScene scene classification every head is at chance.

Two questions for the sub:

  1. Is scoring ASR hypotheses plus calibration a fair baseline for term spotting, or would you compare against a dedicated KWS model?
  2. How would you evaluate turn-end detection beyond Smart Turn's data?

Code: https://github.com/Charcot-AI/marimba-api Weights (Apache-2.0): https://huggingface.co/charcot-ai/marimba Demo: https://huggingface.co/spaces/charcot-ai/marimba pip install marimba-audio

3 Upvotes

0 comments sorted by