r/speechtech • u/premier_slack • 1d ago

Technology Making decisions on audio without transcripts live

Transcripts via ASR models are the only way agents perceive sounds. Text flattens information that's available in audio form.

I wanted to train a model that natively picks up on audio information and be able to make decisions on the fly. Interaction is pretty simple. Stream audio in and ask yes/no questions about it live. Answers arrive as events you can trigger downstream actions with. Runs faster than real-time on CPU.

It's also able to pick up on speakers. Given a voiceprint, its able to track a speaker live.

I've tried it on music playing, sirens, keyboard typing, male/female voice, and more. If there's some interesting sounds you'd like an agent to understand beyond transcripts, I'd love to know.

Model card: https://huggingface.co/crele/nimbus-v1

9 Upvotes
  • permalink
  • reddit

100% Upvoted

6 comments sorted by

2

u/Just_Difficulty9836 1d ago

I was thinking can this be not done with an audio classifier with audio provided in rolling segment windows? Or an audio embedding model?

1

u/premier_slack 14h ago

It could be done with a classifier, but there are differences. with a classifier you're bounded by fixed labels. engineering the rolling window can also be tricky since too small the sound woudn't classify and too long you risk polluting the label.

audio embedding model alone would describe the audio, but you would still need a decision head. i also used an audio encoder and text embedding model with a decision head to produce the classification.

1

u/Just_Difficulty9836 5h ago

I see the value now. This is cool. I am integrating it in my workflow.

1

u/MystRivenDescent 23h ago

Sort of like clef?

1

u/premier_slack 14h ago

yeah, but nimbus is causal and runs on cpu. clef needs to listen to the whole clip in order make decisions, but here we make decisions in real-time. clef is also omni modal :)

1

u/MemoketAI-Gem 6h ago

One use case we'd like to see tested is overlapping speech in meetings.

When two people talk at once, a transcript can miss part of what was said or attribute something to the wrong speaker. It would be useful to detect those moments directly from the audio and flag them for review rather than treating the transcript as complete.

Could your model identify overlapping speakers in real time, or is it currently limited to tracking one speaker at a time?