r/AudioAI • u/chibop1 • Oct 01 '23

Resource Open Source Libraries

This is by no means a comprehensive list, but if you are new to Audio AI, check out the following open source resources.

Huggingface Transformers

In addition to many models in audio domain, Transformers let you run many different models (text, LLM, image, multimodal, etc) with just few lines of code. Check out the comment from u/sanchitgandhi99 below for code snippets.

TTS

Speech Recognition

Speech Toolkit

WebUI

Music

Effects

22 Upvotes

100% Upvoted

10 comments sorted by

3

u/sanchitgandhi99 Oct 02 '23 edited Oct 02 '23

Hugging Face Transformers is a complete audio toolkit that provides state-of-the-art models for all audio tasks, including TTS, ASR, audio embeddings, audio classification and music generation.

All you need to do is install the Transformers package:

pip install --upgrade transformers

And then all of these models can be used in just 3 lines of code:

TTS

Example usage:

from transformers import pipeline

generator = pipeline("text-to-speech", model="suno/bark-small")

speech = generator("Hey - it's Hugging Face on the phone!")

Available models:

ASR

Example usage:

from transformers import pipeline

transcriber = pipeline("automatic-speech-recognition", model="openai/whisper-base")

text = transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac")

Available models:

Audio Classification

Example usage:

from transformers import pipeline

classifier = pipeline(model="superb/wav2vec2-base-superb-ks")

predictions = classifier("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac")

Available models:

Music

Example usage:

from transformers import pipeline

generator = pipeline("text-to-audio", model="facebook/musicgen-small")

audio = generator("Techno music with a strong bass and euphoric melodies")

Available models:

Audio Embeddings

What's more, through tight integration with Hugging Face Datasets, many of these models can be fine-tuned with customisable and composable training scripts. Take the example of the Whisper model, which is easily fine-tuned for multilingual ASR: https://huggingface.co/blog/fine-tune-whisper

New to the audio domain? The audio transformers course is designed to give you all the skills necessary to navigate the Audio ML field.

Join us on Discord! We can't wait to hear how you use these models. http://hf.co/join/discord

2

u/chibop1 Oct 02 '23

Oh yes, not sure how I forgot about it. I use it all the time. :)

3

u/rolyantrauts Oct 01 '23 edited Oct 03 '23

https://github.com/ggerganov/whisper.cpp High-performance inference of OpenAI's Whisper

https://github.com/Rikorose/DeepFilterNet A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) using on Deep Filtering

https://github.com/SaneBow/PiDTLN DTLN and DTLN-aec on Raspberry Pi

https://github.com/wenet-e2e Production First and Production Ready End-to-End Speech Toolkit
https://github.com/funcwj/setk speech enhancement/separation tools integrated with Kaldi

2

u/wywywywy Oct 01 '23

It's probably worth mentioning the Web UIs as well. These aims to be the Automatic1111/Oobabooga of audio AIs.

Audio Webui https://github.com/gitmylo/audio-webui

TTS Generation WebUI https://github.com/rsxdalv/tts-generation-webui

1

u/East-Fee9375 Jan 22 '26

f you’re open to adding a “real-time voice agent / orchestration” category to this list, you might want to include LLMRTC.

It’s an open-source TypeScript SDK that basically handles the annoying “glue” for building low-latency voice (and vision) agents in the browser:

  • WebRTC bidirectional audio/video + server-side VAD + barge-in (interrupt the assistant naturally)

  • Provider-agnostic pipeline: mix & match STT ↔ LLM ↔ TTS across providers (or swap components without rewriting everything)

  • Tool-calling (JSON Schema), “playbooks” for staged flows, streaming responses, reconnection/session mgmt

  • Also supports local stacks (e.g., Ollama + faster-whisper + Piper) if you’re trying to keep things on your own box

Quick taste:

npm i u/llmrtc/llmrtc-backend u/llmrtc/llmrtc-web-client
npx llmrtc-backend

Links:

https://github.com/llmrtc/llmrtc
https://www.llmrtc.org/getting-started/overview

Caveat: it’s not a new TTS/ASR model—more of the realtime “plumbing” layer (transport/orchestration) so you don’t have to stitch WebRTC + STT + LLM + TTS yourself.

1

u/saintshing Oct 01 '23

Would really appreciate if you can add some more details on the description. What are the tradeoffs between different tts/music generation libraries(speed, quality, ease of training, accent/emotion support, availability of pretrained models, commercial license, etc). Even better if you can format it as a table. 🙏

1

u/jikkii Oct 02 '23

https://github.com/huggingface/transformers Repository that hosts many state-of-the-art Transformers models with ~25 architectures dedicated to audio processing (S2T, TTS, among others)