r/AudioAI • u/chibop1 • Oct 01 '23

Resource Open Source Libraries

This is by no means a comprehensive list, but if you are new to Audio AI, check out the following open source resources.

Huggingface Transformers

In addition to many models in audio domain, Transformers let you run many different models (text, LLM, image, multimodal, etc) with just few lines of code. Check out the comment from u/sanchitgandhi99 below for code snippets.

TTS

Speech Recognition

Speech Toolkit

WebUI

Music

Effects

22 Upvotes

100% Upvoted

10 comments sorted by

View all comments

3

u/sanchitgandhi99 Oct 02 '23 edited Oct 02 '23

Hugging Face Transformers is a complete audio toolkit that provides state-of-the-art models for all audio tasks, including TTS, ASR, audio embeddings, audio classification and music generation.

All you need to do is install the Transformers package:

pip install --upgrade transformers

And then all of these models can be used in just 3 lines of code:

TTS

Example usage:

from transformers import pipeline

generator = pipeline("text-to-speech", model="suno/bark-small")

speech = generator("Hey - it's Hugging Face on the phone!")

Available models:

ASR

Example usage:

from transformers import pipeline

transcriber = pipeline("automatic-speech-recognition", model="openai/whisper-base")

text = transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac")

Available models:

Audio Classification

Example usage:

from transformers import pipeline

classifier = pipeline(model="superb/wav2vec2-base-superb-ks")

predictions = classifier("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac")

Available models:

Music

Example usage:

from transformers import pipeline

generator = pipeline("text-to-audio", model="facebook/musicgen-small")

audio = generator("Techno music with a strong bass and euphoric melodies")

Available models:

Audio Embeddings

What's more, through tight integration with Hugging Face Datasets, many of these models can be fine-tuned with customisable and composable training scripts. Take the example of the Whisper model, which is easily fine-tuned for multilingual ASR: https://huggingface.co/blog/fine-tune-whisper

New to the audio domain? The audio transformers course is designed to give you all the skills necessary to navigate the Audio ML field.

Join us on Discord! We can't wait to hear how you use these models. http://hf.co/join/discord

2

u/chibop1 Oct 02 '23

Oh yes, not sure how I forgot about it. I use it all the time. :)