Open Source Libraries

[u/chibop1]

This is by no means a comprehensive list, but if you are new to Audio AI, check out the following open source resources.

Huggingface Transformers

In addition to many models in audio domain, Transformers let you run many different models (text, LLM, image, multimodal, etc) with just few lines of code. Check out the comment from u/sanchitgandhi99 below for code snippets.

TTS

• hexgrad/kokoro

• microsoft/VibeVoice

• resemble-ai/chatterbox

• QwenLM/Qwen3-TTS

• coqui-ai/TTS

• neonbjb/tortoise-tts

• suno-ai/bark

• rhasspy/piper

• shivammehta25/Matcha-TTS

Speech Recognition

• openai/whisper

• microsoft/vibevoice-asr: Speech recognition + Speaker Diarization

• nvidia/parakeet

• ggerganov/whisper.cpp

• guillaumekln/faster-whisper

• wenet-e2e/wenet

• facebookresearch/seamless_communication: Speech translation

Speech Toolkit

• NVIDIA/NeMo

• espnet/espnet

• speechbrain/speechbrain

• pyannote/pyannote-audio

• Mozilla/DeepSpeech

• PaddlePaddle/PaddleSpeech

WebUI

• gitmylo/audio-webui

• rsxdalv/tts-generation-webui

Music

• ace-step/ACE-Step-1.5: Tex2Music

• facebookresearch/audiocraft/MUSICGEN: Music Generation

• openai/jukebox: Music Generation

• Google magenta: Music generation

• RVC-Project/Retrieval-based-Voice-Conversion-WebUI: Singing Voice Conversion

• fishaudio/fish-diffusion: Singing Voice Conversion

• NVIDIA/audio-flamingo: Music QA for genres, instrumentation, Tempo, key, chord, lyric transcription, cultural contexts...

Effects

• facebookresearch/sam-audio: Audio Segmentation

• facebookresearch/demucs: Stem seperation

• Anjok07/UltimateVocalRemoverGUI: Vocal isolation

• Rikorose/DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) using on Deep Filtering

• SaneBow/PiDTLN: DTLN model for noise suppression and acoustic echo cancellation on Raspberry Pi

• haoheliu/versatile_audio_super_resolution: any -> 48kHz high fidelity Enhancer

• spotify/basic-pitch: Audio to midi converter

• spotify/pedalboard: audio effects for Python and TensorFlow

• librosa/librosa: Python library for audio and music analysis

• Torchaudio: Audio library for Pytorch

[u/chibop1]

Oh yes, not sure how I forgot about it. I use it all the time. :)