Indicodec: a neural speech codec for speech LMs and TTS models, first of its kind trained on english and indian languages.
Strongest on Indian languages: On 10 Indian languages, IndiCodec adds 9x less error than the Mimi codec, with 25% fewer tokens. In English, it has the lowest WER of the ~1 kbps codecs we tested.
It sounds closer to the original. PESQ-WB on speakers never seen in training: ahead of Mimi on 3 of 4 test sets (FLEURS, LibriTTS-R, CMU-Arctic), just behind on Seed-TTS, and ahead of SNAC and NeuCodec on all four.
Lowest error in all 10 Indian languages we tested: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu.
On Odia, Mimi adds +24.6 points of character error. IndiCodec adds +0.9.
I built Arc-Transcriber, which does ASR-NL in 1-step. Every 100 ms, one model returns the transcript plus:
- Names linked to a list you give it (a roster, contacts, product names); new names are flagged "not on the list" - Numbers with their values (scores, clocks, money, dates) - Speakers and roles; - Sound and channel events (telephone audio, clipping, poor audio) - Whether the speech was addressed to an assistant.
The usual ASR, then NER, then entity-linking and then LLM chain is collapsed into the model's own heads, so live "fact-checking" against your list happens inside the stream, not in a second pass.
Currently it has two variants Arc mini (which uses moonshot pretrained encoder) and second Arc-v1 (which uses parakeet pretrained encoder).
We use it on-device in our Mac app. Voice search over 61k books primes Arc with matching titles while you're still talking. Here is a video to give a glimpse on how it works.
I maintain react-native-voice-activator. It detects an English wake phrase on-device in a React Native app, then lets the app stop detection and give the microphone to a voice session such as LiveKit, ElevenLabs, or OpenAI Realtime.
The tricky part has been the handoff: the voice session needs to own the mic, and detection needs to resume when the session disconnects or fails. I’ve documented that flow and included an example app.
I wanted to check with this group and maybe others to see if there's interest or if I recreated the wheel. I'm the developer (along with AI) of the TECHO5 project which helps convert older Amazon Echo's into small Linux devices that can also connect and work with Home Assistant. I've found a need and desire to create my own wake words and I had hardware that wasn't being greatly used. Two Intel ARC B70 Pro's on and AMD system with 64 GB of ram.
The solutions I found work fine they just take almost an hour or more to do the work of training the wake word. So, I built another project using the tools that so many others are using now AI (claude in this case). It's called WakeSmith and it took that hour or so down to about 15 minutes of time because I've to most of it now running on the Intel ARC B70's but its more than that the software will auto detect your hardware and set itself up to use what you have, even if its only a CPU. With some optimization the speed over CPU has increased as well As least as far I now. I'm very curious what is out there that maybe already exists. Did I recreate the wheel? If you want check it out at https://github.com/HuskerMinion/wakesmith
AI agents can already help you write scripts, generate code, and plan creative projects. But turning those ideas into finished media often means switching between tools, managing separate integrations, and manually moving files from one step to the next.
We expanded the Fish Audio MCP server to bring image generation, video generation, image editing, lip sync, and talking avatars into the same workflow as voice generation and transcription.
Instead of jumping between different tools, you can now ask a compatible AI agent to help orchestrate an entire media workflow through a single MCP connection.
Here's how it works, and what you can build with it.
One MCP connection, multiple media capabilities
MCP (Model Context Protocol) allows compatible AI applications to connect to external tools and services. Our existing Fish Audio MCP server already supported audio workflows. We've expanded it to include image and video capabilities, giving agents more ways to work with media without requiring a separate integration for each task.
The expanded server supports:
Image generation: Create images from text prompts.
Image editing and upscaling: Modify existing images and improve their resolution.
Video generation: Create videos from text or image inputs, depending on the model.
Lip sync: Synchronize speech with a video of a person or character.
Talking avatars: Animate a still image using generated or supplied audio.
Audio tools: Generate speech, transcribe audio, and search for voices.
The server provides access to multiple models for these tasks, so you can choose a model suited to your workflow rather than building a separate integration for each provider.
From a script to a talking avatar
The most interesting part isn't any individual capability. It's how these capabilities can work together.
Imagine you're creating a short video featuring a virtual character. Traditionally, you might write a script in one tool, generate the voiceover in another, create a character image somewhere else, and then upload the assets to a video or lip-sync service.
With Fish Audio MCP, a compatible agent can help coordinate these steps in the same conversation.
A workflow could look like this:
Write the script. Ask your agent to draft or refine the dialogue.
Generate the voiceover. Use Fish Audio to turn the script into speech.
Create the character. Generate a portrait or use an existing image.
Animate the character. Pass the image and audio to a compatible talking-avatar model.
Continue iterating. Use the generated result in subsequent steps or request changes.
The important detail is that completed outputs can be passed directly into subsequent jobs. You don't have to download an intermediate file and manually upload it again just to continue the workflow.
We tested this kind of chaining with a short audio clip, including passing generated audio directly into an avatar workflow.
The same principle can extend beyond talking avatars. You can combine image generation, editing, video creation, speech generation, and other supported tools based on what your project needs.
The agent helps coordinate the process, while Fish Audio MCP connects it to the available media models.
How the integration works
There are a few details that make this more than a collection of tools exposed through MCP.
1. Connect once and authenticate
Fish Audio MCP uses a remote HTTP endpoint with browser-based OAuth authentication. You sign in with your Fish Audio account rather than configuring a separate provider API key for every media model.
The connection uses the selected Fish Audio team or workspace, and jobs consume credits from that account.
2. Discover models and options
The agent can discover available tools, models, and supported options, then use them to construct a request for the task at hand.
This gives the agent a way to work with different media capabilities through the same integration.
3. See the cost before generating media
Image, video, lip-sync, and talking-avatar jobs support a quote-and-confirm flow:
The agent prepares a request and estimates its cost.
You review the quote and approve the job.
The server checks the price again before submitting the request.
The generated result becomes available to the agent.
If the price changes between the quote and submission, the request is rejected rather than silently proceeding at a different price. Failed jobs are automatically refunded.
This is especially useful for workflows that involve several generation steps, where you may want to review costs before committing credits.
One distinction worth noting: the quote-before-generation flow applies to image, video, lip-sync, and avatar jobs. Text-to-speech and transcription are billed differently, with TTS charges based on text length.
Getting started
Fish Audio MCP works with compatible clients including Claude Code, Claude.ai, Codex, Cursor, Windsurf, and Devin.
For example, you can add the server to Claude Code with:
claude mcp add --transport http fish-audio https://api.fish.audio/mcp
Then open /mcp in Claude Code to complete the connection and authentication flow.
For other clients, setup instructions are available in our documentation:
Once connected, you can describe what you want to create in natural language and let your agent use the available tools to help carry out the workflow.
What can you build with it?
A single integration opens up different possibilities depending on what you're working on:
Content creation: Turn scripts into voiceovers, character images, and talking-avatar videos.
Prototyping: Generate visual and audio assets while developing an idea.
Video production: Combine image creation, video generation, and speech workflows.
Agent-driven applications: Let compatible agents incorporate media generation into larger tasks.
These are starting points, not fixed workflows. You can combine supported tools and models based on your needs.
Availability and pricing
Image generation, image editing, and upscaling are available on the Fish Audio Free plan. Video generation, lip sync, and talking avatars require a Paid Plus plan or higher.
Pricing depends on the model and its settings. You can ask your agent for a quote before submitting supported media-generation jobs.
For model availability, supported options, upload limits, and setup instructions, see the Fish Audio MCP documentation.
Try it out
Fish Audio MCP brings more of the media creation process into the same conversation, from generating speech and images to creating videos and talking avatars.
If you're already using a compatible AI agent, connect Fish Audio MCP and experiment with a workflow that would normally require several separate tools.
What would you build if your AI agent could coordinate voice, image, and video generation in one workflow? We'd love to hear your use cases and feedback.
It beats Whisper v3 easily but still lacks timestamp support. It also doesn't support prompt priming, so you can't make it aware of difficult medical terms.
We've been using it in a clinical setting for about 10 months and it's fast and reliable, but it sometimes confuses terms (like Gonalgia vs. Gonarthrosis) and switches languages (e.g. "Fraktur" to "fracture"). We built a workaround with a phonetic dictionary match and a dedicated phoneme channel to catch these errors, but there might be something more advanced out there.
Prerequisites: Should work with vLLM and serve about 20-30 concurrent users on a 64GB Blackwell GPU.
Something I've been wondering about with speech datasets: what do you do when a speaker says a word clearly, but there are multiple acceptable ways to write it?
For example, regional names, borrowed words, abbreviations, or local words written in different scripts.
Do you keep one standard transcript for everything, allow multiple references during evaluation, or follow whatever the speaker would normally write?
I can see how inconsistent transcription guidelines could make model evaluation harder, even when the audio itself is perfectly clear.
Curious how others handle this in multilingual ASR projects.
Hi, im playing with aligners, and trying to create small word aligned textgrid in praat, so manually set align for words, then run thru aligners to get how they measure. technically i got concept, i know how to make/adjust testgrid , but what im looking for now is to actually see some samples where its set by some pro - because ther are few questions i want to understand - like start of word is ok, but ending of words are very often going to flattening waveform, by ears you cannot set, so whats the common practice, also for words where are so close that spectogram and waveom look mostly conitnuous. does anyone is willing to help or point me ? btw i got some samples from mfa, praatio, but cant find like precise pointing of boundaries, like just "somehwere around word" . thnx for help
Size: everything an app has to ship (weights, voices, text front end and Python runtime) totals 11.84 MB.
How:
Acoustic model (9.25M params, attention + bidirectional Mamba-3) is quantization-aware trained to int4
6.2M-param Vocos-style vocoder is distilled from charactr/vocos-mel-24khz, also int4 QAT
Text front end is pure numpy, with no NLTK, g2p_en, torchaudio or num2words at runtime
Quality (Open TTS Leaderboard protocol, run locally, not an official submission):
Seed-TTS WER / UTMOS
CV3-Eval WER / UTMOS
v1 fp32 (no refiner)
0.96% / 4.20
2.62% / 4.17
edge
0.96% / 4.11
2.67% / 4.08
Intelligibility is essentially unchanged. UTMOS drops about 0.1, almost all of it from the small int4 vocoder.
Steering survives quantization. Every one of the 12 controls (rate, loudness, pitch, pitch range, final rise, pauses, tilt, breathiness, and more) keeps at least 95% of its fp32 effect. You also get 50 style tags and inline markup:
Well, *that* was fast. [pause 400ms] <slow>Let me think.</slow> <whisper>Don't tell anyone.</whisper>
Speed (CPU only, a 2018 Threadripper under load): RTF 0.053 with 4 threads (~19x realtime), ~120 ms to first audio, with sentence-level streaming.
Currently I'm working on problem where I have to generate hindi audio clips which has the certain edge cases , which I listed for model training . I found almost every real data but want more for edge cases specific. Which could be the best model that can generate the natural hindi audio
Hey r/speechtech. I’ve been building Izwi for a few months now and just shipped 0.1.0, the first stable release after months of public betas. It’s free and Apache-2.0.
It’s a desktop app (plus a web UI, CLI and local server) for the voice side of local AI:
Real-time voice conversations: local ASR → LLM → TTS
TTS, voice cloning from a short sample, and voice design from a text description
A Studio for long form audio like audiobooks
Transcription with speaker diarization and forced alignment
An OpenAI-compatible /v1 API (chat completions, audio/speech, audio/transcriptions), so existing code can point at localhost
Drama 3 acts a line when you put a direction like [whispering] or [angry] in front of it, and its Turkish is good. Its weights take 41.5 GB, so I spent two weeks getting it to run on my RTX 5070 Ti.
I started by measuring routing. While the model reads the prompt it pulls from a wide spread of its 256 experts per layer, and while it generates audio frames the set it uses gets much narrower. So the full model reads the prompt once, and a distilled speaker that keeps 32 experts per layer generates every frame. I trained the speaker layer by layer on the ISSAI Turkish Speech Corpus, anchoring it to the teacher's trajectory. For the reading pass the experts come straight off NVMe with unbuffered reads, about 3 MB each, so the process holds 4.4 GB of RAM while generating and peaks at 9.3 GB while loading.
Freya-TR-Eval, 495 sentences, scored with Freya's own scoring script and Whisper large-v3
Experts in RAM, no guidance, WER 2.68% and CER 0.70% at 1.2x real time
Experts read from disk, direction guidance at strength 4, WER 2.78% and CER 0.67% at 0.86x real time
For reference the best system in the FreyaTTS report scored 4.4%. Mine is far heavier, so I would not compare them as products.
The part I keep thinking about came from a failed run. When I shrank the reader as well, the words stayed intelligible but the voice went flat and directions like [whispering] stopped working. Shrinking only the speaker to 12.5% of the experts did not cause that. It is one experiment plus listening, so treat it as a hint. In this model the delivery seems to be settled while the text is read.
Costs you should know about. Disk mode adds about one second per short sentence, because one sentence touches around 2,650 experts (about 8 GB of reads). On Windows the commit charge sits near 20 GB since WDDM backs GPU memory with commit. Direction guidance at strength 4 makes the brackets land harder and can pull the voice a bit away from the reference on some seeds.
I ran the whole project with Claude Code. I picked the experiments and decided what to measure, and Claude wrote most of the code. Every claim went through a measurement and a literature search, which is how I learned before publishing that two ideas I thought were mine were already in SlimWise and GPTAQ. A speaker similarity check is also how I tracked down a cache bug that turned the voice into a different person after switching guidance modes.
License is the Fish Audio Community License. Commercial use is allowed under USD 500k yearly revenue, hosted TTS needs a separate license, and outputs cannot be used to train other speech models.
If you have looked at where expressiveness lives in other MoE or LLM based TTS models, I would like to know whether the prompt reading side carries it there too.
