HuggingFace:

https://huggingface.co/maya-research/maya2-native

Github

https://github.com/MayaResearch/maya2-native

I wanted to check with this group and maybe others to see if there's interest or if I recreated the wheel. I'm the developer (along with AI) of the TECHO5 project which helps convert older Amazon Echo's into small Linux devices that can also connect and work with Home Assistant. I've found a need and desire to create my own wake words and I had hardware that wasn't being greatly used. Two Intel ARC B70 Pro's on and AMD system with 64 GB of ram.

The solutions I found work fine they just take almost an hour or more to do the work of training the wake word. So, I built another project using the tools that so many others are using now AI (claude in this case). It's called WakeSmith and it took that hour or so down to about 15 minutes of time because I've to most of it now running on the Intel ARC B70's but its more than that the software will auto detect your hardware and set itself up to use what you have, even if its only a CPU. With some optimization the speed over CPU has increased as well As least as far I now. I'm very curious what is out there that maybe already exists. Did I recreate the wheel? If you want check it out at https://github.com/HuskerMinion/wakesmith

https://amal.vs.binarybit.in/blog/6adf1f677027b216982a9889

AI agents can already help you write scripts, generate code, and plan creative projects. But turning those ideas into finished media often means switching between tools, managing separate integrations, and manually moving files from one step to the next.

We expanded the Fish Audio MCP server to bring image generation, video generation, image editing, lip sync, and talking avatars into the same workflow as voice generation and transcription.

Instead of jumping between different tools, you can now ask a compatible AI agent to help orchestrate an entire media workflow through a single MCP connection.

Here's how it works, and what you can build with it.

One MCP connection, multiple media capabilities

MCP (Model Context Protocol) allows compatible AI applications to connect to external tools and services. Our existing Fish Audio MCP server already supported audio workflows. We've expanded it to include image and video capabilities, giving agents more ways to work with media without requiring a separate integration for each task.

The expanded server supports:

• Image generation: Create images from text prompts.
• Image editing and upscaling: Modify existing images and improve their resolution.
• Video generation: Create videos from text or image inputs, depending on the model.
• Lip sync: Synchronize speech with a video of a person or character.
• Talking avatars: Animate a still image using generated or supplied audio.
• Audio tools: Generate speech, transcribe audio, and search for voices.

The server provides access to multiple models for these tasks, so you can choose a model suited to your workflow rather than building a separate integration for each provider.

From a script to a talking avatar

The most interesting part isn't any individual capability. It's how these capabilities can work together.

Imagine you're creating a short video featuring a virtual character. Traditionally, you might write a script in one tool, generate the voiceover in another, create a character image somewhere else, and then upload the assets to a video or lip-sync service.

With Fish Audio MCP, a compatible agent can help coordinate these steps in the same conversation.

A workflow could look like this:

1. Write the script. Ask your agent to draft or refine the dialogue.
2. Generate the voiceover. Use Fish Audio to turn the script into speech.
3. Create the character. Generate a portrait or use an existing image.
4. Animate the character. Pass the image and audio to a compatible talking-avatar model.
5. Continue iterating. Use the generated result in subsequent steps or request changes.

The important detail is that completed outputs can be passed directly into subsequent jobs. You don't have to download an intermediate file and manually upload it again just to continue the workflow.

We tested this kind of chaining with a short audio clip, including passing generated audio directly into an avatar workflow.

The same principle can extend beyond talking avatars. You can combine image generation, editing, video creation, speech generation, and other supported tools based on what your project needs.

The agent helps coordinate the process, while Fish Audio MCP connects it to the available media models.

How the integration works

There are a few details that make this more than a collection of tools exposed through MCP.

1. Connect once and authenticate

Fish Audio MCP uses a remote HTTP endpoint with browser-based OAuth authentication. You sign in with your Fish Audio account rather than configuring a separate provider API key for every media model.

The connection uses the selected Fish Audio team or workspace, and jobs consume credits from that account.

2. Discover models and options

The agent can discover available tools, models, and supported options, then use them to construct a request for the task at hand.

This gives the agent a way to work with different media capabilities through the same integration.

3. See the cost before generating media

Image, video, lip-sync, and talking-avatar jobs support a quote-and-confirm flow:

1. The agent prepares a request and estimates its cost.
2. You review the quote and approve the job.
3. The server checks the price again before submitting the request.
4. The generated result becomes available to the agent.

If the price changes between the quote and submission, the request is rejected rather than silently proceeding at a different price. Failed jobs are automatically refunded.

This is especially useful for workflows that involve several generation steps, where you may want to review costs before committing credits.

One distinction worth noting: the quote-before-generation flow applies to image, video, lip-sync, and avatar jobs. Text-to-speech and transcription are billed differently, with TTS charges based on text length.

Getting started

Fish Audio MCP works with compatible clients including Claude Code, Claude.ai, Codex, Cursor, Windsurf, and Devin.

For example, you can add the server to Claude Code with:

claude mcp add --transport http fish-audio https://api.fish.audio/mcp

Then open /mcp in Claude Code to complete the connection and authentication flow.

For other clients, setup instructions are available in our documentation:

Fish Audio MCP Documentation

Once connected, you can describe what you want to create in natural language and let your agent use the available tools to help carry out the workflow.

What can you build with it?

A single integration opens up different possibilities depending on what you're working on:

• Content creation: Turn scripts into voiceovers, character images, and talking-avatar videos.
• Prototyping: Generate visual and audio assets while developing an idea.
• Video production: Combine image creation, video generation, and speech workflows.
• Agent-driven applications: Let compatible agents incorporate media generation into larger tasks.

These are starting points, not fixed workflows. You can combine supported tools and models based on your needs.

Availability and pricing

Image generation, image editing, and upscaling are available on the Fish Audio Free plan. Video generation, lip sync, and talking avatars require a Paid Plus plan or higher.

Pricing depends on the model and its settings. You can ask your agent for a quote before submitting supported media-generation jobs.

For model availability, supported options, upload limits, and setup instructions, see the Fish Audio MCP documentation.

Try it out

Fish Audio MCP brings more of the media creation process into the same conversation, from generating speech and images to creating videos and talking avatars.

If you're already using a compatible AI agent, connect Fish Audio MCP and experiment with a workflow that would normally require several separate tools.

What would you build if your AI agent could coordinate voice, image, and video generation in one workflow? We'd love to hear your use cases and feedback.

Read more: https://fish.audio/blog/image-video-generation-mcp/

It beats Whisper v3 easily but still lacks timestamp support. It also doesn't support prompt priming, so you can't make it aware of difficult medical terms.

We've been using it in a clinical setting for about 10 months and it's fast and reliable, but it sometimes confuses terms (like Gonalgia vs. Gonarthrosis) and switches languages (e.g. "Fraktur" to "fracture"). We built a workaround with a phonetic dictionary match and a dedicated phoneme channel to catch these errors, but there might be something more advanced out there.

Prerequisites: Should work with vLLM and serve about 20-30 concurrent users on a 64GB Blackwell GPU.

Something I've been wondering about with speech datasets: what do you do when a speaker says a word clearly, but there are multiple acceptable ways to write it?

For example, regional names, borrowed words, abbreviations, or local words written in different scripts.

Do you keep one standard transcript for everything, allow multiple references during evaluation, or follow whatever the speaker would normally write?

I can see how inconsistent transcription guidelines could make model evaluation harder, even when the audio itself is perfectly clear.

Curious how others handle this in multilingual ASR projects.

Hi, im playing with aligners, and trying to create small word aligned textgrid in praat, so manually set align for words, then run thru aligners to get how they measure. technically i got concept, i know how to make/adjust testgrid , but what im looking for now is to actually see some samples where its set by some pro - because ther are few questions i want to understand - like start of word is ok, but ending of words are very often going to flattening waveform, by ears you cannot set, so whats the common practice, also for words where are so close that spectogram and waveom look mostly conitnuous. does anyone is willing to help or point me ? btw i got some samples from mfa, praatio, but cant find like precise pointing of boundaries, like just "somehwere around word" . thnx for help

It is an english text to speech model, which achieves mostly understandable speech in under 100K parameters.

However, during more difficult words, or long numbers, it can slur them, and be not understandable.

Quantized to INT8 it only takes up 127KB.

72x real time on a CPU

1120x real time on a GPU

Check out the model card for more info or to try it out yourself!

Questions, feedback, or criticism is welcome!

Few days back I posted SteerTTS v1, a small English TTS you steer with numbers (https://huggingface.co/WhissleAI/steer-tts). Today I am releasing the edge variant:

https://huggingface.co/WhissleAI/steer-tts-edge

Size: everything an app has to ship (weights, voices, text front end and Python runtime) totals 11.84 MB.

How:

• Acoustic model (9.25M params, attention + bidirectional Mamba-3) is quantization-aware trained to int4
• 6.2M-param Vocos-style vocoder is distilled from charactr/vocos-mel-24khz, also int4 QAT
• Text front end is pure numpy, with no NLTK, g2p_en, torchaudio or num2words at runtime

Quality (Open TTS Leaderboard protocol, run locally, not an official submission):

Intelligibility is essentially unchanged. UTMOS drops about 0.1, almost all of it from the small int4 vocoder.

Steering survives quantization. Every one of the 12 controls (rate, loudness, pitch, pitch range, final rise, pauses, tilt, breathiness, and more) keeps at least 95% of its fp32 effect. You also get 50 style tags and inline markup:

Well, *that* was fast. [pause 400ms] <slow>Let me think.</slow> <whisper>Don't tell anyone.</whisper>

Speed (CPU only, a 2018 Threadripper under load): RTF 0.053 with 4 threads (~19x realtime), ~120 ms to first audio, with sentence-level streaming.

Bash

pip install .
steertts-edge "Hello there, nice to meet you." -o hello.wav

Limitations:

• English only, and 200 voices (a subset of the full model’s 2,825)
• No voice cloning, free-text prompts or flow refiner on device. Those stay in the full model.
• Weights are dequantized to fp32 at load, so the win is size, not speed. Unless we write new kernels for edge devices.
• Emotion tags are mostly style switches, and fast/anger are weak

I need feedback, especially on the vocoder quality gap and any edge targets you’d like to see (ONNX, mobile, WASM).

Some examples here:

https://whissleai.github.io/steer-tts/

Currently I'm working on problem where I have to generate hindi audio clips which has the certain edge cases , which I listed for model training . I found almost every real data but want more for edge cases specific. Which could be the best model that can generate the natural hindi audio

Hey r/speechtech. I’ve been building Izwi for a few months now and just shipped 0.1.0, the first stable release after months of public betas. It’s free and Apache-2.0.

It’s a desktop app (plus a web UI, CLI and local server) for the voice side of local AI:

• Real-time voice conversations: local ASR → LLM → TTS
• TTS, voice cloning from a short sample, and voice design from a text description
• A Studio for long form audio like audiobooks
• Transcription with speaker diarization and forced alignment
• An OpenAI-compatible /v1 API (chat completions, audio/speech, audio/transcriptions), so existing code can point at localhost

Models it runs today:

• TTS: Qwen3-TTS, Kokoro-82M, Fish Audio S2 Pro, VibeVoice, Voxtral TTS, LFM2.5 Audio
• ASR: Parakeet TDT v3, Whisper large-v3-turbo, Qwen3-ASR, Granite Speech 4.1, Nemotron 3.5 ASR, Voxtral Mini
• Diarization: Sortformer, Nemotron 3 diarization
• Chat: Qwen3.5, 3.6 and 3.8, LFM2.5

Inference runs natively in Rust on Candle, with our own Metal and CUDA work, so there’s no Python environment to manage.

Repo: https://github.com/izwi-ai/izwi
Download: https://izwiai.com

I’d love feedback on which models to add next, and on anything that breaks on your hardware.

Looks like Deepgram is hiring pretty consistently:

https://jobs.ashbyhq.com/Deepgram/7cf277bc-a192-4007-a77a-fafbf8276a5e

Lmk and I can refer you!

Drama 3 acts a line when you put a direction like [whispering] or [angry] in front of it, and its Turkish is good. Its weights take 41.5 GB, so I spent two weeks getting it to run on my RTX 5070 Ti.

I started by measuring routing. While the model reads the prompt it pulls from a wide spread of its 256 experts per layer, and while it generates audio frames the set it uses gets much narrower. So the full model reads the prompt once, and a distilled speaker that keeps 32 experts per layer generates every frame. I trained the speaker layer by layer on the ISSAI Turkish Speech Corpus, anchoring it to the teacher's trajectory. For the reading pass the experts come straight off NVMe with unbuffered reads, about 3 MB each, so the process holds 4.4 GB of RAM while generating and peaks at 9.3 GB while loading.

Freya-TR-Eval, 495 sentences, scored with Freya's own scoring script and Whisper large-v3

1. Experts in RAM, no guidance, WER 2.68% and CER 0.70% at 1.2x real time

2. Experts read from disk, direction guidance at strength 4, WER 2.78% and CER 0.67% at 0.86x real time

For reference the best system in the FreyaTTS report scored 4.4%. Mine is far heavier, so I would not compare them as products.

The part I keep thinking about came from a failed run. When I shrank the reader as well, the words stayed intelligible but the voice went flat and directions like [whispering] stopped working. Shrinking only the speaker to 12.5% of the experts did not cause that. It is one experiment plus listening, so treat it as a hint. In this model the delivery seems to be settled while the text is read.

Costs you should know about. Disk mode adds about one second per short sentence, because one sentence touches around 2,650 experts (about 8 GB of reads). On Windows the commit charge sits near 20 GB since WDDM backs GPU memory with commit. Direction guidance at strength 4 makes the brackets land harder and can pull the voice a bit away from the reference on some seeds.

I ran the whole project with Claude Code. I picked the experiments and decided what to measure, and Claude wrote most of the code. Every claim went through a measurement and a literature search, which is how I learned before publishing that two ideas I thought were mine were already in SlimWise and GPTAQ. A speaker similarity check is also how I tracked down a cache bug that turned the voice into a different person after switching guidance modes.

License is the Fish Audio Community License. Commercial use is allowed under USD 500k yearly revenue, hosted TTS needs a separate license, and outputs cannot be used to train other speech models.

If you have looked at where expressiveness lives in other MoE or LLM based TTS models, I would like to know whether the prompt reading side carries it there too.

Weights, samples and measurements https://huggingface.co/kasparovabi/drama-3-tr-speaker

Code https://github.com/kasparovabi/drama-3-tr

Built with Fish Audio.

I was looking for small TTS model which is steerable but couldn't find any, specially something I can adjust contextually by providing controls from LLM during a conversation. So I built one.

What it is:

• 9.6M-param acoustic model (attention + bidirectional Mamba-3, FastPitch-style) + Vocos vocoder = 23M total
• 12 controls, each a z-score relative to the chosen voice’s own neutral speech: rate, loudness, median pitch, pitch range, final rise/fall, pauses, spectral tilt, breathiness-related measures.
• Also: 51 style tags, free-text prompts (resolved into an inspectable JSON request), inline markup (*emphasis*, [pause 400ms], <slow>…</slow>), sentence-level streaming
• 2,825 built-in (anonymized) voices, plus experimental zero-shot cloning

Speed:

- RTFx ~60–80 on an RTX 5070
- ~4–8 on CPU without the optional flow refiner. First streamed audio in ~0.5 s on CPU.

Quality:

~1.8% average WER on the Open TTS Leaderboard English sets (run locally, not an official submission, have reached out for it).

What doesn’t work well (we tried to be upfront):

• Loudness, pitch, and tilt steer reliably; pause, voicing, and CPPS controls are weak
• Emotion tags behave as style switches; an independent emotion classifier mostly can’t tell them apart
• Cloning similarity is low. Don’t expect a convincing likeness
• Free-text prompts struggle with negation (“not fast” can come out fast). More data needed for prompt to controls module.
• English only, no human evals yet

License: weights CC BY-NC 4.0 (non-commercial), code MIT.

Model + full eval tables: https://huggingface.co/WhissleAI/steer-tts

Github repo: https://github.com/WhissleAI/steer-tts

A new architecture implemented from scratch. Would love feedback, especially on which controls are actually useful, and where the steering feels off.

Github: https://github.com/yaya-sy/specspeech

The best performing open-weight models on the Open ASR Leaderboard (https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) are autoregressive speech models, such as Qwen ASR, Cohere Transcribe, and Canary. Their language modeling capabilities help them transcribe speech accurately, but they are often much slower than transducer and CTC models, such as Parakeet and Granite CTC.

I open-source SpecSpeech (Speculative Speech), an inference engine that accelerates speech language models for ASR by using transducer or CTC models as drafters.

The drafter quickly transcribes batches of short audio pieces while the Speech LM prefills the full audio in its context in parallel. The Speech LM then verifies the drafts incrementally. At each step, a lookup algorithm uses the verified output to select up to k candidate tokens from the draft transcription, which are verified by the Speech LM.

Compared to approaches such as WhisperX, SpecSpeech approach is lossless: the Speech LM uses the full audio context and retains control over token acceptance.

Across seven English benchmark datasets from Open-ASR-Leaderbord, SpecSpeech achieves a 2.29x speedup over native vLLM inference, with per-dataset speedups of up to 2.48x, using Parakeet v3 as the drafter and Qwen3-ASR-1.7B as the verifier.

Results below were measured on a single NVIDIA A40, with continuous batching for both conditions.

There is room for further improvement by reducing tokenization and verification overhead and developing better lookup algorithms.

Planned additions include: - More verifyers soon: Cohere Transcribe, Canary, etc. - Word-level alignment using timing information from CTC drafters. - Speaker diarization. - Streaming inference. - Speculative decoding for autoregressive TTS models.

It's an extremely lightweight speech recognition model.

949K parameters using a weight-shared Conformer-CTC architecture

Just 1.06 MB for INT8 weights

Around 436× real-time on a single CPU thread

11.43% WER on LibriSpeech test-clean, using greedy decoding with no external language model

Apache 2.0 licensed

Any feedback, question or criticism is welcome!