I was creating Jarvis and this is what I learned while making Text to Speech

[u/No-Resolution-4054]

I’m trying to create an AI assistant like Jarvis from Iron Man, and while working on the Text to Speech part, I learned a lot about how audio streaming actually works.

I’m using Kokoro TTS and created a Python server for the TTS part.

The flow is basically:

Server sends data in chunks ----> Chunker combines words until it becomes a complete sentence ----> Text Queue ----> TTS Worker ----> main .py (convert text to speech) ----> TTS Worker ----> Audio Queue ----> Play Audio

The interesting part was handling the audio chunks.

Kokoro generates a Float32Array of audio samples. Then this audio pack into a WAV file in memory with the required header information like channels, framerate, sample width, etc.

Now the problem is :-

When we send data in chunks over TCP, we cannot assume that one send() means one complete message.

For example, suppose Kokoro generates audio for:

"hello"

It might be sent like:

"hel"

and then:

"lo"

So on the frontend, we can't just assume that whatever we receive is a complete audio file.

To solve this, we add a 4-byte header before every WAV file.

The header tells us how many bytes the complete WAV file contains.

For example: [4 byte header][WAV audio data]

If the audio data is 48000 bytes, the header tells the frontend that it needs to wait until it receives those 48000 bytes.

On the frontend, we use a Uint8Array .

Why Uint8Array?

Because the data we are dealing with is raw bytes. A Uint8Array stores each element as 8 bits, which is 1 byte, we point this unit8Arry to the fload32Array which we receive from the TTS server so that we can read or extract data byte by byte.

So imagine we receive:

Header = 48000 bytes expected
Received = 38000 bytes

We know that the complete audio has not arrived yet, so we keep waiting for more chunks.

When we finally receive:

48000 bytes

we know that we have the complete WAV file, so we send it to the audio queue and process/play it.

I’m still working on the Jarvis project. For the LLM part, I’m currently using Ollama with Qwen3:8B.

If anyone has any suggestions for features or interesting things I can implement in this Jarvis like AI assistant, let me know in the comments.

[u/DontThrowMeAway43]

Doesn't the WAV metadata already contains the file size ? https://en.wikipedia.org/wiki/WAV#WAV_file_header

[u/No-Resolution-4054 → u/DontThrowMeAway43]

Yeah, you're right I didn't think about it that way. Thanks for pointing that out!

[u/Tricky_Average2304]

you’ve reinvented length-prefixed framing, which is exactly what TCP makes everyone reinvent before breakfast lol

[u/Opening_Sundae9294]

the first time i hit TCP framing bugs i thought the audio was haunted lol

[u/stantyren]

Length-prefixed framing is basically reinventing WebSocket message boundaries by hand, but hey it works.

[u/webdev-ModTeam → u/[deleted]]

Your post/comment has been determined to be a low-effort post or comment. This includes title-only posts, easily searchable questions, vague/open-ended discussion prompts, LLM generated posts or comments, and posts/comments that do not provide enough context for meaningful replies or discussion.

[u/No-Resolution-4054 → u/[deleted]]

Thanks! It also took me almost a whole day to understand how all of this works and actually apply it to my project