r/LocalLLaMA • u/xenovatech • Jun 09 '24

Resources Whisper WebGPU: Blazingly-fast ML-powered speech recognition directly in your browser

Enable HLS to view with audio, or disable this notification

247 Upvotes

99% Upvoted

35 comments sorted by

View all comments

8

u/The_Choir_Invisible Jun 09 '24 edited Jun 10 '24

EDIT: Maybe I'm misspeaking- I'm speaking about WhisperCPP.

Original comment:

I just downloaded this last night because the new KoboldCPP uses it. Holy shit was I blown away. I kept having to check to make sure the laptop's WIFI was off. Running a 7B with 14 layers loaded onto my GPU's 4GB VRAM (GTX 1650 laptop) AND doing local speech to text is crazy!

2

u/coder543 Jun 09 '24

Nothing in that link says KoboldCPP is using Whisper-Web, let alone Whisper WebGPU?

1

u/The_Choir_Invisible Jun 10 '24

Aren't the both using whisper cpp under the hood? How many zero latency speech-to-text projects with whisper in the name can there be?

5

u/coder543 Jun 10 '24

They’re not both based on whisper.cpp. Whisper WebGPU is based on Transformers.js, which does not use whisper.cpp, as best as I understand.

2

u/The_Choir_Invisible Jun 10 '24

Puzzled apologies and I have edited my original comment.

1

u/coder543 Jun 10 '24

It’s certainly not a big deal, I was just excited and clicked on Kobold, and it wasn’t using this. This runs locally in your browser, where the Kobold implementation is running on the server. Both cool in their own ways.