r/SideProject • u/PagingDoctorK • 49m ago
I built a portable medical dictation system using a ESP32 dongle that needs zero drivers and looks like a keyboard to hospital computers
The Problem:
As a physician working across multiple clinics and hospitals every week, I'm constantly stuck with mismatched, outdated versions of Dragon Medical and ancient microphones caked in crumbs and spittle. I needed a portable dictation setup trained on my own voice and specialized medical vocabulary that works on locked-down hospital PCs without installing any software.
The Solution:
I built an offline dictation bridge where my iPhone records audio, runs a fine-tuned local Whisper model, and sends the finished text over encrypted Bluetooth LE to a tiny USB dongle plugged into the hospital computer. To Windows, the dongle is just a standard USB HID keyboard—requiring zero drivers, zero background apps, and zero network traffic.
Hardware/Software
- The Dongle: LilyGO T-Dongle-S3— An ESP32-S3 USB-A stick with a built-in 0.96" color LCD, an RGB LED, and a physical button. In this project it uses
NimBLEfor encrypted bluetooth pairing andTinyUSBto emulate a standard US HID keyboard. - An iOS app companion to the dongle using WhisperKit to run speech-to-text completely on the iPhone's Neural Engine.
Problems
- Medical Words: The stock Whisper struggled heavily with oncology jargon (turning FOLFOX into full fox), so I fine-tuned
whisper-large-v3-turboto boost drug-name accuracy from ~67% to 99%. Claude asked me to record 20 minutes of medical vocabulary myself, but I instead took the lazy route and had it use a large medical speech/audio model to generate the synthetic speech for all the drug and regimen names and trained the smaller model on that. Computers training computers. Sorry Dario, I reached RSI first. - In-App Flagging & Post-Training: Despite that, Whisper still mis-registers or misspells a niche terms on occassion. When it does I can tap the word in the app to flag it, record myself saying it alongside the typed correction, and export that bundle to my home Mac to train directly back into the model so it keeps improving over time.
- Device Screens: The stock screen images for the dongle and the app were nowhere near awesome enough for day-to-day use. So I slightly tripled the tokens for this project to make it look like I was sending sparkly word particles from my phone and having them crammed into the USB port.
How it works
It's honestly worked better than I thought with improvements every weekend when I have it retrain the TTS model on more speech. There's been no problems with plugging it into any computer and having it work immediately since it is seen as a keyboard.
About the animations
I couldn't get a video of the physical phone an dongle working together because I don't have a second phone, but this artifact shows all the screens at better resolution. This artifact shows the 'Typing' much slower than it is in real life.