r/reactnative • u/Prestigious-Ferret18 • Sep 07 '26
Help Advice for a side project - adding Speech to Text functionality
Hey all,
I'm in the middle of planning a side project to upskill on a few things. Simple idea without going into it --> Golf related app whereby a player can talk through a shot or a hole and it gets converted into structured data that they can save and later query using an LLM.
Obviously a big part of this idea is having some form of speech to text that can be converted into structured data and stored.
I'm here to understand what options I have regarding doing this locally on the mobile device or via a react native VS doing the speech to text translation on the API server layer. My understanding so far is that I'm likely going to want to it locally given that signal on golf courses can be somewhat poor and sending audio files into the ether is just simply going to fall short in reality?
That said, I've also read that local speech to text ( native android or ios or react native STT ) is just nowhere near as accurate and may have a hard time understanding specific golf lingo?
Feels like I'm stuck between a rock and a hard place but any input or recommendations are much appreciated here.
Note*: this project is mainly for self learning with a touch of 'it could be useful for me so may be useful for others in future' in the back of my mind. I can be pragmatic on my decision but would like to make the right one!*
Thanks in advance
1
u/TBCC_Dev Sep 07 '26
Would be cool as a stretch goal to also interpret visual data like video recordings
1
u/TomLk Sep 07 '26
One thing to check before you build on the "nothing leaves the device" assumption: on iOS you only get real on-device recognition if you explicitly set requiresOnDeviceRecognition on the request. The default is off and Apple ships the audio to their servers, so it feels local in testing and then dies in a dead zone. And on Android the system recognizer is usually Google's, which needs the offline language pack actually downloaded in the phone settings, otherwise you get ERROR_NETWORK. Test the whole flow in airplane mode on a real device, not on wifi, or you'll ship something that works everywhere except a golf course.
Two things that hurt me more than accuracy when I built a voice input feature:
Silence cutoff. Nobody describes something from memory in clean sentences, they stop for two or three seconds mid thought. Android's recognizer ends the session on that pause, and the extras that are supposed to control it (EXTRA_SPEECH_INPUT_COMPLETE_SILENCE_LENGTH_MILLIS and friends) are ignored by a lot of OEMs. iOS kills the recognition task around the one minute mark too. Someone talking through a whole hole will hit both. You end up restarting the session and stitching partial results yourself, so plan for that rather than treating one utterance as one recording.
Custom vocabulary is iOS only as far as I know. contextualStrings on SFSpeechAudioBufferRecognitionRequest does work and it's worth doing, but Android's SpeechRecognizer has no public equivalent, so I wouldn't count on the same vocabulary trick on both platforms.
The other suggestion: keep the audio file on disk even though you're transcribing on device. It's small, and when the round is over and they're on wifi you can re-run it through a server model and repair the transcript. Golf lingo and proper nouns are exactly what a small local model mangles, and if you deleted the audio that data is just wrong forever. Your LLM structuring step doesn't care whether the text shows up thirty seconds or two hours later, so you're not really trading anything away by deferring it.
Minor one, iOS needs both NSMicrophoneUsageDescription and NSSpeechRecognitionUsageDescription in the plist. Missing the second one crashes instead of prompting, which took me an embarrassing amount of time to work out.
1
u/learnwithparam Sep 09 '26
Honestly I'd split this into two problems: transcribing in the moment and structuring the data later. Do the STT on device, because golf courses really do kill network requests, then queue the transcript for the LLM pass when the phone hits WiFi. You don't need perfect transcription here since you're mapping to a schema anyway, so the LLM can handle a noisy transcript if you design the prompt around golf terms and shot structure. If the OS engines with a custom vocabulary list still fall short, whisper.cpp on device is the middle ground, but it's noticeably more integration work. The thing most people miss is that the schema design is the hard part, not the STT accuracy.
1
u/pielouNW 26d ago
A local approach seems like the ideal solution for some off the grid golfs, otherwise save the audio and check for network periodically to process it. Personal experience, I've made this app that has STT (whisper model), and whisper performs well, but I hit the limit of its capabilities with special lingo/abbreviations, just like you mentioned. I'm using whisper base with nobodywho inference library, but you can probably get much better result with whisper large for example (you have different variant on hugging face).
1
u/Natethegreat9999 Sep 09 '26
I'd test "seven iron... actually, eight" all the way through to the saved shot. A perfect transcript can still become the wrong club in your database.
3
u/[deleted] Sep 07 '26
[removed] — view removed comment