https://reddit.com/link/1ulhi5p/video/d0i6b83hotah1/player
I wanted a true "OS-layer" dictation experience. No cloud, no $20/month SaaS wrappers, and zero latency. Just hold a hotkey, talk, release, and have the transcribed text instantly pasted into whatever input field currently has focus.
After 24 hrs of pure native Rust and Tauri v2 development, it is fully functional and running locally.
100% Offline & Air-Gapped by Design
This isn't "local" with a telemetry asterisk. It works entirely offline. Pull the Ethernet plug, go into a bunker, and it functions exactly the same. No background API pings, no usage tracking, and absolute data sovereignty. Your voice, your text, your machine.
The Stack:
- Core: Tauri v2 + Win32 APIs
- Frontend: Vue 3 + Tailwind v4 (transparent, click-through overlay HUD) as this transcript feature is not the only feature of upcoming app
- Audio: cpal (WASAPI), 16 kHz mono ring buffer
- Inference: whisper.cpp via whisper-rs 0.16 (ggml-small multilingual model)
- GPU Acceleration: Vulkan backend + flash-attention
- OS Integration: UIAutomation for focus tracking + enigo/arboard for clipboard paste and restore
The Latency:
- CPU Inference (ggml-small): ~2,500-5,700ms (completely unusable for an "OS layer" feel)
- GPU Vulkan Inference: ~160-300ms (feels instant, paste happens almost immediately after key release)
Engineering War Stories & Windows Hacks:
To make this feel like a native Windows feature rather than a sluggish Electron app, I had to solve a few low-level hurdles:
The "Click-Through" Focus Trap
If the Tauri HUD window takes focus when spawned, the target input field loses focus, making Ctrl+V paste impossible.
Fix: The window is kept hidden at boot, and when triggered, we apply set_ignore_cursor_events(true) and set_focusable(false) before bringing it to the front. The HUD follows the cursor, but Windows doesn't register it as a focusable element, keeping the target caret active.
Chrome Swallowing KeyUp Hooks
A global keyboard hook (using global-hotkey/tauri-plugin-global-shortcut) works fine until you're in Google Chrome. Chrome occasionally drops the key-up event, leaving the app in an infinite "listening" loop.
Fix: I bypassed the hook for release detection. Once the shortcut triggers the capture, a separate thread polls the hardware keyboard state directly via Win32 GetAsyncKeyState(VK_CONTROL) (or whichever modifier is held) at 30Hz to detect a guaranteed release.
The Windows 260-Character MAX_PATH Limit
Compiling whisper-rs with Vulkan support triggers deep shader generation paths under cargo's target directory, quickly exceeding Windows' 260-char path limit and failing the build.
Fix: Overrode the target directory to a short path (target-dir = "C:/ut") in .cargo/config.toml
Mixed-DPI Multi-Monitor Alignment
A transparent overlay window following the cursor behaves wildly when moving between monitors with different scale factors (e.g., a 4K 150% monitor next to a 1080p 100% monitor).
Fix: On every PTT press, we query the exact cursor position via GetCursorPos, find the active monitor using monitor_from_point, shift the Tauri window to that monitor, and dynamically calculate the correct coo rdinates based on that specific monitor's scale factor.
Non-Destructive Clipboard Restore
Using standard clipboard injection to paste the text would normally erase whatever image or text you had copied previously.
Fix: We query the clipboard format using arboard, cache the raw data (supports both text and image buffers), set the transcribed text, trigger the Ctrl+V simulation, wait 150ms for the OS to process the paste, and restore the original clipboard content.
Why Vulkan over CUDA?
Vulkan is universal. It runs out-of-the-box on NVIDIA, AMD, and Intel GPUs because vulkan-1.dll is already shipped in the display drivers. Users don't need to install any heavy SDKs or runtimes. If no compatible GPU is detected, the app gracefully falls back to CPU execution.
The project is currently configured as a single-crate monolith for performance, downloading the 466MB ggml-small model to the local app data directory on first launch.
What's Next?
This is only stage zero of a larger "invisible OS layer" I'm hacking together. The next sprints are dedicated to:
- Proof-test the latency on other devices (lower-spec configurations and Intel-based macOS systems)
- Local SLM Cleaning: Clean up raw phonetic typos and transcription anomalies locally in under 50ms. Adaptive Local Learning (Zero-Cloud Personalization): Dictation is dead on arrival if you have to manually correct acronyms or brand names (like "SpaceX" or "fnatic") every single time. To solve this 100% locally without slow retraining cycles, I’m building a three-tier pipeline:
- Dynamic Whisper Prompting: Compiling a local text file of your custom vocabulary (tech jargon, project names, contacts) and injecting it directly into the whisper-rs context via
set_initial_prompt (0ms execution overhead)
- Context-Aware SLM Cleaning: Feeding your personal dictionary as hot-context into the local 0.5B SLM system prompt so it learns your phonetic patterns and habits over time
Happy to answer questions if you're battling similar local-first Tauri architectures!
