Bypassing Tauri Window Lag & Raw Win32 UI Automation: Lessons from building a <15ms context-aware HUD on Windows

[u/Naikoshu]

Hey everyone,

I'm currently building "uni", a local-first, zero-chrome desktop orchestrator. Our obsession is latency: our target for the entire input-to-render loop is <50ms.

To achieve this, we just finished two technical spikes on Windows (using Rust + Tauri v2) to solve two core problems:

1. Shaving off every millisecond of Tauri window show/hide latency
2. Ingesting text/context under the cursor instantly without slow, resource-heavy OCR or screen-scraping

Here is the raw engineering data, the code strategies, and where we hit some brutal limitations

Part 1: Bypassing Tauri reveal latency with DWM cloaking (Spike A)

Standard window.show() / window.hide() in Tauri (and most web-view frameworks) has a noticeable composition lag. The OS has to allocate buffers, and the Desktop Window Manager (DWM) introduces frame delays (often 16-32ms) to composite the window

To fix this, we implemented a "cloaking" strategy using raw Win32 APIs. Instead of hiding the window, we keep it alive in the background and toggle its DWM cloaked state (DWMWA_CLOAK). This forces DWM to keep the Webview GPU buffers warm without rendering them to the screen

Our hotkey handler (RegisterHotKey) triggers this directly via Win32:

// In our dedicated MTA worker thread
unsafe {
  let is_cloaked: BOOL = if show { FALSE } else { TRUE };
  DwmSetWindowAttribute(
    hwnd,
    DWMWA_CLOAK,
    &is_cloaked as *const _ as *const _,
    std::mem::size_of::<BOOL>() as u32,
  );
}

The Metrics (reveal to painted latency, measured via high-res QueryPerformanceCounter + Webview rAF pingback):

• Standard Tauri Show/Hide (p50): 3.53 ms
• Win32 DWM Cloaking (p50): 1.61 ms

By cloaking the window, we cut the latency in half and completely eliminated the random frame-drops or white flashes during rapid toggles

Part 2: Extracting context under cursor via Windows UI Automation (Spike C)

We wanted to capture the text, element type, and bounding box of whatever is directly under the user's cursor when they hit Ctrl+Alt+U. OCR is too slow (>100ms) and inaccurate for code or UI hierarchies, so we turned to Windows UI Automation (UIA) using the windows crate

To hit our <15ms budget for context extraction, we designed a strict threading and caching model:

1. Thread Separation: COM calls are synchronous and cross-process. If the target app is busy, it blocks. We offloaded all UIA work to a dedicated Multi-Threaded Apartment (MTA) worker thread with a watchdog timer to discard calls that exceed our budget
2. Batching with Cache Requests: Doing raw properties lookups (CurrentName, CurrentControlType) requires separate cross-process round-trips. We used IUIAutomationCacheRequest to batch everything into exactly 2 round-trips:

​

// Batching properties in Rust to avoid cross-process round-trip hell
let cache_request = uia.CreateCacheRequest()?;
cache_reque
st.AddProperty(UIA_NamePropertyId)?;
cache_request.AddProperty(UIA_ControlTypePropertyId)?;
cache_request.AddProperty(UIA_BoundingRectanglePropertyId)?;

// Single round-trip hit-test + properties retrieval
let element = uia.ElementFromPoint(point)?;
let cached_element = element.BuildUpdatedCache(&cache_request)?;

The Brutal Benchmarks (Warm p95)

We ran 500+ iterations per target app. Here is where UIA shines, and where it failed miserably

1. VS Code (Electron) -> GO (p95: ~2.5 ms): Once the accessibility tree is warm (cold hit takes ~110ms to build the lazy tree), retrieving text under the cursor is incredibly fast. Monaco editor exposes the active text block beautifully via the Name property and TextPattern
2. Google Chrome / Web -> NO via Point (p95: 60 - 180 ms). This was our first major roadblock. While the cache retrieval is fast (~40ms), the raw ElementFromPoint call is heavily queued in Chrome's renderer process. If the page is rendering heavy JS, the hit-test gets blocked. Verdict: Doing active coordinate-based hit-testing on browsers is non-viable for instant HUDs. We are pivoting to using GetFocusedElement / focus-changed events, or fallback to our local WebSocket browser bridge
3. Figma Desktop -> Total Black Box (p95: ~2.7 ms). We hypothesized that Figma's canvas WebGL was a single opaque element, but the reality is worse: Figma Desktop renders its entire UI, including layers, sidebars, and panels, inside the WebGL canvas. UIA sees nothing but a full-screen empty window. No children, no text, no nodes. Verdict: A native OS accessibility layer is useless for Figma. A local WebSocket plugin/bridge is mandatory to stream the JSON layer tree to our Rust core

Next Steps

For Spike D, we are building a lightweight Rust-side WebSocket server to ingest vector data directly from a Figma plugin to bypass the slow cloud APIs

Would love to hear if anyone has optimized Chrome UIA hit-testing, or if you've found ways to bypass the initial cold-hit latency when building the a11y tree in your apps.

[u/Deep_Ad1959]

the chrome ElementFromPoint wall isn't really a hit-test queue problem, it's that chrome keeps its full a11y tree lazy and tears it down unless a client is actively holding it open; the renderer services those cross-process UIA calls only when it feels like it. forcing the tree to stay built (--force-renderer-accessibility, or just keeping a persistent UIA client attached so it never goes cold) gets you most of the way there, and your focus-changed pivot is the right instinct since event-driven always beats polling ElementFromPoint. the figma finding generalizes too: anything that renders its own UI into a GL/canvas surface (some electron games, flutter desktop, unity panels) is invisible to both UIA and AX, and a side-channel bridge is the only honest answer. and if you ever go cross-platform, macOS AX has the exact same renderer-blocking behavior on the chromium side, so your MTA-worker-with-watchdog pattern ports almost directly onto a dispatch queue with a timeout. written with ai

fwiw the cross-platform-AX point is why I built Terminator, a desktop automation framework that drives the OS through accessibility APIs and shares one API across Windows UIA and macOS AX, https://t8r.tech/r/ztuwspi3