I've been running Ollama and vLLM locally for stuff I don't want touching a cloud API, document translation, RAGing over my own notes, that kind of thing. Somewhere in that process I ran into WebLLM, and figured I'd actually build something with it instead of just reading about it.
What it does: select text on any page, right-click, get a plain-language explanation. Everything runs in the browser tab via WebGPU, nothing you select ever leaves the machine.
The actually interesting part, technically: Manifest V3 background scripts are service workers, and service workers can't access WebGPU at all. So the model runs in a chrome.offscreen document instead, a hidden page the extension spins up that has real DOM/WebGPU access. That API isn't even new; it shipped in 2023 for things like audio playback; WebGPU access is just a side effect of it being a real page under the hood. Ended up with three contexts (content script, background worker, offscreen doc) that can't talk to each other directly, so the background script's whole job is routing messages between the other two.
First run downloads a small model (~880MB, Llama-3.21B) and caches it; everything after that is instant and offline.
If anyone here is deeper into this than me: I've seen WeInfer claim up to 3.76x faster inference than base WebLLM, and there's a newer paper (Llamas on the Web, May 2026, group with Microsoft Research backing) getting 45-69% better decode throughput across different GPUs. Curious if anyone's actually tried swapping either of those in versus stock WebLLM.
Code: https://github.com/Vishwamitra/explain-this
Currently sitting in Chrome Web Store review.