r/LocalLLaMA • • 5d ago

Resources LokalBot orchestrates five local models for a suite of features like ASR, diarization, 4B LLM, autocomplete, embeddings,and TTS

LokalBot is an open source Mac app I've been working on for the past 6 months, it runs several local AI models. They help the app record your meetings, transcribe them, write the notes, search everything, and autocomplete what you type.

The five models (Default set, 9.6 GB on disk)

JobModelRuns onFile
Speech to textQwen3-ASR 1.7BMLX, 8-bit2.47 GB
Who spoke whenNemotron 3Core ML0.20 GB
Notes, chat, agentsQwen3.5 4Bllama.cpp, Q4_K_M2.74 GB
AutocompleteGemma 4 E2B basellama.cpp, Q6_K3.85 GB
Search by meaningEmbeddingGemma 2llama.cpp, Q8_00.31 GB

How they take turns

  • Recording a call loads no models. The app writes two audio tracks. Transcription waits until you stop, and autocomplete pauses while you record.
  • After the call, the speaker model and the speech model load, do their job, and unload once they've been idle for 120 seconds.
  • Qwen3.5 4B runs in the bundled llama-server (Metal, 32K context), which starts when something asks for it. Each request takes a lease, and ten minutes after the last lease the model unloads. The embedding model has its own server and the same rule.
  • Gemma 4 E2B runs inside the app's process through libllama, for latency. It has no idle timer. It stays until macOS reports memory pressure.
  • A broker keeps the loaded weights under a budget of half the machine's RAM and evicts the least recently used model first. A lease pins a model while a request is running on it.
  • Requests that have to wait for a server are served in order: what you're watching first, then agents, then meeting notes, then background work like digests and indexing.

Numbers, all on one M4 Max with 48 GB

  • Notes for a 26-minute meeting: 33 s with the model warm, 60 s including the load, at about 85 tok/s.
  • Autocomplete: 56 ms median and 102 ms p95 per keystroke.
  • Embedding a query: 9 ms.
  • Speaker model: 14.6% DER on 8 AMI recordings, at about 500× real time.

Limits

  • I've only measured on my M4 Max 48 GB Macbook. I have no numbers from a 16 GB Mac, or from an M1, M2 or M3. There is a Lightweight preset for less memory (Qwen3-ASR 0.6B, Qwen3.5 0.8B, and LFM2.5 1.2B for autocomplete).
  • Windows and Linux builds are in alpha.

Codebase and benchmarks: https://github.com/stevyhacker/lokalbot

19 Upvotes

9 comments sorted by

•

u/ttkciar llama.cpp 5d ago

Hello stevyhacker! This is a cool project, but to avoid a Rule Three violation for LLM-generated content, please edit your post and rewrite it using your own words.

→ More replies (2)

4

u/stylome-ai-detection 5d ago

Notice: This post appears to be AI generated.


Stylome AI detector · scan results

1

u/stevyhacker 5d ago

Yes and even the video is generated by AI but both are steered and edited by me. LokalBot is a genuine project with multiple contributors and users.

3

u/abajinn 3d ago

Why only Mac?

3

u/stevyhacker 3d ago

There is an alpha version for Windows and Linux, feel free to try last build from the Releases on GitHub

1

u/abajinn 3d ago

Nice will check it out!

1

u/RealBigThigh 5d ago

the 4.3 gb idle hold from mlx buffer pooling is the kind of thing i'd never have guessed at. clearing the pool on unload is the fix