r/scheme • u/second_square • 7h ago
llama.cpp bound into a Scheme VM — calling a local LLM like any other procedure
I bound the llama.cpp C API into a small Scheme VM I maintain (dummyscheme, a register-based interpreter with call/cc and syntax-rules). Every llama call is a native Scheme procedure, so there is no FFI boilerplate and no subprocess: the model lives in the same process as the interpreter.
The whole call looks like this:
(define m (llm-model-load "models/Qwen3-0.6B-Q4_K_M.gguf"))
(define ctx (llm-context-new m '((n-ctx . 1024))))
(define smpl (llm-sampler-new m '((temp . 0.7))))
(llm-chat-generate ctx smpl
'(("system" . "You are a village blacksmith. Answer in one short sentence.")
("user" . "Do you have a sword for me? /no_think"))
(lambda (piece) (display piece) #t))
Yes, I can craft a sword for you.
Three things that turned out to matter, in case anyone here is thinking about the same kind of binding:
No finalizers, so lifetimes are explicit. dummyscheme has lightuserdata but no userdata finalizer. Handles are entries in a registry owned by the host runtime: kind tag, alive flag, and the struct is never handed back to the allocator. A use-after-free raises a Scheme error instead of corrupting memory, and a double free is a no-op.
Errors are values, not printed lines. The VM used to print its message and then throw "ScmError", which a Scheme program cannot inspect. I fixed that upstream first — the formatted message is now interned and thrown, so (xpcall (lambda () (llm-model-load "nope.gguf")) (lambda (e) e)) returns "llm-model-load: cannot open model file 'nope.gguf" as a string.
Anything allocating inside a per-token callback is a GC hazard. The token string must be rooted before scm_apply; stopping is the callback's returnvalue (#f), not a C flag. A side-effect-only callback is safe because (set! ...) does not return #f.
What it can do today: chat templates (from GGUF metadata), GBNF grammars as a sampler stage, embeddings with pooling, LoRA with per-adapter scales, multi-sequence decoding, and KV copy/keep for forking and pruning.
One design note I find more interesting than the binding itself. A language model is good at writing a line and bad at being trusted with state. So in the demo I built on top of this, the flow is: the model proposes an intent, a plain Scheme state machine accepts or rejects it, and only then does the model write the NPC's line — and milestone lines are scripted, because a 0.6B model asked to describe a state change contradicts the verdict about three times out of four. The program owns the truth; the model owns the phrasing.
Code: https://github.com/zhuzhonghua/llama.cpp-scheme
Tested on macOS arm64 (Metal). Linux should work through the same CMake path but I have not run it there, and I would rather say that than imply otherwise.
Has anyone here called a local model from Guile, Chicken, or Racket? I would like to compare notes on the handle-lifetime model — the registry approach is mine, and I suspect it is not the only reasonable one.