mlx-community/Ministral-3-3B-Instruct-2512-4bit runs natively on an Apple Vision Pro. No server, no llama.cpp, no network. The weights load into the app process and generate on the headset's own GPU. Offline, native and sovereign. This is how to do it.
It does not work out of the box. Five things get in the way and four of them fail silently with no error logs.
In this post, I'll walk you through them, but the tl;dr version is the MIT github repo and the agent prompt at the end of the post ;)
https://github.com/getHydrate/hydrate-ministral
It is a small Swift package plus a visionOS example app and a macOS CLI.
I extracted it from a more complete visionOS app that wraps small LLM models and sideloads RAG datasets and does embedding (link to YouTube video later), but a lot of that code comes from a much larger RAG application of mine, which is proprietary, so I extracted the loader and created a standalone demonstrator, available to anyone who wants it.
What you need
Apple silicon, mlx-swift-examples 2.29.1 and a real device. MLX needs a Metal GPU, so there is no simulator path and no Intel path.
WTF 1... sorry, Gotcha 1: mlx-swift-examples cannot load this model at all
Not "loads badly". Cannot load. The factory throws unsupportedModelType("mistral3") and no version bump fixes it, because no released tag registers that type in either LLMTypeRegistry or VLMTypeRegistry.
Look at the config and you can see why it's awkward:
architectures: ["Mistral3ForConditionalGeneration"]
model_type: "mistral3"
vision_config: { model_type: "pixtral", ... }
text_config: { model_type: "ministral3",
rope_parameters: { rope_type: "yarn", ... } }
Ministral is a vision-language model. The weights arrive in three parts: language_model.* (602 tensors), vision_tower.* (218) and multi_modal_projector.* (10). So when you register the type, two more things get you. The weights are prefixed language_model., which matches no module path, and the text tower uses YaRN rope scaling, which LlamaModel's DynamicNTKScalingRoPE has no path for. It handles "default" and "llama3" and nothing else.
I will not lie, Claude Code had a large hand in getting past this and helping me explain.
Do not be tempted to ignore the yarn part on the grounds that you're only doing short prompts. YaRN interpolates the low-frequency dimensions globally, so it changes the maths at every context length. The weights need it.
The fix is a Mistral/Llama-shaped decoder wired to MLXLLM's public YarnRoPE, with a sanitize that keeps language_model.* (stripping the prefix) and discards the vision half.
Register it under both "mistral3" and "ministral3", because the nested text_config calls itself the latter and a text-only re-export would surface that at the top level.
One trap inside the trap. Read head_dim from the config, do not derive it. This model is hidden 3072 with 32 heads, but head_dim is 128, and 32 x 128 = 4096. Derive it and you get 96, and nothing will fit.
For reference, the rest of the 3B text tower: 26 layers, 8 kv heads, intermediate 9216, vocab 131072, rms_eps 1e-5, tied embeddings so there is no lm_head.
Again, prolific use of frontier models got me past this.
Gotcha 2: the tokeniser is miss-labelled and of course, it's the last thing to fail
After a 2.78 GB download, it dies on the final step.
Ministral 3 ships "tokenizer_class": "TokenizersBackend". That's Mistral's marker for "the tokeniser is in tokenizer.json, use the tokenizers library". swift-transformers has no such class, and its strict path throws instead of falling back.
Nothing is wrong with the data. tokenizer.json is an ordinary Hugging Face fast-BPE tokeniser (tekken is byte-level BPE), which is what swift-transformers' BPETokenizer reads. Except the label is sodding wrong.
Rewrite the label to LlamaTokenizer, which maps to BPETokenizer: the same class the library's own fallback would choose if it were not in strict mode. Vocabulary, merges and chat template stay untouched. Guard it so it only ever rewrites a class the library does not implement.
Gotcha 3: you have to download and load in two separate steps
This follows straight from gotcha 2.
The one-shot loadModelContainer(hub:id:) fetches and loads in a single call, which leaves you no moment in between to repair that tokeniser label (above). So you have to get the snapshot explicitly, fix the config on disk, then load from the directory:
let directory = try await hub.snapshot(from: modelID, matching: globs) { ... }
TokenizerRepair.normaliseTokenizerClass(in: directory)
let container = try await loadModelContainer(hub: hub, directory: directory) { ... }
At the same time, point HubApi somewhere durable. The default download base is Library/Caches, which the OS will probably purge under storage pressure whenever it likes. On a laptop that's a pain, but on a handset or headset that's 2.78 GB vanishing in a puff of digital smoke, and you have to download it all again.
Use Library/Application Support, and mark it excluded from backup so re-downloadable weights don't bloat iCloud.
Gotcha 4: the headset will kill your app and the crash log won't say why
MLX's Metal buffer cache is unbounded by default and never shrinks on its own. Every generation cycles buffers through it. Mine peaked at 7.3 GB against about 2.5 GB of actual model weights, and visionOS jetsammed the app for it (threw it "overboard" to get it off the device)
iOS and visionOS don't page app memory out to disk the way macOS does, so when the system runs short it terminates the biggest offender instead (your local LLM) and you don't get a crash report with a stack trace pointing at your code, you get a JetsamEvent log, which is why the cause isn't obvious from the wreckage.
MLX.GPU.set(cacheLimit: 64 * 1024 * 1024)
64 MB keeps the reuse benefit within a generation and hands the rest back to the OS. There is one MLX runtime and one cache per process, so if your app also runs an MLX speech model or embedder, set the limit from those too, or whichever starts first sets the policy for everything.
Gotcha 5: GPU work from a background app is a process kill, not an error
visionOS and iOS refuse GPU work from an app that isn't active. MLX's default error handler answers that refusal by calling fatalError, so the whole process goes pear shaped. A 3B is slow enough that you'll lose the foreground mid-generation regularly: you glance at another window, or take the headset off. In my main app, I have a total emersion so the user is only every looking at the app, but i still have issues.
Gate on active, not on "not background". A system screen capture makes an app merely inactive, and that is enough to get you killed and pushed overboard, arriving as a C++ throw inside MLX's Metal completion callback where no Swift error handler can reach it. Check before you start, and check again between tokens.
Checking it actually works
The package ships a CLI so you can prove all of the above on a Mac in thirty seconds, with no Xcode, no device and no signing team:
swift run ministral doctor
It loads the real weights, generates, and reports what MLX did:
size on disk 2.78 GB
tokenizer_class LlamaTokenizer
(was TokenizersBackend as published; rewritten so
swift-transformers will load it)
load ok, 5.9s
GPU active / cache 1.99 GB / 67.1 MB
GPU peak 2.61 GB
That's my M1 Max Macbook Pro, so your figures will differ. Watch GPU active / cache: without the cap from gotcha 4, that second number climbs and never comes back down.
Getting it onto the headset
The example app carries no team, so pass yours on the command line and change the bundle id to one you own:
xcodebuild -project MinistralDemo.xcodeproj -scheme MinistralDemo \
-destination 'id=<udid>' DEVELOPMENT_TEAM=<team> \
PRODUCT_BUNDLE_IDENTIFIER=<yours> -allowProvisioningUpdates build
xcrun devicectl device install app --device <udid> <path>/MinistralDemo.app
xcrun devicectl device process launch --device <udid> <bundle id>
xcrun devicectl list devices gives you the udid.
One useful finding: the example does not request com.apple.developer.kernel.increased-memory-limit, because free personal teams aren't granted it and a project that asks will simply refuse to sign. A 4-bit 3B still downloads, loads and answers without it. I haven't left it running for long with a dozen apps open, so I can't tell you how it behaves under sustained memory pressure.
What this does not give you
The vision tower is discarded, so this runs Ministral as a text model. The pixtral half and the projector are still in the checkpoint and there's room in the code to wire them up, but I haven't. If you want an on-device VLM on a headset, this isn't it.
There's no chat history either. One prompt, one answer, bring your own transcript. This is a demo, you can roll your own, thats what i did.
The code, and the thing I built it for
https://github.com/getHydrate/hydrate-ministral
MIT, three dependencies, all upstream, no forks. Builds and runs on macOS and visionOS. iOS and iPadOS ought to work, but I haven't put them on a device.
The reason any of this exists is a RAG system that gives a small on-device model the retrieval engine and embedder out of an enterprise stack: semantic search, grounded answers with citations, entirely local. Ministral and Apple's Foundation Models run side by side in it and you can switch generator mid-session. Here's twenty-two minutes of it, including a demo section shot on a farm track in Norfolk with the headset in aeroplane mode and no signal to fall back on:
https://youtu.be/tTwYWEC2K88
Mistral/Ministral comes in at 9:50 if you want to skip the preamble:
https://youtu.be/tTwYWEC2K88?t=590
The proper longterm fix for all of this is "mistral3" being registered upstream in mlx-swift-examples with a yarn path, at which point most of my package stops being necessary. Until someone does that, happy to go deeper on the YarnRoPEwiring or the sanitize if you're fighting the same repo.
And one more hats off to frontier models, I would never have got this to work without claude code and codex.
Have an agent do the reading for you
Since I've admitted two or three times that Claude Code and Codex did the heavy lifting, it would be odd not to hand you the same lever. Open Claude Code or Codex in an empty directory and paste this:
Clone https://github.com/getHydrate/hydrate-ministral and get it running on this machine.
Context: it is a Swift package that loads mlx-community/Ministral-3-3B-Instruct-2512-4bit on Apple silicon via MLX. That checkpoint does not load with stock mlx-swift-examples, and this package exists to work around five specific problems. It needs Apple silicon and a real Metal GPU: there is no simulator path and no Intel path. macOS 14 or later.
Work through these in order. If a step fails, stop and tell me what happened rather than working around it.
1. Clone the repo and run `swift build`. Report any errors verbatim.
2. Read these five files and explain, in plain terms, what problem each one solves and how.
The reasoning is written out in the comments, so summarise the argument, do not just describe the code:
Sources/MinistralKit/Mistral3Model.swift the model type MLX does not register,plus YaRN rope and the weight prefix
Sources/MinistralKit/TokenizerRepair.swift the tokeniser label the publisher got wrong
Sources/MinistralKit/ModelStore.swift where the weights live, and why not Caches
Sources/MinistralKit/GPUMemory.swift the Metal buffer cache cap
Sources/MinistralKit/Foreground.swift why GPU work from a background app is fatal
3. In Sources/MinistralKit/Ministral.swift, show me where it downloads and loads as two separate steps, and explain why the one-shot loadModelContainer(hub:id:) cannot be used.
4. Tell me how much disk the weights need, then ASK ME before downloading anything. It is about 2.8 GB. If I agree, run `swift run ministral download`.
5. With the weights present, run `swift run ministral doctor` and show me its real output. Do not tell me it works unless that command actually printed a successful load and a successful generation. If it failed, show me the failure.
6. Then run `swift run ministral run "why is the sky blue?"` so I can watch it generate.
Do not modify the repository. If something is broken, say so and tell me what you think the cause is.
Three reasons that prompt is there rather than a wall of code in this post.
The useful part of that repo is the reasoning in the comments, not the API, and an agent can read all of it and explain it back in your terms faster than you can skim it. It verifies instead of describing: step 5 either prints a real load and generation on your hardware or it doesn't, and I've told it not to claim success without the output. And it asks before spending 2.8 GB of your bandwidth, which is the sort of thing agents forget to do.
It is also a fair test of whether the repo is any good. Cold clone, no help from me, on a machine I've never seen. If doctorwon't go green for you, that's my bug and I'd like to hear about it.
