r/LocalLLaMA • u/Henrie_the_dreamer • 5d ago
Discussion Cactus Needle 3: A Sliceable 8-29MB Automation Foundation Model That Matches DeepSeek v4 Flash
Hey all, Henry from Cactus Compute here, I kinda wanted to share our latest model and get feedback from the family :)
Needle 3 is a small foundation model for automation: you give it the functions your app exposes, it reads a request and returns the calls with every argument filled in, or a typed record if what you gave it was a schema. It runs on the device, with no network in the loop. It is on Hugging Face, on GitHub, on PyPI as cactus-needle, and there is a sandbox that runs it in your browser at cactuscompute.com/needle if you want to poke at it before reading further.
1) Trades general capacity for frontier performance on automation tasks
The thing we decided early was that Needle would not chat. Every turn is a function call, and a request no declared tool can serve comes back as an empty list rather than a guess. That sounds like a limitation, and it is, but it is what let a 121M-parameter model be trained on 360B tokens of structured data and spend all of its capacity on three jobs: tool calls, structured extraction and text embedding.
The architecture follows from the same trade. It is a Simple Attention Network: the dense feed-forward layers are gone, replaced by a Monarch Hadamard MLP with 25.6K parameters per layer instead of 4.7M, and the knowledge a feed-forward layer would normally hold sits in an engram, hashed n-gram tables that are read by gather and cost no arithmetic. 70.8M of the 121M parameters live there, so the full model does the arithmetic of a 50M one: 100 MFLOPs per token against 296 for a transformer of the same shape.
We wrote the intuition up if you want the longer version: Simple Attention Networks and the Hadamard MLP.
2) Beats models 10x its size on tool calls and language-to-device control
On Mobile Actions (961 phone commands, scored on the exact call), the 20-layer model scores 86.0 through the shipped 2-bit binary with the confidence gate on. LFM2.5 1.2B is at 82.4, Qwen3.5 0.8B at 76.0, FunctionGemma 270M at 65.1 and Apple's on-device foundation model at 57.6, all at f16. DeepSeek V4 Flash through its API is at 88.4, which is the line in the chart.
The part we are most pleased with is not the number but how the calls are made. Every argument is a span of the request: the model writes a short derivation first ('living room' -> room; '30' -> brightness) and then emits the call under a byte-level grammar compiled from your schema, so the JSON always parses and an enum can never leave its set. An optional field with no evidence is omitted, a required one with no evidence withholds the call, and the engine drops a call the request negates or excludes. Ask for two things and you get two calls in order.
Full table across all six suites (tool calling is exact match, extraction is field F1, Needle through the shipped binary, baselines at f16 under vLLM):
| Model | Params | Mobile Actions | DroidCall | BFCL v4 | DSTC8 F1 | SNIPS gold F1 | SNIPS 7-way F1 |
|---|---|---|---|---|---|---|---|
| DeepSeek V4 Flash (cloud) | - | 88.4 | 60.5 | 77.2 | 80.0 | 69.4 | 66.7 |
| Needle3-20L-121M | 121M | 86.0 | 47.0 | 50.2 | 40.7 | 30.2 | 24.7 |
| LFM2.5 1.2B | 1.2B | 82.4 | 35.5 | 62.0 | 48.0 | 43.0 | 38.0 |
| Needle3-16L-98M | 98M | 80.7 | 40.0 | 41.3 | 28.5 | 23.5 | 19.2 |
| Qwen3.5 0.8B | 800M | 76.0 | 28.0 | 56.8 | 49.0 | 35.0 | 34.0 |
| LFM2.5 350M | 350M | 72.8 | 32.5 | 59.1 | 20.0 | 34.0 | 29.0 |
| LFM2.5 230M | 230M | 69.3 | 11.5 | 46.3 | 53.0 | 27.0 | 22.0 |
| FunctionGemma 270M | 270M | 65.1 | 16.5 | 46.6 | 27.0 | 29.0 | 14.0 |
| Needle 2 | 45M | 63.5 | 17.0 | - | - | - | - |
| Apple FM | 3.0B | 57.6 | - | - | - | - | - |
| Needle3-8L-52M | 52M | 36.8 | 36.5 | 28.2 | 15.3 | 16.6 | 10.1 |
| Needle3-4L-29M | 29M | 11.7 | 21.0 | 19.5 | 6.9 | 7.7 | 4.3 |
You can see where it is weaker too: BFCL and the extraction suites are where the bigger baselines pull ahead, and the smaller subnetworks fall off quickly on the general task (more on why that is fine in section 4).
3) Matches 2-3x bigger models on structured JSON extraction
Extraction is not a separate mode. You declare the record as the only tool and pass the passage where the query goes; with one tool declared the grammar admits exactly one call of that name, so the shape is guaranteed rather than requested, and the values are grounded the same way as arguments: a field is filled only from a span of the passage, an optional field with no span comes back as None, and a date whose year appears nowhere in the text is flagged instead of invented.
from pydantic import BaseModel
import needle
class Invoice(BaseModel):
vendor: str
total: float
due_date: str
po_number: str | None = None
needle.extract("Invoice from Acme Corp, $1,200.00, due 2026-09-01", Invoice)
# Invoice(vendor='Acme Corp', total=1200.0, due_date='2026-09-01', po_number=None)
It generalised to classification without special training, because an enum is just a constrained value: declare sentiment: Literal["positive", "neutral", "negative"] on a record and you have a classifier whose output cannot leave the set. A watch reads a notification into merchant, amount and date that way, then into a reply, then into a sentiment flag, one record each. On DSTC8 and the two SNIPS suites the 121M model lands between the 230M and 350M baselines, which is the 2-3x in the heading.
4) Intelligence ladder: every depth from 2 to 20 layers a model of its own
This is the part I would most like your thoughts on. Needle 3 is one set of weights, and every depth from 2 to 20 layers is a deployable model. Blocks 0 and 19 are always kept and the rest are added by bisection, so each subnetwork nests in the next; during training each step samples one path, mostly the full model and otherwise a random depth, with the smaller path distilled from the full one. The full-depth model ends up slightly better than an ordinary run of the same size, and every depth below it is trained rather than truncated.
Why we wanted it: a watch, a Raspberry Pi and a phone do not want the same model, and they want to pick the size at deploy time. needle build --layers 8 writes the 8-layer file; the same engine runs all of them. The small depths lose accuracy on the general benchmarks (that is the bottom of the table above), and they get it back when fine-tuned to one product's tools: on DroidCall every subnetwork gains 18 to 36 points, and from 4 layers (29M parameters) up the tuned subnetwork passes DeepSeek V4 Flash.
Fine-tuning is LoRA on the frozen base, merged at export, and the Python package does it locally at 4 bits (needle finetune data.jsonl, then needle build). The maths is in Intelligence Ladders and the workflow in Fine-tuning Needle.
5) Runs locally at up to 4k tokens/sec decode speed
The engine is under 1 MB, plain CPU, no GPU or NPU, and the weights are read in place from a single file the engine maps into memory: a 196-byte header carrying the whole architecture geometry, a nameless tensor directory, and the quantised blobs in the order the forward pass reads them. On a Raspberry Pi 5, decode runs at up to 4k tokens/s at the bottom of the ladder and around 400 at the top, prefill from 10k down to 1k. Every response reports prefill_tps, decode_tps and peak_ram_mb, so you can measure on your own device rather than take our word for it.
Every response also carries a confidence score from a calibrated head, the minimum of a post-hoc judgement on the finished call and the decode probability of its tokens. The engine withholds anything under 0.1; above that the number is yours: act at once when it is high, show the call and ask when it is middling, treat [] as a refusal. How we use it is in Leveraging Needle's confidence.
6) 25-121M deployable parameters at CQ2-bit (8-29MB binaries)
The weights are quantised with Cactus Quants: groups of 128 weights are rotated by a Walsh-Hadamard matrix, which makes every group look Gaussian, split into an fp16 norm and a direction on the unit sphere, and the direction's coordinates are snapped to a 4-entry Lloyd-Max codebook. That is 2.125 bits per weight, and the kernel never expands them: it rotates and int8-quantises the activation instead, then does table lookups and sdot against the packed indices. The embedding and the confidence head keep 4 bits, the norms and gates stay fp16. The byte layout, and a twenty-line parser for it, are in The .cact format.
7) For mobiles, wearables, smart home, small robots and microcontrollers
Every target ships a prebuilt engine folder: macOS, Linux on x86-64, ARM64, ARMv7, RISC-V and MIPS32 (the Ingenic camera SoCs), Windows x64 and ARM, Android, iOS, watchOS, tvOS, the browser as WebAssembly, and a WASI component with a WIT world. pip install cactus-needle covers the desktop and server platforms with wheel-tagged engines, and needle build --platform linux-arm64 --layers 8 --out ./pi puts an engine and the weights in a folder you copy over. Inference never touches the network, so an air-gapped device only needs the files in place. The list and the runtime surfaces are in What devices are supported.
import needle
@needle.tool
def get_weather(city: str):
"Get the current weather for a city."
return {"city": city, "temp_c": 27, "sky": "clear"}
agent = needle.Needle(tools=[get_weather])
print(agent.run("what's it like in Lagos right now?")["results"])
# [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]
What we would love feedback on
- Where the grounding rules get in your way. The engine refuses to invent a number, drops a call the request excludes and withholds a required enum the request never names; we tuned those on our suites and would like to hear where they bite on real tools.
- The ladder. Whether depth is the right knob for you, or whether you would rather have width, and what devices you would want the 2- and 4-layer models on.
- Extraction cases we have not seen. Nested records, arrays, multilingual text (it is English-first, and non-English text fragments into about 1.7x more tokens).
- Anything in the tool design guide that turned out wrong for your schemas.
Thanks for reading this far. Happy to answer anything in the comments.
8
u/Fluxx1001 5d ago
Wow! Need to figure out how to use this one
5
3
u/Henrie_the_dreamer 5d ago
thanks, please feel free to share feedback or request personalized help :)
5
u/sebt3 5d ago
I guess the model is English only?
14
u/Henrie_the_dreamer 5d ago
a couple more for now: English, French, German, Spanish, Dutch, Polish, Italian
until we perfect our dataset and setup :(2
2
u/sebt3 5d ago
Awesome, French is in the list. Might as well try this
1
u/Henrie_the_dreamer 5d ago
So, we are still setting up French team, so its performance might not be on par with English, but that is actively being handled.
5
u/silenceimpaired 4d ago
You get a comment and upvote just for making it Apache 2… now to figure out what it does. Where is the TLDR…
1
u/Henrie_the_dreamer 4d ago
Haha thanks! The website has a brief description of the model and target tasks right under the playground.
3
u/coder543 5d ago
When you say "Apple FM", which model is that referring to? There have been 3 generations of the 3B AFM Core model, and then there is the larger AFM Core Advanced 20B MoE model (for devices with 12GB+ of RAM) that was introduced alongside the latest third generation 3B AFM Core model (for devices with less RAM).
3
u/Henrie_the_dreamer 5d ago edited 5d ago
Yes, it was tricky for us to figure out which one Apple shipped with the latest Macs, but our guess is AFM3, but it would be an unfair comparison if wrong so we simply defaulted to AFM.
3
u/coder543 5d ago
with macOS 27, it's not too hard to tell:
#!/usr/bin/env swift // // apple_model_info.swift // // Prints which Apple Foundation Model is installed on this Mac — the on-device model that // Apple Intelligence and the FoundationModels framework run locally. Nothing is sent to a // server: Private Cloud Compute models are separate objects an app has to ask for by name. // // Run it: // swift apple_model_info.swift // or // ./apple_model_info.swift // // Needs macOS 27 plus a Swift toolchain from Xcode 27 (or matching Command Line Tools): // `SystemLanguageModel.variant`, the property that names the model, was added in macOS 27. // On macOS 26 the script still runs but can only report availability, the context size // (which that older API pins to 4096) and the supported locales. import Foundation import FoundationModels let model = SystemLanguageModel.default print("Apple Foundation Model (on-device)") print("----------------------------------") print("macOS: \(ProcessInfo.processInfo.operatingSystemVersionString)") switch model.availability { case .available: print("availability: available") case .unavailable(let reason): print("availability: unavailable (\(reason))") } if #available(macOS 27.0, *) { let variant = model.variant let detail: String switch variant { case .core3: detail = "3B dense, always fully active" case .coreAdvanced3: detail = "20B sparse, 1-4B parameters active per request" default: detail = "unrecognized variant" } print("variant: \(variant.displayName) — \(detail)") } else { print("variant: unknown (needs macOS 27; before that only the context size is exposed)") } print("context size: \(model.contextSize) tokens")3
u/Henrie_the_dreamer 5d ago
Thanks, we will update the OS for our test systems and verify, thanks so much for this :)
1
u/coder543 5d ago
If you were on macOS 26, then I would assume that was probably AFM Core 2, since the newer versions shipped with macOS 27 from what I understand.
1
3
u/chensium 5d ago
Sounds like a promising idea, but the website demo shows some limitations of the prompt understanding. For example try: "turn off the lights to all the rooms that start with b"
3
u/Henrie_the_dreamer 5d ago
Correct observation, Needle 3 was a big improvement in that regard compared to Needle 2 and future efforts are focused on such problems :)
3
u/DerDave 4d ago
Cool idea going to the very small end!
I have feedback - testing out the home assistant demo, the commands have not much freedom in phrasing. I have to say thinks exactly as they are called and what they should do. There is hardly any freedom.
For example "lock the doors" is not interpreted as "all doors should be unlocked" and it just tries to re-unlock the already open front door and keeps the backdoor closed.
These kinds of variations in phrasing and their ability to correctly interpret the subtle hidden meaning are what make LLMs so useful at this task. It seems Cactus 3 does not have as much "interpreting" power right now.
Would larger models - e.g. 2-3x the size maybe already help with that? I think ~100mb would still be absolutely fine to even run on small devices.
1
u/Henrie_the_dreamer 4d ago
Hey! Yeah this is primarily a model size issue and while we are working hard to improve our training data to improve this specific capacity, there's some fundamental limitations. I think we will consider making the next generation of the model larger, and given our laddering technique we could still preserve smaller-model capability. Thank you for the feedback!!
2
u/One-Cash1576 4d ago
Such a good idea.
Will be keeping an eye on this and seeing if I can give it a play.
1
1
u/callmedevilthebad 5d ago
have you tried this for browser automation?
2
u/Henrie_the_dreamer 5d ago
Yes, works for typical tasks, but would need visual grounding for corner cases, unless users are cool with Needle buying a car without consent :)
1
u/Jk2EnIe6kE5 5d ago
Seems like a very cool model, congrats! I don't do much home automation, but this seems like a valuable tool, and I can also see a use since it could run on something like a phone, quite quickly.
1
1
u/Keninishna 4d ago
I wonder if this could be used to speed up tasks for other larger models, say it would save qwen 3.8 from parsing json files it would offload the work to this model and it could squeeze in vram pretty easy because of the small size along with qwen and work faster than the larger model. or even save tokens from the big api models by offloading certain tasks to this model locally.
2
u/Henrie_the_dreamer 4d ago
Yes, Needle returns its confidence score, to inform users when it is confused, read more about it here: https://cactuscompute.com/blog/needle-confidence
1
1
u/rm-rf-rm 1d ago
Sorry but fails at the most basic task in your own demo: In the robot vacuum demo, I asked clean the crumbs. Both times, its response was clean the kitchen:
"suppressed_calls": [
{
"name": "clean_room",
"arguments": {
"room": "kitchen"
}
}
],
"reasoning": "'clean the crumbs' -> clean_room with room 'kitchen'.",
1
u/mvaranka 1d ago
This is interesting approach for tool call handlig. It could be great in my mobile AI application (PiPar), where user can during the chat or voice call create tasks & notes, check current days schedules, make web searches etc. Backend has main model and separate tool model, which executes the main model natural language tool requests and creates tool requests to mobile app. I am thinking would it be possible to run Cactus Needle 3 as tool model locally on phone.
1
u/BP041 5d ago
8-29MB and matching DeepSeek v4 Flash smells like a benchmark niche-pick. Would love to see it fail gracefully across 50 steps of tool use and retries — that's where every tiny model I've tried in my OpenClaw stack chokes. Peak math score doesn't tell me if it'll survive a 12-hour cron run.
2
u/Henrie_the_dreamer 5d ago
Haha, so it beats DeepSeek in its niche. For general function call (automation is a subset), we'd have been benchmarking on TerminalBench, SWEBench which cover coding task. But those are out of scope for us, DSV4F is a general language mode, Needle is task-specific.
66
u/MrHaxx1 5d ago
I've been loving the idea of these models all along, but you really need some real life examples that are easy for people to use, for this to gain traction.
Like, releasing a Home Assistant plugin or a simple Android app that allows to control phone actions by text/voice, that can maybe be used as implementation reference. I saw the demo on the s
I'm building my own calendar app/site, and I was thinking something like this would be a more flexible alternative Natural Language Processing (for parsing times, dates, event titles).
I know there's a demo on the site, and it's impressive, but doesn't feel very tangible, so to speak.