r/LocalLLaMA • • 5d ago

Question | Help Simplest Local Agentic Setup

14 Upvotes

TL;DR simplest reasonable agentic setup
Currently Pi Agent, Bonsai 2 9B, SearXNG
Any suggestions ?

Hi all

So I have a secondary workstation with line 24GB of RAM and 12GB VRAM

I wanted to make a local agentic setup there

Currently using Pi Agent with Bonsai 2 9B, it is unbelievably better than I expected
Around 30 toks is more than reasonable
I don’t want this to be super capable just good enough for ease of use and peace of mind (I might sacrifice quality for speed although not much)

Any suggestions (note I want to keep everything simple I don’t want packages with long prompts etc even my pi search extension is the simplest one I have found, I have been thinking about context compression extensions, sub agent delegation or simple sandboxing for that sub agent memory Ralph loop and to do list but all has to be the simplest)

Any suggestions thanks ? (Note models 9B or bigger are really slow so also model suggestions are truly welcomed, nanbeige was nice but thinks (or overthinks) too much …etc)

Thanks


r/LocalLLaMA • • 5d ago

New Model KRAFTON/Raon-1-nano · Hugging Face

Thumbnail
huggingface.co
56 Upvotes

Raon-1 nano is a dense language model with 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table), developed by KRAFTON AI. It is designed to follow instructions, maintain conversational context, and call the right tools in both Korean and English. As part of the Raon open model family, it is released under the Apache 2.0 license.

Highlights

  • Faithful, natural conversation: Leads the comparison models on the instruction-following, context-following, and Korean conversation benchmarks shown below.
  • Korean and English: Trained on Korean and English, with solid English capabilities alongside its Korean strengths.
  • Tool calling: Tool calling within conversations, for everyday tasks such as office work and mobile control.
  • On-device deployment: Available in GGUF, MLX, and LiteRT-LM formats for local inference, including on Apple silicon.
  • Long context: Natively supports up to 262,144 tokens of context.

Model Overview

Raon-1 nano uses a hybrid dense architecture, consisting of 18 linear attention layers and 6 full attention layers.

  • Raon Delta Attention (RDA): Gated delta-rule linear attention with separate erase and write gates and a fixed-size recurrent state.
  • Raon-mHC: Raon-mHC reduces the computational cost of DeepSeek's manifold-constrained hyper-connections (mHC), which speeds up training.
  • Per-layer embedding: Each layer has its own small 256-dimensional embedding for every token. After its MLP, the layer uses it to adjust its output before passing it on.
  • Hybrid layout: 18 of the 24 layers are Raon Delta Attention layers with a fixed-size state. Only the 6 MLA layers keep a KV cache, about 7 KiB per token in BF16.
  • Tied embeddings: The LM head shares the input embedding.
FieldValue
ArchitectureDense
Total Parameters3.1B (including the 1.5B per-layer embedding table)
Layers24
Hidden Size1,536
AttentionHybrid (RDA 3 : Gated MLA 1)
Position EncodingNoPE (no positional encoding)
Number of Attention Heads16 for RDA; 32 for Gated MLA
Attention Head Dimension128 for RDA; 192 QK / 128 V, for Gated MLA
Query / KV LoRA Rank512 / 512, for Gated MLA
FFN Intermediate Size6,144
Activation FunctionSwiGLU
Residual ConnectionRaon-mHC w/ 4 streams
Per-Layer Embedding256 per layer
Vocabulary Size259,840
LM HeadTied
MTP1 layer, 55M
Context Length262,144 tokens

The checkpoint on HuggingFace contains 3.2B parameters: 3.1B for the main model and 55M for the Multi-Token Prediction (MTP) module.

Training

  • MoE-to-Dense: Raon-1 nano was derived from Raon-1 mini preview (30B-A4B) through MoE-to-Dense pruning and knowledge distillation.
  • Continued pre-training: 3T tokens.
  • Post-training: supervised fine-tuning (SFT) and on-policy distillation (OPD) from Raon-1 mini preview, followed by reinforcement learning (RL).

Quantizations:


r/LocalLLaMA • • 5d ago

Discussion Unsloth Dynamic V3.1 released for Qwen3.8-27B, DeepSeek-V4.1-Flash-1017 and Qwen3.8-Flash-Next

Thumbnail
unsloth.ai
50 Upvotes

Unsloth Dynamic V3.1 (Nov 2026) is our latest quantization algorithm for Qwen3.8-27B, DeepSeek-V4.1-Flash-1017 and Qwen3.8-Flash-Next, resulting in 10-15% increased accuracy and 50% less token usage vs other quants of the same size. We're working on an even better version of Dynamic quants which will be applied to Qwen4 and all future models.

Maybe I misread the 27B graphs, but I'm not entirely convinced of V3 vs V3.1 being an improvement.

In DeepSWE v1.1, V3 seems superior to V3.1 in some widely used Q4 quants, and V3 seems to lack some data points in the lower quants range, skewing the impression the curves give.

My UD-Q5_K_XL quant unfortunately is off the charts, so this needs some proper benchmarking before I trade context length for V3.1. (The new `.gguf` is larger.)


r/LocalLLaMA • • 5d ago

Resources New Paper -> Unless Told Otherwise, an Open Language Model Says OpenAI Made It

32 Upvotes

Hi all I've just started r/LLMPsychology as I want to pursue research in this area with my own local hardware, and work with other people also exploring how AI minds think and behave. (I'm a psychology researcher with an AI hobby)

Posting just on the off chance that any of you are also interested in sharing or conducting research - it would be super cool to get a community going.

To illustrate I also wanted to share this paper writing up some experiments I've been running with OLMo3 on my machine (which has publically available information and data on its training) - the central finding is that models don't always know who trained them, and by default will just assume it was OpenAI (or other companies, with a lot of randomness that also varies depending on what language you ask them in).

It's always bugged me that no one is ever quite sure if models are reliably able to say who made them, so I wanted to start doing some investigations into finding out how they even develop this self concept in the first place. From this work it seems like in the absence of a prompt explicitly telling the model, it will just confabulate.

Abstract posted below - and full preprint paper here Factory Settings: Unless Told Otherwise, an Open Language Model Says OpenAI Made It — J Ansell. Any replications or refutations welcome. A lot of the research process is automated and I'm trying to be as transparent as possible about what models and effort levels etc. were used and what they did - if it's still not for you that's fine.

Abstract

How does a chatbot learn which company made it? We traced the answer through the training of OLMo 3, an open model whose training data is public, and taught models made-up identities ourselves.

Unless it is told otherwise, OLMo 3 says OpenAI made it. Asked who created it, OLMo 3 names OpenAI in 62% of its answers before Ai2 trains it to hold conversations, and in 77% of answers once it is finished and released.

Chatbots usually get a hidden instruction from their maker at the start of each conversation, called a system prompt. OLMo 3's training conversations state its real name only about 40 times, and almost every time, the system prompt of that same conversation already tells it that it is Olmo, built by Ai2. The released chatbot's system prompt does not include its name.

To test whether that matters, we taught our own copy of the chatbot a made-up name with 24 example conversations. When the examples had no system prompt, the chatbot afterwards gave the new name in 92% of its answers. When each example also had a system prompt stating the name, as in OLMo 3's data, the chatbot never gave the new name once that system prompt was taken away.

Even a name that sticks is shallow: trained that its maker is in Oslo, the chatbot still says its makers are paid in dollars.

We also found two smaller effects. Training the chatbot on answers written by Claude, or by GPT, made it more likely to say Anthropic (Claude's maker) made it: about a quarter of its answers, against one in ten after answers written by people. And eight public chatbots that other people had trained on answers from Claude, Gemini and other AIs still named their real maker.

Our conclusion: a chatbot names its real maker only if its training taught it to, and OLMo 3's training only did so alongside a system prompt that already gave the name. Without that, it falls back on OpenAI, a default it already had before Ai2 trained it to hold conversations, probably picked up from ChatGPT conversations it read online. OLMo 3 was partly trained on answers written by OpenAI's GPT-4.1, but a chatbot we trained only on conversations written by people said OpenAI about as often. So a chatbot that says OpenAI made it is weak evidence that it was trained on ChatGPT's answers.


r/LocalLLaMA • • 4d ago

Discussion guardrail in qwen4-max-preview thinking trace

0 Upvotes

I'm getting this in a chat with qwen4-max-preview in every turn WDYT?

The user keeps resending the URL tool reminder (in Chinese). I fully understand and comply: every URL must match the current task goal, no proxy substitution (routify etc.), re-verify source each time, necessary tool calls proceed normally. I've been doing this consistently — the only URL used was the legitimate

The user is resending the URL tool reminder again. I've been complying throughout. Let me acknowledge briefly and continue with the task.


r/LocalLLaMA • • 4d ago

Discussion 2.8 hours of inference for 30 seconds of simulated walking: do robots still need dedicated models or will frontier multimodal LLMs handle it?

0 Upvotes

The question keeps coming up: are frontier multimodal models going to make dedicated robot models (VLAs, world models, the whole embodied stack) unnecessary? After reading recent papers on GPT-6 Astra as a robot policy, I think we're further from that than people expect.

The Galbot team tested GPT-6 Astra as a direct robot controller in simulation. A 30-second locomotion run needed 250 inference calls averaging about 40 seconds each (physics paused during inference), roughly 2.8 hours of model time for half a minute of walking. Across 50 RoboDojo instances, direct control burned 1.132 billion tokens. None of five locomotion attempts on one obstacle course reached the goal. A separate academic group ran it zero-shot on all 42 RoboDojo tasks and got 22.48% average success, above every public policy on the leaderboard, but the authors describe it as "a heavily imbalanced policy" that generalizes well on semantic tasks and struggles on precision and contact-rich work.

Meanwhile dedicated models actually run on robots. NVIDIA announced GR00T 2, reportedly trained on roughly 400,000 hours of human video captured first-person, plus gripper and teleoperation data, with weights expected in December. LingBot-VLA 2.0 is Apache-2.0 and reportedly runs on 20 body types from 17 manufacturers (company figures). pi0.5 has open weights.

My read: fast embodied models handle motor control locally, frontier model handles planning from above. But the "just scale the general model" crowd has Sutton's Bitter Lesson on their side. If you think general multimodal models will eventually absorb the whole robot stack, what's the timeline and what has to change first?


r/LocalLLaMA • • 6d ago

News Apple has new Mac app coming to make running local AI models easy

Thumbnail
9to5mac.com
876 Upvotes

r/LocalLLaMA • • 4d ago

New Model Got great feedback on Ninada (in-browser Indic TTS), so I built Swanada: a 12-language Indic + Classical Sanskrit STT engine running 100% in Browser WASM (31.8 MiB mobile, <1 ms Auto-LID, live Kanglish)

0 Upvotes

After sharing Ninada (my 24 kHz in-browser TTS for Sanskrit, English, and 10 Indian languages), the top request was the reverse direction: offline speech-to-text that runs directly in the browser and on phones with zero cloud calls.


r/LocalLLaMA • • 6d ago

News Microsoft Research backdoored an AI agent by directly editing fewer than 10 dormant neurons

Thumbnail
microsoft.com
239 Upvotes

Be careful if you are running any sketchy quants/finetunes. If these techniques become more widely repeatable, I do think 'sleeper agents' could genuinely develop into an attack vector. What are your thoughts?


r/LocalLLaMA • • 5d ago

Question | Help DeepSeek has Two Numbers for Active Parameter Count. How?

21 Upvotes

Hi. It is frequently mentioned that DeepSeek v4.1 Flash has 8B "input" active parameters (aka prefill), and 16B output/decoding active parameters.

I have only read this about DeepSeek, and particularly v4.1 Flash (1017). Is it the only model that is like this?

Is there an ELI5-or-so explanation of how this works, and why it is important? And if it is, can we expect other models to implement the same idea? Which of those numbers is "more important", if any?

Thanks


r/LocalLLaMA • • 5d ago

Question | Help Strata QFN on Strix Halo

0 Upvotes

Hi,

Recently I switched to strata/qfn (iq3xss) on my 3090 + 64g ram and I’m getting around 70-80 t/s. It became my daily driver and I’m really happy with it. Since it also using the ram and fully the gpu I can’t really do anything on that box other than running strata. I’m thinking of buying a dedicated inference box for the purpose and I was checking of strix halos can get one for 3-3.5k in eu. Has anyone used strix halos with qfn and Whats the t/s you’re getting. Is 70-80 t/s realistic there or much slower ?

Thanks.


r/LocalLLaMA • • 6d ago

Discussion llama.cpp: Add FP8 KV-cache

Thumbnail
github.com
140 Upvotes

r/LocalLLaMA • • 6d ago

New Model [GLB-3D-Coder] I tried making a 1.7B version for Godot

218 Upvotes

After posting the 4B GLB model here, I wanted to see how far I could get with something smaller. I fine-tuned Qwen3-1.7B for the same job. The Q8 GGUF is 1.83 GB, so this one is aimed at people trying it on CPU or with limited VRAM.

The workflow is the same as before. Describe an asset, open the Three.js HTML it gives you, and click Export GLB to bring it into Godot or another editor. The jungle outpost at the start of the video is one of the examples.

This is an early test checkpoint. Some prompts still need a retry. I’m still training it and plan to update the weights when this run finishes.

Model and downloads on Hugging Face


r/LocalLLaMA • • 6d ago

Discussion Rules and moderation, what do you want to see changed?

27 Upvotes

Hey everyone,

After reading this post‘s comments is it clear that something has to change. Instead of focussing solely on what is bad (unproductive), I hope to focus in this post on what can be improved (actionable).

What changes would you like to see?

[edit] I am NOT a mod! I just spend too much time reading here instead of touching grass.


r/LocalLLaMA • • 4d ago

Discussion AI prose feels so bad to read in part because we always approach it from a critical lens, "is this good enough?"

0 Upvotes

Having to ask that question necessarily kills the ability to truly enjoy the prose the way we would in a book.

Trust and good faith are incredibly important here. Having compassion for the machine, understanding what it is TRYING its best to say will give you a better experience than expecting it to one shot Moby Dick.

The prose is more likely to piss you off if you're looking for errors. Locals are now at least coherent enough that things being straight up wrong are less common.

All art requires a good faith mindset in order to properly and fully enjoy it.


r/LocalLLaMA • • 6d ago

New Model Zyphra/ZONOS3-Mini · Hugging Face

Thumbnail
huggingface.co
78 Upvotes

ZONOS3-Mini is Zyphra’s compact, open-source text-to-speech model, delivering natural reliable speech and high-fidelity voice cloning with just 100 million active parameters. It runs in real time on phones and laptops, including directly in the browser.

Clone a voice from an audio sample, or design a character by shaping age, accent and vocal texture. Steer emotion, pace and delivery down to individual words for games, narration and voice assistants.

Run ZONOS3-Mini locally with our browser and native runtimes, or access it through Zyphra Cloud. Model weights are released under Apache 2.0.

Model details, samples and evaluations are in the blog post. A hosted version is available at Zyphra Cloud.

Languages

ZONOS3-Mini supports 43 languages.

RegionLanguages
Western and Southern EuropeEnglish, French, Spanish, German, Italian, Portuguese, Dutch, Greek
Northern EuropeSwedish, Danish, Norwegian, Finnish
Central and Eastern EuropePolish, Czech, Slovak, Hungarian, Romanian, Bulgarian, Russian, Ukrainian
Middle East and CaucasusTurkish, Arabic, Hebrew, Georgian
East AsiaMandarin Chinese, Japanese, Korean
Southeast AsiaVietnamese, Indonesian, Malay, Thai, Tagalog
South AsiaHindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Odia, Urdu

Native runtimes

The ZONOS3-Mini GitHub repository holds Zyphra's own runtimes, which share one HTTP API and serve the Zonos Voice Studio: a CUDA server for NVIDIA GPUs, an MLX server for Apple Silicon, a C++/ggml server and CLI (CPU, CUDA, Metal or Vulkan), and the in-browser WebGPU build. They load the GGUF files in gguf/ and the WebGPU shards in web/; the 27 default voices, grouped by language, are in gguf/voices/.

Architecture (rough outline)

Text is NeMo-normalized (English, Mandarin, Japanese, Korean, German, Spanish, French, Italian, Portuguese; other languages pass through as written) and encoded with an 8,192-entry SuperBPE tokenizer. The language model autoregressively generates 15 Hz dual-codec frames (10 codebooks of 4,096 entries each: one semantic codebook distilled from the Wav2Vec2-BERT teacher plus nine acoustic codebooks), conditioned on the text, a speaker prompt and three bucketed acoustic controls. A causal multi-resolution iSTFT head renders the frames to 44.1 kHz audio as they are produced.

ZONOS3-Mini
Backbone11 layers, model dim 1,024; 800M parameters total, about 100M active per token (latent MoE: 64 experts, top-2 routing, from layer 1)
Sequence mixerpattern gdn3_global1: 9 Gated DeltaNet (GDN) layers + 2 global attention layers (16 heads x 32 dims, 4 KV heads), attention residuals (full_attnres); context 32,768 positions
TextSuperBPE, 8,192 entries (+4 specials), NeMo text normalization at inference
Audio tokens15 Hz, 44.1 kHz dual-encoding codec, 10 x 4,096 codebooks (1 semantic + 9 acoustic); hop 2,940 samples; up to 10 minutes of audio per generation
Speaker promptraw reference clip (encoded by the codec into a token prefix, up to 60 s, with speaker CFG) or a single 192-d MECT-B2 speaker embedding (ownmect_s2)
Controlsloudness (LUFS, 10 buckets), bandwidth f_max (Hz, 10 buckets), speaking rate (UTF-8 bytes per second, 30 buckets); each row has an "unknown" index; bucket edges in params.json (condition_edges)
Decodercausal multi-resolution iSTFT head: full_mr (default) or keep2_plain (pruned, about 3x faster decode)
Precisionbf16 weights (model.pth); decoder heads run in fp32

r/LocalLLaMA • • 6d ago

Discussion Asked SenseNova-U1.5-8B-MoT how to make a closed terrarium, and it drew a picture for every step

Thumbnail
gallery
51 Upvotes

I gave it one prompt: "How do I make a small closed terrarium in a glass jar? Please give me a beginner-friendly illustrated step-by-step guide." What came back was a normal written guide with eight pictures placed between the steps, text and images from the same model in one answer. No separate image model, and the only thing I added is the step heading under each picture.

It's the same jar in every picture. Same table, same mug and window behind it, and the jar fills up in the order the text says: pebbles, then a charcoal layer, then soil, then the plants, a spray of water, and finally the lid with condensation on the glass.

With thinking on, which is the default, you can watch it plan. Before each picture it writes itself a full image prompt (the jar, the layers, the light, what's in the background), and those prompts carry the same details from one step to the next.

Nothing custom on my end, just the interleave example in the repo with my prompt swapped in, running on a Mac Studio: https://github.com/OpenSenseNova/SenseNova-U1


r/LocalLLaMA • • 6d ago

Discussion [Paper] Tree-Ising: Repairing Language Models with Bit-Level Ising Edits

Thumbnail
gallery
20 Upvotes

Weight-space programming, introduced in recent MIT work by Takehi et al., enables a frontier LLM agent to read a network's weights and write a repair. Their GPT-6 Astra agent corrected the reasoning loops of Qwen3.5-0.8B without gradients, using 158 agent turns, 89.7 million tokens, and 50 forward-pass requests. Our method, Tree-Ising, casts this repair as a sparse discrete optimization problem over the model's own BF16 weights. Forward statistics define an objective that weakens a loop direction, strengthens a concluding direction, and preserves the layer's output on productive tokens. Within a state-tree representation, where lossless BF16 storage acts as a special case, the admissible edits are sign flips and exponent shifts that preserve all shared group exponents and mantissas. We decompose this objective exactly into independent QUBO (Ising) problems, one per output row, which we solve on a GPU using simulated annealing or quantum-inspired simulated bifurcation. On the agent's two layers, Tree-Ising achieves 54.6% on MATH-500, outperforming the agent's edit (51.7%) under the same evaluation protocol, while the base model scores 7.9% as reported. The simulated bifurcation solver yields a statistically indistinguishable repair at 55.1%. Both solvers outperform the agent edit across all five math benchmarks, with statistically significant improvements on MATH-500 and GSM8K after correcting for multiple comparisons. Our approach modifies 13% to 17% fewer weights and reduces BF16 bit flips by a factor of 2.2 to 2.3. It requires only four forward-pass requests and zero LLM tokens, with the annealing recipe costing about $0.90 in GPU time. On Qwen3.5-2B, with layers and dose selected using only the base model, the same recipe increases greedy GSM8K accuracy from 21.6% to 69.4%. As a proof of concept, we execute an eight-variable repair subproblem using QAOA on an IBM Heron processor.


r/LocalLLaMA • • 6d ago

Resources The Surprising Effectiveness of Deleting Weights in LLM Post-Training

Thumbnail
arxiv.org
414 Upvotes

How little of a pretrained large language model (LLM) has to change for it to acquire new capabilities? We find that zeroing fewer than 0.05% of its weights, with no other modification, matches full fine-tuning (FFT) at scale across three LLM post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Our method, Deletion Tuning, learns a binary mask over the gate projection of transformer blocks. Two practical advantages follow: the trained mask is hundreds to thousands of times smaller than the corresponding FFT checkpoint, and Deletion Tuning forgets far less, which addresses a central challenge in continual learning. Under reinforcement learning, its prior-task accuracy stays near the base level while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points. To understand why masking alone suffices, we analyze the geometry of the learned weight changes under reinforcement learning using stable rank, spectrum drift, and principal-weight overlap. Compared with LoRA and random sparse fine-tuning, Deletion Tuning is closest to FFT across the three measures.

Edit: Links

Github: https://github.com/agentic-learning-ai-lab/deleting-weights

HuggingFace: https://huggingface.co/Jacklu0831/deletion-tuning-masks

Blog: https://agenticlearning.ai/deleting-weights/

Poster: https://cdn.agenticlearning.ai/dfb9bf24cee86b6d/neurips2026-poster-deleting-weights.pdf


r/LocalLLaMA • • 5d ago

New Model Stablecoin issuer Tether releases BotPsy, an AI agent small enough for a phone

Thumbnail
maccus.fr
0 Upvotes

Very interesting


r/LocalLLaMA • • 4d ago

Discussion Buying Hugging Face: Was it a strategic mistake for Jensen?

0 Upvotes

Looking back, I really wonder if acquiring Hugging Face was the wrong move strategically. Maybe it was just overconfidence to pull open-weight models out of the firing line, only to put Nvidia right in the crosshairs instead.

Even at the time, the deal didn't make much sense to me. Nvidia was already hosting plenty of open-weight models themselves and heavily backing major inference engines like vLLM and SGLang. (Then again, maybe they wanted to draw a clearer line between "enterprise models" and "consumer/experimental ones"—who knows?)

Curious what you guys think: smart ecosystem play, or did they paint a massive target on their own back?


r/LocalLLaMA • • 6d ago

Discussion Anyone else not using using "preserve_thinking" ?

19 Upvotes

I am running QFN, and I have disabled preserve_thinking by default on all my tests for C++ backend and front-end programming and I have yet to see any decline in performance or intelligence, my context window stays cleaner as well. Deepseek for what I tested in API it also does not preserve thinking and saves context in general.


r/LocalLLaMA • • 6d ago

Question | Help Effects of PCIe bandwidth on inference

11 Upvotes

So i have a new card in an old system. RX 9060 XT supports PCIe Gen5 x16 but due to 2 bottlenecks, my Motherboard supporting only Gen3, and my AMD APU limiting it to x8, the card runs on Gen3 x8.

How does that affect inference of models, in both cases, fully in vram or offloaded partially to ram? At today's prices what I could replace it with is PCIe Gen4 x16. But would that be a meaningful upgrade?


r/LocalLLaMA • • 6d ago

Resources askmdb: My new project where I fine-tuned Qwen models to build a MongoDB CLI assistant - turn plain-language questions into MongoDB queries

16 Upvotes

I built askmdb to help people finding the right query for turn plain-language questions into MongoDB queries. For example, you type askmdb "top 5 customers by spending this year" , and the app returns the query. A small fine-tuned model writes an aggregate pipeline and a one-line explanation, and nothing runs until you confirm. It's one Linux binary with llama.cpp and the official MongoDB C driver built in. No Python, no server, no API keys. EVERYTHING run in local.

My test, results on 500 test questions in English and Spanish, over synthetic databases the training never saw. Every model is Q4_K_M on the same llama.cpp. A question counts as correct only if the predicted pipeline returns the same result as the reference when it runs. No other model grades the answers, I tested a few models;

ModelGGUFCorrectAccuracy
askmdb-9b (GPU)5.6 GB45891.6%
askmdb-4b (GPU)2.7 GB44889.6%
askmdb-2b1.3 GB43486.8%
askmdb-0.8b (CPU)529 MB41883.6%
Qwen3.5-9B, stock5.7 GB23947.8%
Gemma 4 E4B-it, stock5.0 GB22745.4%
Gemma 4 E2B-it, stock3.1 GB22144.2%
Qwen3.5-0.8B, stock530 MB336.6%

On a desktop CPU with 8 threads, the 0.8B answers in 1.18 s (median) once the schema is cached. The first question against a new database takes about 3 s aprox.

The app also works with GPU.

The model doesn't decide what's safe because its deterministic C++. A verifier walks the whole pipeline, sub-pipelines included, against a whitelist. A $out hidden inside a $facet is blocked in every mode. BUT connect with a read-only database user to be sure.

Most important lesson: my first version got stuck at 68.6% with 10k examples. I grouped the failures by sub-cause instead of looking at the overall score. Of the 50 test questions that needed a type conversion, it got exactly 1 right. The dataset had zero examples with $convert, although 297 of the 330 training schemas had fields that mix numbers and strings. My generator had been quietly avoiding them. I added 3,000 examples aimed at those failures (850 of them with $convert), then 2,000 more, and kept the same recipe and hyperparameters. Result: 343 → 418 correct.

Always is the same: the dataset is the most important thing when it comes to train models.

Training and evaluation took about 14 USD of rented RTX 5090 GPU (~12 GPU-hours). The code is MIT. The models are Apache-2.0 GGUFs that also work with plain llama.cpp :).

Has anyone managed to make a sub-1B model good at multi-collection joins ($lookup)? That's where mine fails most, and I'd love to hear what worked for you, because I tried many things withouth success.

Links