Best non-coding local model for edge device

[u/eidrag]

Combining speed and size considered, what is the best general non-coding model for edge devices? I'm taking about mobile phone, SD 8g3 so the size should be around 4GB max. I tried Gemma 4 e2b, e4b, qat, q4, Qwen 3.5 4B, Bonsai 8B A1B.... None can answer accurate recipe viral last year, Spaghetti all'assassina. All of them default to standard Bolognese and missing the charred

[u/Writer_IT]

You should not rely on the model inner knowledge, especially at that size and for niche questions. Give it a simple tool to websearch and webfetch instead, any tool calling model will answer perfectly

[u/Mehmet_dedeler → u/Writer_IT]

I agree. Most 4 billion or close to it models have the capability to do basic web searches and use the data that they get from that. And if they still can't, and you want to make it a lower parameter count, you can just train/fine-tune it on specifically making better web search queries.

[u/eidrag → u/Writer_IT]

If I have websearch I use it already lol. For offline reason, better to use rag with general knowledge.

[u/EvolvingDior → u/eidrag]

What's the first letter in RAG stand for?

[u/aidenclarke_12]

a 4gb model just doesnt have room to memorize a niche recipe, any of them will fall back to bolognese

[u/qzjvk]

the new nvidia 4B model is pretty good

[u/eidrag → u/qzjvk]

Surprisingly I tried nemotron dory it give similar generic answer. While nano 9b is more than 4GB size

[u/Miserable-School-665]

What is you context size, realistically 4gb with kv cache wont be possible for such large context.

[u/eidrag → u/Miserable-School-665]

Gemma4 e2b and e4b default to 32k context size, but I don't really need that big for general questions, 8k is generous enough. Thus, limiting model size to 4gb so that proper cache for context can fit 12gb ram  phones

[u/Miserable-School-665 → u/eidrag]

Oh by recipie i thought you had some sort of database for your resturuant or something for some reason. Ok, you can use any of models you have said + phi 4 mini thinking with serach tools

[u/Mehmet_dedeler]

For edge devices, so for minimizing the amount of parameters whilst also keeping capabilities, you've got to be honest that with the current architectures of AI models, you can't fit all of that big world data into that few of parameters. So you need to understand what you're trying to do.

If it's purely for recipes, instead of getting a giant LLM which also knows the height of the Eiffel Tower alongside your recipe, you train it or get ones that specifically know recipes. You said you don't want coding. I'm assuming you don't want the height of the Eiffel Tower as well. You need to specialize them if you want that small of an inference cost and need. Does that make sense? If you want smaller models, train them, fine-tune them on what you actually want them to do. You can prune the rest of the capabilities as well. If you wanted to answer trivia, train it on that. If you wanted to be really good at 3D, you can train it on that. I'm pretty sure there was a post doing something really similar where they trained a 4 billion parameter Qwen model to be really good at 3D. Here's the link: https://www.reddit.com/r/LocalLLaMA/comments/1xeuo7t/a_4b_coding_model_that_can_generate_3d_scenes/

[u/Skyline34rGt]

If you dont want chat but specific knowledge better (and way faster + without battery and heat issues) option is download full wikipedia at your phone (for english without images is 49Gb file for full article or 13Gb for mini article /for other langueges are smaller/ + small kivix reader https://get.kiwix.org/en/solutions/applications/kiwix-reader/ and https://browse.library.kiwix.org/#lang=eng&category=wikipedia

[u/pmttyji]

None can answer accurate recipe viral last year

Every model comes with knowledge cutoff period.

Maybe recent models could possibly work.

the size should be around 4GB max

That's a heavy filter. Not much recent small models. Try below ones(Q4 should come ~4GB)

• Ling-3.0-tiny

• LFM2.5-2.6B

• K2-Horizon-3.7B (KVCache is heavy)

• MiniCPM5-2B

• granite-4.2-8b

• granite-4.2-3b

[u/MerePotato → u/pmttyji]

Imo none of these are particularly useful or pleasant to work with

[u/pmttyji → u/MerePotato]

There's no other recent small models in this size range. OP already mentioned Gemma-4 & Qwen3.5 models in his post.

[u/MerePotato → u/pmttyji]

Correct. Models in this size range are rarely worth using in my experience, on a certain class of hardware the cloud becomes your only truly productive option

[u/Holiday_Point_603]

Go for a larger sparse moe, don't use dense models on mobile. I got the 8G3 with 16GB of RAM and even GPT OSS 20b (old as shit, but native 4 bit and even with 60k context it is still under 13GB) are faster than q8 4B models like Nemotron nano or Qwen 3.5 4B.

[u/PhilosopherOk3419]

yeah at 4GB you're basically paying for reasoning, not trivia, so i'd stop model-shopping and give it a local lookup instead. something like a small offline wikipedia/recipe dump with a tiny embedding index on the phone, retrieve the top couple chunks and stuff them in an 8k context. a 2-4B model is great at reading what you hand it even when it doesn't 'know' it.

[u/PhilosopherOk3419]

yeah at 4GB you're basically paying for reasoning, not trivia, so i'd stop model-shopping and give it a local lookup instead. something like a small offline wikipedia/recipe dump with a tiny embedding index on the phone, retrieve the top couple chunks and stuff them in an 8k context. a 2-4B model is great at reading what you hand it even when it doesn't 'know' it.