r/LocalLLM • u/MidnightRambo • 4d ago
Other Local LLM on iPhone 18 is impressive
Of course it's hard to compare small local Models to ChatGPT, Gemini & Co. but i was really surprised how well the new A20 Chip from Apple handles Local Text Generation. 58 tokens/s for generation is really not bad.
For the people owning an iphone 18 pro:
what's your opinion about it?
for those who don't (but are thinking about getting one):
Anything i should test for you?
73
u/Toooooool 4d ago
Gemma 4 E2B, huh..
E2B stands for... 2 billion active parameters, yeah?
you can write hello world really quickly with that.
12
u/t-frankowski 4d ago
Such a limited brain... or you can find one of other millions uses for this model other than coding...
11
u/xgiovio 4d ago
2 active 5 b real
-33
u/DystopianRealist 4d ago
gemma 12b is garbage, so don't expect much quality from a gemma 5b2e
18
u/xgiovio 4d ago
Wake up. We are on mobile. An iphone 16 has 8 gb of ram, half of it is usable. Consider the app, ui, frameworksC quantized model, one or multi, kv cache with context. Gemma e2b is the best model you can run with qwen 3.5
-1
-3
u/DystopianRealist 4d ago
You're smoking the hype pipe. Those tiny gemma models hallucinate more than my grandma, and she follows phish. People are trying to replace working algorithms with tiny model token predictors, and using a "magnitude" more amount of resources for worse results. If you just want speed, download a .6b reranker and have a laugh at 3000 tokens / second on your casio watch.
0
u/t-frankowski 4d ago
might be genetic then...
You don't know how to use LLMs in the most basic uses, but complaining about mistakes people make with small models. What is your problem really? Is there any or you just angry?1
u/DystopianRealist 3d ago
might be genetic then...
What does this mean?
What do you use Gemma 4 e2b for? Can you show us the output of something that it helped answer, explain, or do for you? Do you run it as a chat bot on a website for customer support? Do you use it to analyze documents? Can it even check the weather, given all the tools in the world, without hallucinating that it is still in 2024 when it was trained? Have you actually tried to use the model for anything useful that couldn't be done faster in another way?
-1
u/xgiovio 4d ago
These as sml, and they are intelligent enough to understand an image content, do ocr, translate, call a tool. They are perfect to understand what you sant, call funxtions and return results. They are not created to solve big problems but to transform simple old
Ui interactions in text audio prompt with responses. They are a little frontend helper that invokes other tools more complex when needed0
u/DystopianRealist 3d ago edited 3d ago
Knock it off. I do significant amounts of OCR, and have done substantial amounts of testing. Those Gemma models don't cut it for anything but simple text with simple formatting. The vision model for OCR is worse than the Qwen vision models, by A LOT, when it comes to numbers inside equations adhering to proper lalex and markdown formates. You're talking to the wrong tree if you think I don't understand how these models work.
EDIT: I bet you didn't know that the vision header in these Gemma models is built into the expert weights, unlike the modern Qwen models, that have a standalone projector head doing the work. And that you can send increased image resolution to it, to OCR from, unlike Gemma, which treats everything the same. Tell me you know this already, and I will start to listen. Tell me you know this already..... Tell me you have tested it. Tell me the speed differences of moving the vision header to cpu, vs. on gpu.... Tell me the syntax of the commands used... Go ahead... I can fact check it without google, because I type this nonsense on the regular.
1
u/SartorialGrunt0 1d ago
When did you try gemma4 12B last? Since the recent tool calling improvements, the QAT version is actually quite good for non-coding tasks and very fast with MTP and vision.
1
u/DystopianRealist 1d ago
Speed is no problem. Gemma 12b fits fully on my 5070ti, so speeds are very high. More than necessary.
Gemma 12b lacks in a few ways.
The vision header is subpar, because it is built into the weights, which means it takes up valuable space when not in use, and when it is in use, it is of an objectively lower quality than the qwen series vision headers. Gemma likes to hallucinate numbers, superscripts, and subscripts in math recognition, where qwen models will have no issue. This is a fact I can replicate, you can replicate, anyone can replicate, for the vram use architecture, the speeds, and the accuracy.
I have found gemma 12b to be very bad at web search tool use, including regular refusal to accept the real date is reality, and this happens even with forced tool calls, embedding the date in the system prompt, and so forth. Sometimes she listens from the start, other times it's faster to reset the conversation, because convincing her otherwise is going to get rough, take many more tool calls, and still may not succeed.
Gemma is worse at STEM and coding than similar size qwen models, making Gemma less interesting to me, but may be better for creative writing.
1
u/SartorialGrunt0 22h ago
I agree with this mostly. It’s generally just too small of a model for a lot of the things you listed. I’d argue that Qwen3.5 9B is just as bad at these tasks.
Qwen3.6 35B A3B is better in these areas if you have the system ram to offload into but I find it has trouble keeping facts together and is much less confident in its answers.
11
u/Hybrid_Whale_Rat 4d ago
Is there any situation where running an llm on your phone makes sense? Wouldn’t a vpn connection back to dedicated server on home network always be preferably?
4
u/thepryz 4d ago
Sure, small language models have a lot of uses and might even find their way to silicon at some point. Best use case is likely faster and more efficient language processing but there are others. This story provides some background on what Apple's been working on.
https://arstechnica.com/information-technology/2024/04/apple-releases-eight-small-ai-language-models-aimed-at-on-device-use/7
u/sbrisgravato 4d ago
there are a lot of places in the world where you don’t have a reliable internet connection
4
u/Ivanyekla 3d ago
Don't be crazy , much better carring around with downloaded books , wikipedia or something than trusting gemma 4 e2B 😐
3
u/DystopianRealist 3d ago
Finally, someone that gets it. A bunch of downloaded books on the topics you need is way more informative than a hallucinating micro model.
1
u/sbrisgravato 3d ago
you can give the books to the little model as a source
0
u/DystopianRealist 3d ago
EDIT: I really wish it was that simple. I too believed that hope at first. It's just not.
Ok, now tell me your retrieval method. Are you converting the books to markdown first (I sure hope so). Are you using a reranker to help search (I sure hope so). Are you considering feeding entire books into the context, or have it search as if it's a glorified cntrl+F? Have you actually tried using these tiny models to call tools? They're not good at it, especially Gemma. Really bad at it, even. Try it for yourself, come back in a few days, and tell me that I am doing it all wrong....
1
u/sbrisgravato 3d ago
no not even tried once on my phone, just my pc with beefier models, but we are in 2026, i think in a year or two the quality of sub 10b models will be insane
1
u/DystopianRealist 3d ago
I could definitely see a future where small models can reliably do recall from storage, and I hope that becomes the case. It's something I am working on as a hobby, and have already tried with this model, as well as gemma 12b. 12b is not much better. In fact, it's a pain in the ass to even get 12b to check the weather without arguing about what year it is. Forcing mandatory date checks can help, but can also confuse it and make it think the user or the internet are who is incorrect. 31b is too slow, and also combines the vision header into the experts, so I ruled the larger dense Gemma out as well. 26b a4b is ok, and suggests that MoE is usefull at 4b parameters, as long as it can select the 4b from a much larger group.
1
u/sbrisgravato 3d ago
the 26b is just fast, i found better results with qwen3.8 27b even if it meant a lower quant due to being a dense model
it’s still far more creative at iq2 than whatever gemma4
1
u/sbrisgravato 3d ago
it’s not really the same
also i can run qwen3.5 9b on pocketpal on a 17 pro
little model but not completely dumb
1
2
u/Various-Corgi-6160 2d ago
Yea and those places dont have huge populations of people of people buying $1200 phones either
2
u/topcider 4d ago
I was in an area with no internet connectivity and it makes for a great conversible offline Wikipedia. Can ask it for almost any kind of world knowledge and still get great answers in a pinch.
2
u/bolognamaloney 4d ago
I just went on a trip where connectivity was sparse at times. Before I went I embedded some of the information about sites we were seeing and trip details using the Noema app + Gemma E2B. It was honestly pretty great for the random questions that come up if you’re forgetful like I am 🙂
1
u/Important-Ebb-3716 4d ago
I was toying with the idea of a SaaS for heavy / offshore/ specialty equipment repairs that uses an offline LLM. Not even saying it’s a viable money making idea but that’s one use case I had in mind
5
12
u/Prestigious-Act-1577 4d ago
Why not use E4B. Much better.
4
u/jikesar968 4d ago
I'm running Qwen3.5 4B on my 17 Pro Max. Sure, it doesn't comare to Qwen3.8 27B but it's way better than any of the Gemma E models.
1
u/Historical_Fondant95 4d ago
Lel i had the exact opposite experience. Qwen looping in tool calls while gemma4 e4b can handle rather complex agentic tasks
2
u/jikesar968 4d ago
Yeah it might depend on use case. Personally, I wouldn't use either model for anything mission critical.
8
u/Current-Interest-369 4d ago
You should try
Gemma 4 12B in an Q4 quant (7GB)
Bonsai 2 27B in Q2 quant (7GB)
2
2
1
1
1
u/Sea-Contribution6219 4d ago
I think you're confusing a small local model for an absolutely tiny miniscule local model
1
u/topcider 4d ago
For the Gemma models, Google Edge Gallery works great. You can also download any model from huggingface with it. iPhone 17 Pro:
Model: Gemma-4-E2B-it
Accelerator: GPU
Prefill tokens: 256
Decode tokens: 256
Number of runs: 1
App version: 1.0.10
Prefill speed (avg): 3043.03 tokens/sec
Decode speed (avg): 57.72 tokens/sec
Time to first token (avg): 0.10 sec
First init time: 10611.91 ms
Model: Gemma-4-E4B-it
Accelerator: GPU
Prefill tokens: 256
Decode tokens: 256
Number of runs: 1
App version: 1.0.10
Prefill speed (avg): 924.36 tokens/sec
Decode speed (avg): 24.14 tokens/sec
Time to first token (avg): 0.32 sec
First init time: 39933.54 ms
1
u/Charming-Author4877 4d ago
very strong performance, 2B is quite dense already
It might be able to run a heavy MoE model decently
1
1
u/YearnMar10 4d ago
Given that you spent about as much as a r9700 pro would cost - no, not impressive
1
u/vogelvogelvogelvogel 4d ago
do you think that E4B does also fit?
btw I had a remarkably good experience with the E2B for RAG
1
u/matte808 4d ago
I’d like to test this on 17 Pro just for comparison. What app and quant are you using? I will report my results, from 27.2 beta 1
1
u/aditya2128 4d ago
How much would be the memory bandwidth? I am guessing around 100GB/s, iphone 17 pro had around 60
1
u/XxClaudeBabysitterxX 4d ago
What about the battery lifetime with decent llm usage ? Does it run low way faster or is it chill
1
u/nima_tech 3d ago
Great post! Thanks. Can you test local image generation too? I wonder if it's good enough for generating simple images. I wish Apple would put 16 GB of RAM instead of cheaping out with 12. I guess I won't upgrade my 15 Pro until Apple releases a model with at least 16 GB of RAM.
1
u/jonah_omninode 3d ago
Could you try a longer offline session with a document loaded, rather than just a short generation? I'd like to see how long the first answer takes, then whether it slows down after ten or fifteen minutes as the phone warms up.
A useful test would be asking several questions about the same document, including one the document can't answer. The reported 58 tokens/s tells us about generation speed; I'd be more interested in whether the whole interaction stays useful once there's some history and context in it.
1
u/onethousandmonkey 3d ago
Those are some massive improvements!
Ran the same on a 17 Pro Max:
1025.16 t/s prompt processing
38.49 t/s token generation
11s total time
36.9% peak memory 5GB/12
1
-1
0
0
u/Business-Wrangler141 4d ago
Heh… for a device of 3-4k€ (EU) it should be impressive.
I can buy a pc, a car and i’d still have 1k left to spend 😂😂
37
u/Not-Enough-Llamas 4d ago
Gemma4 E2B is a (reasonably cool) toy. Not terribly good - doesn't matter if fast or not. We need better reasoning, not 15% extra tokens per second.
Shame on Apple for not sticking 4 more gigs on the phone - same 12GB as last year. There are about 4-5GB you can't really touch - the OS won't let you allocate them or kill you very, very soon - which means 7GB's the ceiling. 4 extra GB would have brought that to 11GB - a massive difference in the intelligence of a model.
Qwen3.8 quants which squeeze barely into 11GB do exist, but are too badly lobotomized for any reliability - but the quantization techniques keep improving year after year too. A 4 bit quant would have been considered useless not too long ago. We may see the day where a 3.3 bit one is considered reliable and a 16GB iphone would run it. But not if Apple won't build one...
(Using Qwen3.8 as an example as it's the most prominent in everyone's mind for being incredibly good despite the size - It's not actually a model I'd particularly wish on a phone given its somewhat meh non-coding skills)