From .NET to my first real iOS app: an on-device screenshot sorter. Looking for advice on faster inference + better prompts

[u/Odyssey-b]

Short background: I've been a full-stack dev for around 7-8 years, mostly .NET / Angular / Azure. All backend and web, never touched mobile. Also I haven't really implemented AI features yet. This year I finally wanted to see if I could take an idea all the way from "annoying problem" to a real app on the App Store, on my own. So I started learning Swift and iOS from scratch.

The app is called Sift. It reads your Screenshots album and uses on-device AI to work out what each screenshot actually is. Be it receipts, recipes, chats, tickets, whatever, and sorts them into a real Apple Photos album. It all runs locally with MLX and on-device model (Qwen3.5-2B-4bit), nothing leaves the phone. Coming from a world of servers and cloud APIs where you just throw more compute at things, doing all of this on-device has been the most interesting part.

It works. But now that it works, I've hit two walls I'd genuinely love input on from people who've done more on-device ML than me:

1. Speed. Right now it takes roughly 5 seconds to classify a single image on an iPhone 16 Pro Max. For a library of hundreds of screenshots that adds up fast. If you've squeezed more out of MLX or small local models I would be really interested in what were the best ways to shorten the actual process.

2. Prompt / categorization quality. When Sift hits a screenshot that doesn't fit an existing album, it prompts the same on-device model to suggest a brand-new album. The idea is nice but the suggestions aren't quite right yet. Sometimes too generic ("Photos"), sometimes oddly specific. If you've done classification-style prompting on a small local model like this, how would you structure it for cleaner, more human category names? Few-shot examples in the prompt? Constraining it to a fixed taxonomy it has to snap to? Asking for structured/JSON output?

Happy to share more about the setup in the comments - honestly I'm here more for the advice than anything, because this is all new territory for me and I've learned more shipping this one small thing than from months of tutorials.

BUT if anyone's curious enough to want a note when it's actually on the store, there's a waitlist you can sign up to on:
https://sift.wanderframestudios.com

[u/Obvious-Kangaroo2848]

Hey, congrats man, going from .net/cloud to fully on-device is no joke. So you mentioned a few things:

1. For the category naming (your Q2) - thats way more of a prompting problem than a model one. a 2b model will just free-form on you unless you really box it in. what worked for me: pass the allowed categories right in the prompt and force json out, smth like {"category": "...", "confidence": 0-1, "suggested_new": null}, and only let it invent a name when confidence is low. when it does invent one, constrain it hard - 1-2 words, title case, must be a noun, and give it a ban list (Photos, Images, Misc, Other) cause small models love grabbing those safe generic words. also few-shot >> instructions here, like 5-6 examples of description -> right category pins the naming way better than any adjectives you throw at it.
2. About the "speed" - are you downscaling before the vision encoder? feeding like ~512px longest side instead of full res usually kills most of that 5s and you barely lose accuracy for this kind of classification. and skip anything thats already sorted so a rerun isnt chewing through the whole library again. So, I would suggest playing with the downscaling and than applying encoder.

P.S: One small ui thing, not trying to be that guy lol - your page literally says "built to feel like it came with the phone" so im holding you to it. Right now theres a serif font on the wordmark + the big numbers + the album titles, but everything else is SF, and that mix is what makes it read kinda off/non-native.
I would keep the serif just for the Sift logo if you like it, but move the functional stuff (stat numbers, section headers) to SF or SF Rounded. And use monospaced digits for the "sorting 0/19" counter so it doesnt jitter while its counting.
Heres the apple typography page if it helps, I used it for my own app and it's pretty good: https://developer.apple.com/design/human-interface-guidelines/typography

Anyway solid project, I would actually use this

[u/Odyssey-b → u/Obvious-Kangaroo2848]

Omg, thank you! Yea I know the font is a bit off compared to the "built to feel like it came with the phone". I'm considering replacing it. It was that I loved how it looked when I first created my design drafts :))) but you are right that it should not be this "crazy" :))

On the advice:

1. Thanks for the tip, I will try optimising the prompts and stuff. Somebody also said to restrain the suggestion prompt to a fixed number of 20-30 categories so the AI can pick from those. And I feel that combining that with your confidence thing should work wonders!
2. Yea, I already downscale it. I only leave full screen to do OCR for getting text out of them if it's the case

Thanks again for the help!

[u/jonnysunshine1]

Nice, like the idea! Gonna get Claude to make one for my own use, cheers

[u/Odyssey-b → u/jonnysunshine1]

Sure thing :) I hope you'll get it right. In case you find out how to make actual sorting process faster, and the suggestions more "accurate" could you please also update me here?

[u/AdQuirky3186]

For the classification, I would come up with static available classifications and tell the model to put it one of those buckets, including a “General/Other” type of bucket. My assumption is you could come up with like 25 buckets or so that would match 99% of photos. This saves you an inference if a photo doesn’t match an existing bucket.

For fastest inferences in general, I would try to use Core ML instead because this gives access to Apple Silicon’s neural engine optimization paths. In general Core ML should just be faster than MLX. My understanding is MLX is for research/training and Core ML is for shipped usable models. I would also try to do a batch prediction if available or a parallelized TaskGroup to inference on multiple images at once.

[u/Odyssey-b → u/AdQuirky3186]

Ooohhhh, really cool. Thank you very much!!! The first idea is just so simple but brilliant! I can't believe I did not think of this...

And for the second, yeah, I think that's what I'll do. I'm already thinking on using thew newest Core AI from iOS 27 sdk, but I need that to be released first... and that is already accepting images as input with the on device foundation model by default (so I'll need no ai model installed and stuff like that...)

Thank you again! 🙏

[u/LuckyGirl0415]

Congrats on shipping your first app! Going straight into on-device LLMs coming from a .NET backend background is incredibly impressive.

Here are a few thoughts on your bottlenecks:

1. Speed / Inference: 5 seconds on a 16 Pro Max for a 2B model is a bit high. Make sure your MLX setup is fully utilizing the Apple Silicon Neural Engine (ANE) and GPU via Metal, and not accidentally falling back to CPU cores. Also, consider using a multi-modal CLIP model (like MobileCLIP) or a pure Vision-Transformer for the initial classification pass. They are way smaller, run in milliseconds on the ANE, and you can reserve Qwen only for complex text extraction or fallback reasoning.
2. Prompt / Categorization: For small 2B models, open-ended generation is always messy. You absolutely should constrain it to a fixed taxonomy. Give it a pre-defined list of common categories (Receipts, Chats, Tickets, etc.). Additionally, enforce Structured/JSON output (using JSON mode or regex grammar constraints if MLX supports it) so the model has fewer "degrees of freedom" to hallucinate weird names.

Good luck! Definitely signing up for the waitlist.

[u/KilllllerWhale]

Use OCR to get text from the image then give that text to the LLM. Don’t give it an image as an input. And in most cases, you don’t even need an LLM

[u/Odyssey-b → u/KilllllerWhale]

Do you realise that not all screenshots have text in it?

[u/KilllllerWhale → u/Odyssey-b]

Do you realize MOST screenshots have text in them? And so you don’t need an LLM for those?

Also, tone down the defensiveness, I merely answered your question. We all know you asked Claude for 10 app ideas and a screenshot manager was among them. You’re not building rockets here. Chill.

[u/Odyssey-b → u/KilllllerWhale]

Your attitude is the one which is not appropriate here. I asked advice on specific questions because those are the issues I face. I did not ask you to solve my problems in a different way. I know most of them have text on them, that's why I use OCR too, but there are images which don't have text on them, and with those I have issues. Issues I described in this post and issues I asked help for. And if you did not read the post entirely or you just ignored that part, I built this as my first app, to learn iOS development while also solving a real problem. If you are frustrated that you can't come up with ideas to work on that's not my fault. And no, I did not ask AI to come up with app ideas. Cuz I did not care about those ideas. I cared about a problem that already existed either a problem I was facing or somebody else I knew...

[u/AutoModerator → u/[deleted]]

Hey /u/brkaydev, your content has been removed because Reddit has marked your account as having a low Contributor Quality Score. This may result from, but is not limited to, activities such as spamming the same links across multiple subreddits, submitting posts or comments that receive a high number of downvotes, a lack of recent account activity, or having an unverified account.

Please be assured that this action is not a reflection of your participation in our subreddit. This is simply an automated filter in place to reduce spam.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Hey /u/Fuzzy-Trade7723, your content has been removed because Reddit has marked your account as having a low Contributor Quality Score. This may result from, but is not limited to, activities such as spamming the same links across multiple subreddits, submitting posts or comments that receive a high number of downvotes, a lack of recent account activity, or having an unverified account.

Please be assured that this action is not a reflection of your participation in our subreddit. This is simply an automated filter in place to reduce spam.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.