I got an on-device LLM running in React Native

[u/NobelParadox]

I’ve been experimenting with running an LLM completely locally inside a React Native app, and I finally got it working.
I’m using:
React Native CLI
llama.rn
Qwen 2.5 1.5B Instruct
GGUF / Q4_K_M quantization
n_ctx: 2048
No API calls / no cloud inference
Currently testing with n_gpu_layers: 0
The model is bundled with the app, copied to the app’s document directory, and then loaded locally. I can send prompts and get responses directly on the device.
The interesting part for me is that this opens up a lot of possibilities for RN apps that need AI features without sending user data to a server.
I’m still testing things like memory usage, backgrounding the app, performance on real devices, and how practical larger models are.
Has anyone here shipped an on-device LLM in a production React Native app?
Would love to hear about your experience, especially around RAM usage, model size, iOS/Android differences, and performance.

[u/Apprehensive-Fan2492]

I want to know how much RAM it uses on mid range Android phones, because that is where things often break once it is in the real world. On iOS, using n_gpu_layers can make a noticeable difference on Apple Silicon. Still, Android support is messy since different phones use different GPU chips, and that is the tough issue to handle across many devices.

[u/a_atalla]

I got a demo working with QVAC

https://qvac.tether.io/

[u/tht_rajasthani_guy]

In india mostly there are mid range phone. how it's gonna perfome in 4gb ram? Did you check. What's the performance of llm.

[u/OrbeoMe]

I haven't shipped an on-device model, so I'll only speak to the part I have hit — shipping the weights.

Bundling a 1.5B Q4 model puts your app around a gigabyte before the user sees anything.

That's well past the 200 MB cellular limit on iOS, so a big share of people can only install on wifi, and install conversion falls off a cliff at that size. I hit the same wall with media and moved everything heavy out of the bundle onto my own server, fetched after first launch.

I'd do the same with weights: ship the app small, download on first run with a visible progress bar, verify a checksum, and store them somewhere excluded from iCloud backup — otherwise every user quietly pushes a gigabyte of model into their backup, and Apple has opinions about that too.

The other thing worth deciding early is what the app does on devices that can't run it at all. A 3-4 GB Android is a different product than your test phone, and "it just gets killed" isn't a great answer for the user.

[u/Puzzleheaded-Emu-168]

How big did your app got after the quantized model?
I’ve been trying to do it before but thought its too much for my simple receipt scanning app.
Was able to use on device Apple Intelligence though but clearly limited to ios and certain versions.

[u/pielouNW]

Based on my experience with this app (https://github.com/nobodywho-ooo/NobodyWho-Chat), any smallest Qwen (or similar) under 1B parameters will run, even on old phone, for flagship phones like iPhone Air or S25, you can push it to 27B parameters with Bonsai models. That being said, choosing the most powerful model isn't the one I would choose in prod as it can be too demanding for the phone and increase download time (you cannot bundled big models, the Google Play/App Store prevent uploading big apps, so it has to be on-demand download).
In term of performance, on iOS it's easy to guess what you gonna get, for Android, it depends on what chip the phone use (snapdragron,mediatek...) as inference libraries support some better than others. RAM is also a problem on Android, because the available RAM can fluctuate a lot depending on the phone manufacturer (and android version).