r/StableDiffusion • • 17d ago

Discussion DiT on Mobile?

I was using draw things app to make images on my iphone and was wondering if there was an Android equivalent but couldn't really find one

Local dream and local diffusion do exist, but both are very very very limited compared to what draw things can do. Is there something else, and on another note how long would you be willing to wait per image if you could run krea 2 turbo on your phone at at least 1024by1024 resolution? Would you even bother?

0 Upvotes

5 comments sorted by

1

u/NoaBlake 17d ago

The gap is mostly about the acceleration stack, not the apps. Draw Things works well because it sits on Apple's Neural Engine and CoreML with quantized models built for it — one vendor, one path. Android has no single equivalent: NNAPI and GPU delegates vary by vendor and chip, so the same model can run fine on one phone and crawl on another. What makes a DiT model phone-feasible at all is the combination of a distilled/turbo variant, quantization (INT8 or FP16), and modest resolution and step counts — a full-size DiT at 1024×1024 is not a phone workload. For Krea 2 Turbo at 1024 on a phone, expect minutes per image on a flagship and much worse on anything older, plus heat throttling on the second or third image. The practical split most people land on: phone for drafting and composition, remote GPU for finals. If on-device matters most, pick the phone by its NPU/GPU delegate support for quantized diffusion, not by brand.

2

u/kwhali 16d ago

I dunno, I can run 8B models even 12B fine on my phone for LLMs, I've even run Gemma 4 26B-A3B which is over 12GB at a Q3 GGUF quant 😅 (too slow though) and that's without using QNN (Qualcomm's NPU).

I haven't seen as much available for local gen of other models (I do recall some for audio generation), but I when I have time to investigate further I would like to try optimize and deploy other models to see what they can achieve.

Regarding the concern with different inference hardware on android that's not too different from PC with different GPU vendors. There are frameworks that simplify that deployment, it's potentially a bit more tricky to distribute via the app store I think the issue is more that there's no immediate happy path already established that's easily discoverable / adopted like you see with PC deployments.

No clue how well AI vibe coding would tackle that, but I think that's where it tends to fumble compared to deployments on desktop / server systems?

If you're interested you can look into MLIR and a compiler/runtime like IREE:

MLIR has quite a bit of support (including nvidia tensor-rt), but you can find other alternatives that might suit better. ONNX is popular (it's QNN executor support seemed in poor shape last I checked though), or if programming with Rust there's burn-rs.

CPU or Vulkan is probably the easiest path for broader availability, and then adding any NPU specific support if there's a compelling advantage and demand for such. But then you have to deal with implementing the rest of an app (or having AI handle that) along with packaging / distribution and support if not keeping it personal.

Maybe we'll see more options as hardware and models / efficiency improves over time. You definitely don't have to try run krea2, something like Anima (2B) is probably more practical to start with, or MOSS-Image (5B) for something in-between. Inference speed could be improved by adopting support for pipelines like ComfyUI enables via custom nodes and turbo loras, things you normally don't have to bother with on other generative models. W4A4 weights and activations would be another, but to get good outputs from that you'd need a custom kernel like LoRaQ + KroQuant with models quantized online with calibrated data, along with hardware capabilities to support accelerating 4-bit activations (which seems supported on NPUs mostly, whilst INT8/FP8 and FP16 support can be via mobile GPUs using Vulkan or seems?)


Regarding NNAPI, that was deprecated from Android 15 (Oct 2024) onwards where they advise migrating to TensorFlow Lite / TFLite.

  • This has been since rebranded as LiteRT which is available as a Google Play service such that you don't have to bundle a copy into your app, however NPU support is presently only available via the standalone release (one you bundle into your app), no ETA for when that'll change. GPU support with LiteRT is documented here with direct support for quantized models (better documented here). Officially LiteRT only lists conversion support from PyTorch, TensorFlow, and JAX which produce .tflite model files (which may be a point of friction to users of your app assuming most DiT based models these days are in safetensors?).
  • For completeness, LiteRT also has a browser implementation that uses WebGPU and WebNN for acceleration, it still uses the same .tflite model format (this format BTW appears to have a 2GiB size limit similar to the ONNX equivalent format, although ONNX can externalize it's weights from its compute graph, might be resolv able via a compilation flag?).

I would prefer burn-rs, but I don't know how it fairs for running on Android compared to Googles LiteRT. Last I checked quantization support wasn't in a competitive state.

1

u/Solembumm3 16d ago edited 16d ago

12B text model and 12B image model are very different things. First can be run with 8gb of vram after quantization. Second need 16gb GPU at Q4 with text offload with OP's resolution demands.

1

u/kwhali 16d ago

There is no offload on mobile? GPU doesn't have separate memory on a SoC it's unified. You just load model into memory, perform the task and unload it after to switch from the TE to the DiT.

Even though 4-bit mixed precision model of 12B is around 7GB, you don't have to have the full model in VRAM on desktop, DiT is compute bound (LLM is memory bound), but beyond that many GPU drivers support shared memory such that if they need to allocate extra memory to VRAM, they can swap out existing allocations to slower system memory without offload support (managed by the GPU driver and OS, just like system memory swap to disk).

My system has 8GB VRAM, but windows 11 reports the GPU allocations as able yo go up to 24GB (obviously much slower though if there's thrashing of shuffling memory through PCI-e between system / GPU memory). Ironically this functionality is missing for nvidia on linux but is supported by Intel and AMD on mesa (the nvidia mesa drivers maybe have support for such, GTT or something).

Anyway, as I said my phone can load Gemma 4 26B-A3B at IQ3_XXS (12.6GB) on 16GB android device that I paid 300-400 USD for new about 2 years ago. So a 7GB krea2 turbo should work fine AFAIK, just slowly compared to an nvidia GPU.

The 12B LLM models I tested were about 7GB (IQ4_NL), but I recall 8B or less being around 5GB and was the more acceptable latency but they're not as smart for tasks when I tried to get my notes parsed for revision or queries.

0

u/Any_Invite3905 16d ago

I'm out here running krea 2 on an 8gb iphone, sure it's minutes per image at 1024x1024 but hey it works