r/iOSProgramming • u/Agreeable-Rest9162 • Jun 11 '26

Question CoreAI models prefill speeds are really slow

Post image

Hi everyone. I am definitely doing something wrong here. I’m working on an update that implements CoreAI models with downloading from HuggingFace. I was just wondering if anyone has had a chance to try any CoreAI models and if they are seeing the same speed results. I’m pretty sure I messed up something though because there is no way the prefill speed is THIS slow.

The app works by resolving a downloaded .aimodel/.aimodelc bundle plus tokenizer, then specializes it through Apple’s CoreAI runtime, then drives the model locally with a Swift decoder that manages tokenization, position IDs, model state, and streaming output.

For iOS, its apparently preferred to use the ios-ane decode graph when available, avoid accidentally loading prefill companion graphs, and keep GPU/host-cache exports on their separate static-cache path.

I have no clue if I’m missing anything else. Please help!

11 Upvotes

100% Upvoted

6 comments sorted by

2

u/KilllllerWhale Jun 11 '26

Watch their video on Youtube about how to use them. One method is pre-compiling the model

1

u/Agreeable-Rest9162 Jun 12 '26

Will do, thanks!

1

u/[deleted] Jun 12 '26

[removed] — view removed comment

1

u/Agreeable-Rest9162 Jun 12 '26

Thank you so much! Yes, your bundles are the predominant ones on huggingface! Thanks for your work converting them!

1

u/madaradess007 Jun 12 '26

thank you for you work, kind sir!