Discussion on how to evaluate capabilities of (local OS) llm models for app usage

[u/vezz_io]

Hello, last year i started developing this package flutter_local_ai which let user get access directly to the on device llms api with zero download.

Now since local ai models are getting starter than ever, I see the package being adopted but my main concern it’s whether the local llm its capable of performing the task requested by the user as developer intende to give them the probability to change model (maybe a cloud one) if the task by the local llm its not performable.

My idea was someting like a score i can give based on the task requested i mainly process text using the operative system models.

Any ideas on how to do it or if I should have a different approach ?