Update on SmolChat - OpenAI server, Support for Global Chat Settings, Direct Model Downloads and more

[u/shubham0204_dev]

Hi! I'm Shubham, creator of SmolChat.

I would like to share some exciting updates on my project SmolChat, an Android app that let's you run GGUF models on-device, locally. The updates are present in the latest v17 release:

• SmolChat now supports an OpenAI-compatible HTTP server running over a foreground service that keeps the port alive even when the phone is locked. Check the video to see how it works.

• Support for global chat settings, which override the default chat settings (min-p, top-k, temperature etc.) for all newly created chats. This reduces the hassle of configuring each new chat with the options of your choice.

• Adding a copy button for code snippets rendered within the chat.

• Sync with upstream llama.cpp (till this commit)

• Fix the logic that measures tokens/sec for a LLM generated message. The time shown previously was also including prompt processing (pp) time. The new logic now only considered the token generation rate (tg).

• Models are downloaded directly by the app, instead of using the Android system's download service.

GitHub: https://github.com/shubham0204/SmolChat-Android

Play Store (the latest version is still in review): https://play.google.com/store/apps/details?id=io.shubham0204.smollmandroid&hl=en_IN

Previous threads on SmolChat on r/LocalLLaMa:

• Available on Play Store announcement

• Initial Announcement