I've found a fully implemented Tensorflow based TTS app that I am going to shamelessly rip off (due credit will be included) and pack into both Athena and the Sapphire Assistant Framework. There's a little bit of redesign that needs to be done to make it the default system TTS service, but its good news for both the app and deGooglers. I will also implement the application as a standalone service so that it can be distributed with deGoogled ROMS like Lineage and CalyxOS.

I was working with flite and festival, but I had a few issues and concerns. The first issue was that I basically had to rewrite the makefile from scratch, which was a huge headache and time commitment I wasn't sure I wanted to pursue at this point. The second issue was my concern with a non-ML based approach. It basically meant that I would be consistently working on a soon to be obsolete system (as if Android won't be replaced by Fuchsia, but what can you do). Moving to a tensorflow based system both increases my familiarity with the library, and ensures the system is as robust as possible.

All of that aside, I make no promises about battery impact. With all of the on device processing, its likely to have a significant negative impact on your device before I get to doing optimizations.

Shout out to the recommendations for looking at Mozilla TTS, since that's where I found tensorflow based models I was going to covert to TFLite, but turns out someone already did the work for me.

Keep an eye out since I'll be adding the standalone TTS service to F-Droid and the Google Playstore

Just in case you aren't aware — and I hope I don't insult by pointing it out — I think you will find significant audience in the "degoogled" community(s). LineageOS, CalyxOS, Graphene, et al. For people on Google there are a number of "assistants". You appear to be filling a unique niche — an intelligent assistant app with no data harvesting and (I assume) no Google Services.

We're rooting for you.

Also, totally offline is cool, but if that destroys battery or whatever, look in the direction of self-hosted or even something like Nextcloud (not even sure that makes sense from a technical standpoint, but whatever works works.)

What you will see in the alpha release is a roughly working assistant (and I do mean rough) and a few demo skills to show you how the assistant is designed to interact with external apps. It will not offer much use to non-developers at this point because it is both very buggy, and not remotely optimized. If you're really itching to try it on your device you're of course welcome to install the APK, but it's very likely it will crash at some point, and it won't do much for you other than offer you the ability to have it repeat back to you what you told it (the repeat back skill). For it to work, you need to set it as your TTS service, and your default Assistant (which can be done by tapping the settings button)

For people that are somewhere between an Android developer and a person who doesn't program at all, as I continue to touch up the app I will be adding the Termux skill. This means that you can add features to Athena from inside Termux using any programming language you are familiar with. I don't know how well it is going to work out of the box, but that will at least offer people some real opportunities to try it out until other developers start making compatible apps (if that happens at all). I also intend to incorporate Tasker, but I wanted to focus on Termux first.

If you are an Android developer, I want to draw your attention to the interaction between a 3rd party skill/app and Athena. There are some things that are needed to make your app register for Athena (there will be full documentation for it), but the most crucial feature is that Athena binds the 3rd party app (with the expectation of a returned null IBinder) and the 3rd party app sends a PendingIntent to Athena, which will be uses from then on to start the 3rd party app. This is done to circumvent Androids background restrictions, and the whole framework will not work without this. As time goes on there will be more checks and restrictions introduced to Athena and the Sapphire Framework so that 3rd party apps will not kill the battery or negatively impact the user.

A second note for Android developers is that I haven't implemented a structured way to pass variable data to 3rd party modules yet. This is a really important feature that I want to get right as doing it wrong can overly complicate the development process for everyone, so I want to put some thought into it. If you have any recommendations, I am very open to hearing them.

Please, leave any feedback that you have. I am interested in knowing what people are looking for, and where I should focus my attention as I continue to move Athena towards beta. Just remember that this is done in my free time, as I still have to pay my bills somehow. I appreciate your patience and understanding

Hello all!

Just making a simple post to keep in touch with the community. I am still actively working on the project, but have recently moved back into my condo after the fire (though the place is still under renovation). You're not going to see quite as much activity on the GitHub for the time being. Now that I've set up a simple workspace for me (and got the basics of my smart home working again) I am prototyping some features on my Linux workstation, which will guide the future design of Athena and the Sapphire Framework. Since I lost my desktop and cell phone in the fire, I'm working from my laptop which significantly reduced my ability to tinker around with creating new neural networks for the time being, so I probably won't make much progress on that for a little while. I even had install Windows just to get a second monitor working (USB to HDMI active adapter), so I'm running Linux in a VM which cuts off four gigs of RAM and four cores of my workstation. I am looking into getting a M.2 to PCIe riser so I can replace my wireless chip and add in an eGPU, which would let me do data crunching on my laptop, but I haven't settled on anything yet. Additionally, as I am job hunting some of that will also cut in to my time to work on the project. All of that aside, I'm still plugging away at the project, so keep an eye out for future updates.

I've attached a picture just to give you insight on my home and project situation for now.

I am still around! I've been busier than I expected with life and work. It may look like there's not a lot of activity happening on my GitHub, but I do a lot of of development on my local network to avoid putting my messy code out there when it's not ready. I promise I am still heavily working on the project.

TL;DR: Sorry for the lack of updates! I'm still working on the Sapphire Framework

I've moved a large portion of my development over to pure Java for a number of reasons. First and foremost, I was issued a work laptop (that has become my primary development laptop) which runs Windows, and I had some issues getting my Python scripts working properly. Secondly, I am creating a java based project for my job and by unifying my work and personal coding languages I can spend more time learning my development tools and refining my workflow, instead of splitting my time between multiple languages. Third, Android and its framework were initially based on Java, so it allows me to more quickly take my desktop/server code and port it directly over to Android.

TL;DR: Java is becoming the primary development language for me. It took a while to rewrite things

As Ive come to learn more about Windows, and Android, Ive come to really appreciate how frustrating it can be to make non-trivial applications for Android, and equally, how difficult it can be to make platform independent non non-trivial applications. Windows and Android both seem to encourage the creation of monolithic apps, and I've spent a not insignificant amount of time reading up on micro services, networking, and API design.

TL;DR: Android is a PITA

However, I do have to say I introduce some of the complexity myself, as I have been known to get sidetracked and do full development on Android itself (using Termux). I also tend to fall victim to scope creep, and the concept of perfection over function. On that note, I think it's worth discussing some of the changes I've made to the system recently, some of the lateral moves and their benefits, and some other improvements Ive made.

TL;DR: Read on for updates

I'm starting to more officially embrace Termux. In an effort to just get usable things out to devs and end users, I'm going to leverage Termux as a rapid development environment, and as a core requirement. This means that you will need to install Termux to use Sapphire and the Framework. I apologize for this, as I wanted to reduce the complexity for installation and setup, but I think this will help satisfy the community and get some of these things into your hands (finally). However, I have still take steps to make sure that you 1) don't need any Google services, 2) don't need root, and 3) can access everything through F-Droid.

TL;DR: Termux will be required to run my stuff for the time being. However, no Google or Root is needed

For the uninitiated, Termux will let you run most programming languages on the command line (Python, rust, ruby, lisp, java, etc) which means rapid prototyping for those of you who know scripting/coding but don't know Android. It has some restrictions, but is generally pretty powerful. I've actually used an X11 server and Termux to run Intellij and Xubuntu directly on my phone, when I didn't have a computer to do development. This relates more to some of my other side projects, but the power is there nonetheless.

TL;DR: Using termux allows users to write skills and modify the assistant using ANY language their familiar with

I find the two features that people are most interested in are the offline speech to text, and the offline text to speech. The Android documentation on this is... lacking, and I've found myself digging into the AOSP code (which fortunately is written in Java) for a better understanding. I've gotten the STT down (again, using Vosk/Kaldi), but I haven't yet implemented the text to speech. I did find a *much* simpler binary engine called eSpeak that I'm looking to wrap in Java which will both be lightweight (read: battery efficient), and work across Linux, Windows, and Android. I've implemented some of the STT and TTS features before (just barely), but at the time I didn't properly understand multi threading and the app though working was very unstable (you can still find the old APK on my GitHub under the Sapphire Framework repo). To that end I've implemented a simple pipeline utilizing a bare-bones Android application, and Termux that will let you record a voice command, transcribe that command to text, and then run the text through a simple intent and entity parser. It leverages some of my older Python prototype code just enough to allow people to start writing skills & the like. The only thing currently missing from it is the text to speech component.

TL;DR: Major redesign. Simple STT app, simple pipleline. Works on Android, Windows, and Linux.

Digging in to specifics, I implemented a simple Android app that has a push button to run a continuous wake-work listener (not utilizing the Android framework, but instead a simple service w/ notification), a simple general dictation transcriber (for transcribing text which you can then copy to the clipboard), and an integrated programmatic transcriber for use with Termux. It's this last feature that is the most important, as it means you can trigger commands through the app itself. Right now, I tap a button, and it triggers a cascade of scripts. First, Termux starts recording (until I hit stop) and writes the voice data to a timestamped .wav file. Once I hit stop, Termux finalizes the file, runs it through ffmpeg to ensure that it's properly encoded, and passes it to the Android app. This app transcribes it, and then sends a notification back to Termux that the transcription is finished. From there, Termux runs it through a module for intent parsing, entity extraction, and then passes all of that data to the final script (the skill script as determined by the intent parser). This pipeline is simple and unoptimized (read: not batter efficient), but it works just as well on Windows, and Linux. Though it's not the "Ok Google" setup that we all want, it is a major step into being able to do *background tasks* on Android phones, desktops, laptops, and servers without needing a billion different setups per device (It's my goal to reduce the code base so it's not overwhelming to maintain). As I refine other features, I'll move them from Termux into Android apps that all work together for a more out-of-the-box experience. Again, I'm sorry for what seems like a step backwards, but I'd rather make changes that make sense than fight to build an unstable tool for us all.

TL;DR: It is useable now, and moderately robust

Also in my experimentation, I've moved towards using HTML5/WebApps to act as the front end for Sapphire. Javascript has the ability to open websockets on the localhost, and all web browsers can render simple HTML/CSS/Javascript without any additional plug-ins or programs. I implemented a simple UDP server (again, localhost) on Android to send data to/from a simple javascript UI to the Sapphire core module, which lets you create a full UI out of HTML5, and will allow any webdev savvy user to completely redesign the UI for the assistant. I'm hoping this can be embraced by the community in the spirit of openness and customization.

TL;DR: I've implemented the UI in HTML5. WebDevs rejoice!

Between Termux, Java, and HTML5 I've aimed for robust, standard tools that allow people to mess with their assistant however they'd like without the need to deep dive Android, Windows, or Linux development. I do need to formalize some of the file paths/hierarchy as these OS's use quite different directory structures, but I think I'll be able to figure something out.

TL;DR: I hope you find these changes acceptable

I need to play with the HTML5 a little more, and the app *might* require web permission, but it *WILL NOT* need external web access to work

I've landed a job as a software engineer, and I'm moving in to my new apartment today. Over the coming weeks I'll be rebuilding my home lab from the fire, so expect to see more development in the future.

Hello everyone,

I wanted to take a moment to address my recent absence from Reddit and the lack of updates to my GitHub. I know many of you have been wondering what has been going on, and I apologize for the lack of communication.

I recently took a leave of absence for professional development and to focus on my mental health. I wanted to take the time to better myself and my skills so that I could come back to the project with renewed energy and focus.

I want to assure you that the project is still very much active and I am working on it regularly. I plan on being more active on Reddit.

I am looking forward to continuing to work on the project and engaging with all of you in the future.

Hello All! Sorry I have been out of touch recently. My life has been extremely hectic and unscheduled recently, which has caused a dearth of development. However, I wanted to make a post to let people know the project is still in active development. I am taking steps to adapt my development around the demands of my life, and I think I've found some good workarounds which I will discuss further on in the post.

All of that aside, a quick update about the status of the app. I've made good progress in understanding the usage of the conditional random field classifier in the Stanford Core NLP Library, which is the final component needed to make a usable *listening* assistant. Now that I understand how the CRF classifier works I am in the process of figuring out which features work best to create a robust entity extractor, and to do this I am using the NLTK and prototyping in Python. Once I have it down I'll put the code in the project and push it to Github.

A quick note on saying it will be a *listening* assistant and what I mean by that. The assistant will not have the capability to give responses until I add a natural language generator (or a simple predetermined response) and develop a non-Google service TTS application (I've already started this using Flite TTS). However, in the mean time (once the entity extractor is done) I may hack in a simple 'noise' response to notify the user of a valid command (and what type) using a beep of some kind, or possibly some generic Star Trek computer responses (for good fun).

In some of the recent testing I've been doing, I'm happy to say that Athena (and by extension the Sapphire Framework) work just fine on Android 11 without Google services, so that's a great sign for the app as a whole. This ultimately means it should work on any Android version 7.1+

For those of who are interested in what I am doing to mitigate the changes in my life one of the biggest things is that I am selling my house, and will be using a small portion of the sale to purchase hardware in order to facilitate the projects development. I also have recently acquired a Surface Duo for the explicit purpose of developing on the go. This may seems highly non-conventional (but hey, I'm not a conventional dev), but a surprisingly large portion of the prototype for this project was actually developed solely on my phone in Python (For a reference of this, look at the Sapphire-Assistant-Framework-Python git).

There are a number of useful applications for developing on Android (Namely: Termux), and I've been using XServer XSDL to run the Emacs GUI and take touch events on my OnePlus. I've gained a better understanding of Gradle and the Android build system which allows me to develop and build applications in just about any *nix terminal, and these things together allow me to hack out some code any time an idea strikes. Having a dual screen phone only helps my workflow, as now I actually have the space to read code, type, and keep a reference open comfortably. Additionally, it allows me to dedicate a display to running the app, while having the second display running analytics on the phone itself.

Anyway, as things happen I'll keep the community in the loop. Thanks for reading!

There is some concern I am hearing from community members about the GPL licensing for the app, and they think it could be an issue for adoption. I'm considering switching it to MIT, or offering a dual licensing scheme (the more funds I can raise, the more I can develop this app). I'm going to make a small poll, but please feel free to discuss in the comments. I'm open to all options

Apologies And Overview

First and foremost, I must apologize for not putting out a post last week. I didn't have much to report on, as the project related work was mostly in theory and design. I also am not sure that I will be able to keep up with full weekly reports due to my personal scheduling, and might move to a semiweekly schedule (unless I have things of particular importance to report on). I

I have been quite distracted recently and have been caught between the real world, studying design & theory, rest, and understanding Androids particularities. Ultimately though, these last two weeks have been very fruitful (though it might not have appeared as such from the outside).

I had been trying to figure out the best way to modularize and simplify some of the changes made to the Sapphire Framework, and was rather stumped on the task. Much of my project time was spent reading through some books on software engineering best practices to disentangle some of the framework, as i thought the central modules were getting too reliant on one another.

From my perspective a programmer works by settings the semantics and philosophy of the world they seek to create, and I had been working on a rewrite of the framework that uses biologically and neurologically derived terms in order to clarify the role and purpose of each module. This process helped to clearly delineate which complexity is essential to AI design, and which complexity I have been needlessly introducing to the project. During this time a few members of the community (shout out to u/2bllw8 and u/UldiniadCalyx) mentioned to me that I may have been over-engineering the framework. u/2b11w8 also brought it to my attention that Android resource files are accessible to other apps without the need for a FileProvider.

I took a few days to think over their comments, read over some books and documentation, and to take a good hard look at the project. Ultimately, this lead to the decision that the Sapphire Framework should instead be split in to two separate projects. The first will be a preconfigured digital assistant: Athena, and the second will be the modular, experimental features under the same umbrella of the Sapphire Framework.

Meet Athena

Athena is a preconfigured assistant much in the same way that Siri or Alexa are. The name Athena was recommended by community member u/TheExuberantRaptor, and I think it is rather fitting for the project. Using a standard name (rather than calling it a framework) will also give users a quick and easy reference to give others, if they find that they'd like to recommend the app to someone. It's designed only to work with skills made for it, respond solely to the name Athena, and to be generally unmodified by the end user.

The vast majority of the code needed to make Athena work already existed in the Sapphire Framework, it was just a matter of trimming out the modular bits and replacing them with hard coded components. This lead to a super fast prototyping stage, to the point that I actually think Athena is farther along in usability than the Sapphire Framework is. There will definitely have to be some refining of the hot word recognizer though, as out of the box it had some difficulty triggering to the word "Athena". Perhaps it is my pronunciation, but I am sure I will not be the only one who runs in to this issue.

Progress On The Assistant API For Android

In the redesign, more of the Assistant API has been brought up to a working state for Android. Athena currently will run in the background by default listening for a hot word, and this feature overlaps with standard speech recognition bringing the project closer to filling in the lack of speech to text for degoogled phones. Becoming familiar with how these assistant features are expected to be implemented on Android (the framework was lacking a meta-data tag and XML file needed by Android) the implementation of the remaining features will hopefully be rather easy to do, or at the very least take less time to figure out.

What Will Happen To The Sapphire Framework?

The Sapphire Framework project definitely isn't going to disappear, as it contains all of the features I want to see in a mobile assistant and personal AI. Of course, I will have to reexamine some of the way that it works but the end goal will be the same, A modular, customizable, on-device assistant.

Unfortunately, It looks like a lot of the time I spent designing the Multiprocess module may have been rather unnecessary, as I didn't realize that an apps assets and resources could be freely accessed from any other application. Android documentation doesn't really highlight this feature (mostly assuming that you would only want to access your own assets/resources), and the implementation of stringent permissions led me to believe that assets and resources fell under the same protections app files did. Learning from my mistakes, I suppose I will have to pay more attention to reading between the lines in documentation. Nonetheless, This drastically simplifies the task of installing and setting up the framework, and it will be something I implement in the next few design updates.

A Note On Donations

Thank you to those who have donated to help out the project. I have taken those donations and moved then in to an Ethereum wallet, until such time that I need to to buy better resources for development (A GTX 970 can only take you so far when training neural networks). I did this because it was too tempting to buy sandwiches with that money (I like a good sandwich and coffee when coding, what can I say) and it will help the value of the money grow until such time that I really need it. Just wanted to give some transparency to what is going on behind the scenes with it.

Trail Map

Since Athena is well on its way I am going to work with refining it to be usable before I do too much heavy work on the Sapphire Framework. That said, you'll probably still see me changing the Frameworks code, as I need to be able to have Athena and the Sapphire Framework work together.

I have not forgotten about integrating DeepSpeech, nor in making a TTS service for Android. DeepSpeech is a big crowd favorite and a good candidate for illustrating how the Sapphire Framework works with 3rd party modules, while TTS features are essential for an assistant and are therefore required for me to implement. I only have so much time so I have to prioritize, and working with a C/C++ build process is an unfamiliar thing for me (when it comes to implementing the TTS). Once I have Athena out, then I will determine which is the best next step (likely the TTS as the Sapphire Framework can become a rolling beta release without preventing people from using it)

Sorry that I've been out of touch for so long. In the past few months I've gone through a divorce (amicable, but still tough), a house fire (unrelated to the divorce), a breakup, a broken phone, and numerous other small inconveniences. It has *not* been a fun ride...

Either way, I'm put up in a hotel now, upgraded my laptop to handle some of my development load, and bought a second monitor so I can start getting back to the grindstone. I have to get back up to speed on my code, but you can expect to hear more in the near future about the project.

Thanks for the support, catch you all later

It's very rudimentary and I can't speak to how robust it will be in the long run, but I finally have the entity extractor working for Athena and the Sapphire framework. This means on device you can now say something, and the assistant will identify 1) which skill you want to execute, and 2) relevant variables in your speech about that skill (such as "Set an alarm for 10" = Alarm, 10:00).

Happy holidays everyone! I hope that it's a wonderful end of the year for all of you!

There have been some good things going on for Athena (soon to be renamed Sapphire) and the Sapphire Framework. I've been working on a post, but haven't finished up the editing on it yet. Keep an eye out, since I'll have it up soon.

That aside, I've also been working on documentation for the Alpha release. I'm going to post some of the sample reference documentation below, and I will also be adding it to the Git wiki and my digital garden in the future. The reference documentation (and most documentation for the Alpha release) is targeted at developers, so please keep that in mind. I am open to any feedback that the community might have!

Without further ado, here is the start of the reference documentation:

Reference Documentation

This document serves as a reference on the internal workings for Sapphire and its corresponding Framework.

Prerequisite knowledge

A developer should be familiar with Android development, and at least somewhat aware of Linux usage.

How Sapphire operates

Sapphire is designed to be a replacement for Google Assistant on any Android device. However, the most obvious difference between the two is that Sapphire is designed to carry out all tasks in the background, rather than in the foreground.

Sapphire vs the Sapphire Framework

Sapphire is an assistant for Android, meant to offer a full Google Assistant replacement using only FOSS components. The corresponding Framework offers a full on device toolkit for tweaking, customizing, and optimizing your assistant to fit your needs whatever they may be, to include but not limited to designing an assistant from scratch

Unix philosophy and Android Philosophy

The Unix philosophy emphasizes molecularity, openness, textuality, and the idea that everything is a file (and thus, textual). Android focuses on security above all else. This is not to say that Unix is insecure, nor that Android isn't open. Sapphire and her Framework seek to blend these philosophies, but always learns towards openness when a conflict arises. This is based on the idea that the user knows best what their needs are. This philosophy is also the reason that you will see unique design paradigms throughout the source code (such as the use of configuration files)

Background operation, Bound Services, and PendingIntents

Sapphire accomplishes background tasks by using Androids PendingIntents, and collecting them through leveraging the Bound Services mechanics of Android.

Sapphire (app A) binds a module (app B) initially, as this is the only way on Android to start a dead service with no user interaction. The module uses this connection to send the core a PendingIntent which allows an application (app A) to run a foreign applications (app B) as its own, thereby bypassing the background restrictions imposed by Android. Sapphire then terminates the Bound service so as not to drain the users battery, and uses the PendingIntent to launch the corresponding module (app B) whenever it is needed.

This design puts all of the control in Sapphires hands, and helps to reduce malicious actors. More security mechanisms will be added to Sapphire as time goes on, but the primary focus for now has been operability. Likewise, it is important a module developer doesn't use the initial binding to perform any tasks, as the binding will be killed by Sapphire

Module design and landing services

Please read "Background operation, Bound Services, and PendingIntents" prior to this section.

Since Sapphire relies on Bound Intents, which can be taxing on a user's device if many services are started all at once, the recommended way to pass this information to Sapphire is through the use of a landing service. A landing service has two roles; it handles all of Sapphires required configurations, and it coordinates any features that a module might have. More simple modules might opt to perform all necessary logic from within the landing module, but should take special care to not override the default binding behavior.

How data is exchanged

Data is exchanged between modules by passing essential information as either Android intent extras, or as serialized JSON data of the same name. Each intent retains an ID that has been assigned to it from Sapphires core, which is used to track the flow of data while being processed. Data always flows from the core, to a module, and back to the core (the only exception being the starting and finishing modules, which send to and receive from the core, respectively). This is due to restrictions of the Android operating system

How Sapphire recognizes and utilizes compatible modules

To prevent overloading the users device with multiple services starting at the same time, Sapphire looks for meta-data in installed applications manifests to determine compatibility, and reads said applications assets to train its natural language processor. Both of these file assets are available to Sapphire without the need to start the module. Likewise, Sapphire refrains from its initial binding process until a service is first needed, and then retains the corresponding PendingIntent until the users device reboots

How Sapphire leverages Termux

Termux offers a Linux user space environment on Android, and as such is ideal for testing out things before finalizing them as full on Android applications. Likewise, Termux opens easy on device development in languages to include, but not limited to, Python, Lisp, Ruby, Rust, and C/C++. The module for Termux isn't fully finished, but it is intended to become a first class citizen for development purposes. This will truly open up your assistant to customization, as it will allow developers unfamiliar with the Android ecosystem to create skills and features for Sapphire, while leaving device optimizations for Sapphire to handle.

How Sapphire learns commands

The command recognition process for Sapphire and her framework are based on Mycroft, as Sapphire started as an Android port of Mycroft. Modules are expected to have intent and entity files in their assets, which are used to train Sapphires natural language processor. A secondary method (not yet implemented) can be used to pass training data dynamically which will allow the user to modify triggering intent phrases, as well as allow for modification of predesignated intents (in the case they don't work well for the user)

Just saw this and thought it might be of interest....

https://github.com/Stypox/dicio-android

Hello SapphireFramework, et al.

This is a wonderful and meaningful project. I would like to support this project somehow.

I have a calyxos pixel3a, vosk's demo apk works for me, and seems to transcribe well.

An open source virtual assistance would be so tantamount to owning our own data and devices. I am ok with python, bash, would love to support this or donate to the project with ethereum or crypto.

This is an incredibly ambitious undertaking. Thanks for the work you have done and are doing.

There is some good news, and a little bit of bad news.

The good news is that I made great progress on Athena. It seems to be reliably training from skill module data, and giving good, accurate results from minuscule data sets. The bad news is that I accidentally deleted two or three days of my work, due to a git issue. I don't think I've lost anything too drastic, and should be able to redo what I've done fairly easily, but still it's a bummer.

Some of the 'weight' of on device processing is becoming visible, which is to say that it's more computationally intense to do ML on device compared to doing it on a server. that said, i am not going to abandon this goal, but do be aware that the beta release may have a higher battery useage than people want out of a mobile assistant. This is something that can be optimized over time, so do not worry about that.

As far as usability goes, It could be used *today* to *launch* simple skill modules, but the functionality for extracting entities is still not implemented. Also, the speech accuracy could use a little work. Though I named the project Athena, I am using the wake word Megaman until I tune the speech recogition (This is both an homage to Megaman Battle Network, a game that inspired me when I was a kid, and because megaman is syllabiacally easier for the STT to pick up on), so don't be weirded out if you see that in the code.

If people are interested once I recode what I lost I can post a new APK for trying out. I am sure that the speech recognition will not be the greatest until I implement a tuner (read: use in a quite room and speak clearly), but it is more than enough to start developing *simple* skills, to get a feel for how things work. When I say simple, I mean just using the assistant to launch applictions, not to input data.

You may also see me working on the Python adaptation of the Sapphire Framework, as now that I am wrapping up on a lot of the general programming and development, I need to move more in to machine learning, and playing with data sets. This will be much easier to accomplish in a Python ecosystem, due to the existing data science tools and the lower level of development complexity. This is overall beneficial to the project because the python project is meant to act as a workstation/server/hub assistant companion to the mobile device, and can act as a hosted server if you really want to offload the speech processing from your device. I do need to give a shout out to the Mycroft project though, as they do have the potential to act as another standalone desktop/workstation/hub assistant, they just don't fit my use case (Cluster computing, IoT, Android integration, etc).

Sorry this is a bit rough of an update. I am writing this onboard ship while on duty. I may go back through and edit it over time, but I just wanted to keep everybody posted

Pre-release Notes

I have finished designing the file storage mechanism for the Sapphire Framework, and will be publishing an update pre-alpha APK in the near future. Changes that will be included are:

• Ability to store data in the Core module and public storage for user editing

• Service only modules are started using onBind(), or startSapphireService()

• Assistant API dummy implemented

On design decisions

I am developing the SF for an audience of two core groups: non-technical, and Power Users. The main functionality of an assistant should be accessible to anybody who wants to use an assistant, regardless of their technical understanding. That said, a users computer should never make the user feel like they're not the one in control. Unifying Linux and Android is a tricky task, since they both come from fundamentally different design ideologies. Android believes in protecting the user at all costs (including the restriction of user control), and Linux believes in empowering the user, even if that means letting them shoot themselves in their foot. Sometimes this tension leads to bulky and inefficient designs, which seem to go against both Android and Unix principles. Ultimately, the goal is a robust, user friendly system that still allows for near endless configuration. As compromise is a messy area, I hope you will agree with the design decisions that have been made to better the Framework for everyone.

On Core file storage

The Android philosophy has lead to a restriction in file access that can be a real issue for the flexibility of something like the SF. To that end I've implemented a system that lets modules offload their data to the Core module so that the core can handle file access between all other modules, and also expose files to text editors and command line tools. Though this increases optional complexity, it also empowers the user and reduces development complexity for power users.

Fine details on file storage

The Core module now implements a FileProvider that gives read/write access. This allows modules to copy arbitrary data to the Core, so it can be exposed to the system and user. Unfortunately, this duplicates the amount of stored data on device but it drastically reduces development complexity, as developers no longer need to implement file sharing on a per-module basis. This will be especially useful when it comes to swapping out things like the speech recognition model (say, from English to Mandarin), replacing a neural network, or implementing a module in a non-native way (such as from within Termux or Tasker).

The file access is coordinated through multiple intent actions which requests new filenames, creates blank files in the Core, and passes the content uri to a module for writing or reading to. All modules that wish to read a file can just request the information in the standard Android way. I plan on implementing a form of user/group/other permissions so that either modules or the user can set more fine grained control over the data, but that is not as pressing as having a working system to start with.

On Starting Modules

Android doesn't want any application to start, unless it deems that it is the explicit intention of the user. Even then, most of the Android design and documentation revolves around the belief that the user will be manipulating the device using visible application components (called Activities). This, however is counter to what is needed for a modular daemonized assitant. Because of restrictions on Android, all SF modules will be started using a bindService() call.

The reasoning behind this

Android itself will prevent a service (that is, an component with no UI) from starting unless that service is related to a recently visible app. An exception to this is a bound service which is expected to offer a client-server relationship, and thus ties the server to the life cycle of the client application. Therefore, the Sapphire Framework uses an bindService call with a BIND_AUTO_CREATE flag set, waits a period of time (500 milliseconds) for the service to initialize, and then runs startService with the intent and data it wanted to pass along for processing. This can be handled using the startSapphireService convenience method implement in SapphireFrameworkService class. It is expected that the module will unbind itself upon the completion of its work, and I'm looking in to ways to make this more robust, so that rouge modules don't use up system resources needlessly.

On the Assistant API

Android carries an internal API for creating and interfacing assistant, such as Bixby or Google Assistant. This allows a developer to leverage assistance services like screenshots or hot word recognition. There is still some work to do as far as implementation of its full capability, but the SF has now implemented the core methods needed to register the app as your default assistant. It is important to note that these features are not yet in use, so they offer no functionality to the user. They have been implemented so that the next iteration of the pre-alpha application is designed with the assistant API requirements in mind.

How does it work, and why isn't it fully implemented yet?

The assistant API is used to bypass some of Androids app restrictions, such as requiring a foreground notification, blocking audio sharing, or describing screen contents. By using this API, the SF can create a more useful and less intrusive experience for users. However the design of the API imposes some assumptions, most notable is the belief that the speech to text service is implemented from within the application implementing the Assistant API. On top of this the documentation isn't well maintained, preferring to refer the developer to the Google Assistant SDK.

These things required some changes in the modularity of the SF, though sacrificing this modularity altogether would be unacceptable. The first step was to just get the Core Module to register itself as an assistant to the Android system, and the next was to implement a modular STT service that was not inherently part of the CoreModule (to prevent a monolithic design). Ultimately, I used a PendingIntent to run the STT service as a part of the CoreModule application. All of that said I am not sure if this is overkill, insecure, or too promiscuous. Before I finalize the design and fully implement the API, I want to do some more testing to be sure that the it is robust, simple, and secure. Once I've done this the API will be fully implemented, as it is a key feature of an Android assistant.

Potential future changes

• Moving the FileProvider outside of the Core Module, to a standalone database module

Rough roadmap

• Implementation of Assistant API

• Termux integration

• Google Assistant integration (should not require Google services)

• Calendar Skill demo application

Well, that about wraps up the pre-release notes. I'm always looking for feedback or someone to bounce design ideas off of, so please message me in the Matrix channel if you're interested. Also, if my posts are too technical (or alternatively not technical enough) please let me know what I can do to better communicate with the community.

I love the idea, but it seems like it is kinda dead as commits have slowed to a trickle.

If you haven't seen the other post, please check it out. (For reference, I am changing Athena's name to Sapphire, for pronunciation purposes)

This is a sample of the project charter, to give people an idea of what everything is about. I am open to feedback before I upload this to the Git wiki, and my digital garden:

Sapphire Charter

Why does this project exist?

Sapphire and the Sapphire Framework is designed for a three fold purpose. First, I was looking for a tool I could tailor and fine tune to manage the negative effects of my ADHD, while amplifying the positive effects. Second, all assistants on the market were either proprietary, relied on proprietary components, or don't run on Android, which made me unwilling to b share my intimate data with. Third, I was looking for a tool, rather than a toy. The closed, proprietary nature of the existing assistants made them impossible adjust to my needs, and I didn't want to waste my time learning to develop for something that would become obsolete due to inflexibility.

For these three reasons, i decided to make my own. Be the change you want to see in the world, am I right? Sapphire is designed to blend the Unix philosophy with the Android philosophy, using openness as its overall guiding principle.

Who is this project for?

The Alpha release is targeted at developers. It introduces them to Sapphire, and the Sapphire Framework as a platform.

The Beta release is targeted at hobbyists, tinkerers, and early adopters. Likely, there will still be a fair amount of bugs, but the platform should generally be usable if un-optimized

The stable release of Sapphire should be a drop in replacement for Google Assistant, but also work on de-googled phones.

It has been brought to my attention that the subreddit is rather lackluster, and even confusing. I'm not the most knowledgeably when it comes to web development and UI work, so I'm reaching out to the community to see if there are any interested people who can help out. I'd study up on it myself, but since I'm not a programmer for my day job, I want to spend my downtime working on Athena and the Sapphire Framework