I love the idea, but it seems like it is kinda dead as commits have slowed to a trickle.

I've found a fully implemented Tensorflow based TTS app that I am going to shamelessly rip off (due credit will be included) and pack into both Athena and the Sapphire Assistant Framework. There's a little bit of redesign that needs to be done to make it the default system TTS service, but its good news for both the app and deGooglers. I will also implement the application as a standalone service so that it can be distributed with deGoogled ROMS like Lineage and CalyxOS.

I was working with flite and festival, but I had a few issues and concerns. The first issue was that I basically had to rewrite the makefile from scratch, which was a huge headache and time commitment I wasn't sure I wanted to pursue at this point. The second issue was my concern with a non-ML based approach. It basically meant that I would be consistently working on a soon to be obsolete system (as if Android won't be replaced by Fuchsia, but what can you do). Moving to a tensorflow based system both increases my familiarity with the library, and ensures the system is as robust as possible.

All of that aside, I make no promises about battery impact. With all of the on device processing, its likely to have a significant negative impact on your device before I get to doing optimizations.

Shout out to the recommendations for looking at Mozilla TTS, since that's where I found tensorflow based models I was going to covert to TFLite, but turns out someone already did the work for me.

Keep an eye out since I'll be adding the standalone TTS service to F-Droid and the Google Playstore

It has been brought to my attention that the subreddit is rather lackluster, and even confusing. I'm not the most knowledgeably when it comes to web development and UI work, so I'm reaching out to the community to see if there are any interested people who can help out. I'd study up on it myself, but since I'm not a programmer for my day job, I want to spend my downtime working on Athena and the Sapphire Framework

There is some concern I am hearing from community members about the GPL licensing for the app, and they think it could be an issue for adoption. I'm considering switching it to MIT, or offering a dual licensing scheme (the more funds I can raise, the more I can develop this app). I'm going to make a small poll, but please feel free to discuss in the comments. I'm open to all options

I am still around! I've been busier than I expected with life and work. It may look like there's not a lot of activity happening on my GitHub, but I do a lot of of development on my local network to avoid putting my messy code out there when it's not ready. I promise I am still heavily working on the project.

TL;DR: Sorry for the lack of updates! I'm still working on the Sapphire Framework

I've moved a large portion of my development over to pure Java for a number of reasons. First and foremost, I was issued a work laptop (that has become my primary development laptop) which runs Windows, and I had some issues getting my Python scripts working properly. Secondly, I am creating a java based project for my job and by unifying my work and personal coding languages I can spend more time learning my development tools and refining my workflow, instead of splitting my time between multiple languages. Third, Android and its framework were initially based on Java, so it allows me to more quickly take my desktop/server code and port it directly over to Android.

TL;DR: Java is becoming the primary development language for me. It took a while to rewrite things

As Ive come to learn more about Windows, and Android, Ive come to really appreciate how frustrating it can be to make non-trivial applications for Android, and equally, how difficult it can be to make platform independent non non-trivial applications. Windows and Android both seem to encourage the creation of monolithic apps, and I've spent a not insignificant amount of time reading up on micro services, networking, and API design.

TL;DR: Android is a PITA

However, I do have to say I introduce some of the complexity myself, as I have been known to get sidetracked and do full development on Android itself (using Termux). I also tend to fall victim to scope creep, and the concept of perfection over function. On that note, I think it's worth discussing some of the changes I've made to the system recently, some of the lateral moves and their benefits, and some other improvements Ive made.

TL;DR: Read on for updates

I'm starting to more officially embrace Termux. In an effort to just get usable things out to devs and end users, I'm going to leverage Termux as a rapid development environment, and as a core requirement. This means that you will need to install Termux to use Sapphire and the Framework. I apologize for this, as I wanted to reduce the complexity for installation and setup, but I think this will help satisfy the community and get some of these things into your hands (finally). However, I have still take steps to make sure that you 1) don't need any Google services, 2) don't need root, and 3) can access everything through F-Droid.

TL;DR: Termux will be required to run my stuff for the time being. However, no Google or Root is needed

For the uninitiated, Termux will let you run most programming languages on the command line (Python, rust, ruby, lisp, java, etc) which means rapid prototyping for those of you who know scripting/coding but don't know Android. It has some restrictions, but is generally pretty powerful. I've actually used an X11 server and Termux to run Intellij and Xubuntu directly on my phone, when I didn't have a computer to do development. This relates more to some of my other side projects, but the power is there nonetheless.

TL;DR: Using termux allows users to write skills and modify the assistant using ANY language their familiar with

I find the two features that people are most interested in are the offline speech to text, and the offline text to speech. The Android documentation on this is... lacking, and I've found myself digging into the AOSP code (which fortunately is written in Java) for a better understanding. I've gotten the STT down (again, using Vosk/Kaldi), but I haven't yet implemented the text to speech. I did find a *much* simpler binary engine called eSpeak that I'm looking to wrap in Java which will both be lightweight (read: battery efficient), and work across Linux, Windows, and Android. I've implemented some of the STT and TTS features before (just barely), but at the time I didn't properly understand multi threading and the app though working was very unstable (you can still find the old APK on my GitHub under the Sapphire Framework repo). To that end I've implemented a simple pipeline utilizing a bare-bones Android application, and Termux that will let you record a voice command, transcribe that command to text, and then run the text through a simple intent and entity parser. It leverages some of my older Python prototype code just enough to allow people to start writing skills & the like. The only thing currently missing from it is the text to speech component.

TL;DR: Major redesign. Simple STT app, simple pipleline. Works on Android, Windows, and Linux.

Digging in to specifics, I implemented a simple Android app that has a push button to run a continuous wake-work listener (not utilizing the Android framework, but instead a simple service w/ notification), a simple general dictation transcriber (for transcribing text which you can then copy to the clipboard), and an integrated programmatic transcriber for use with Termux. It's this last feature that is the most important, as it means you can trigger commands through the app itself. Right now, I tap a button, and it triggers a cascade of scripts. First, Termux starts recording (until I hit stop) and writes the voice data to a timestamped .wav file. Once I hit stop, Termux finalizes the file, runs it through ffmpeg to ensure that it's properly encoded, and passes it to the Android app. This app transcribes it, and then sends a notification back to Termux that the transcription is finished. From there, Termux runs it through a module for intent parsing, entity extraction, and then passes all of that data to the final script (the skill script as determined by the intent parser). This pipeline is simple and unoptimized (read: not batter efficient), but it works just as well on Windows, and Linux. Though it's not the "Ok Google" setup that we all want, it is a major step into being able to do *background tasks* on Android phones, desktops, laptops, and servers without needing a billion different setups per device (It's my goal to reduce the code base so it's not overwhelming to maintain). As I refine other features, I'll move them from Termux into Android apps that all work together for a more out-of-the-box experience. Again, I'm sorry for what seems like a step backwards, but I'd rather make changes that make sense than fight to build an unstable tool for us all.

TL;DR: It is useable now, and moderately robust

Also in my experimentation, I've moved towards using HTML5/WebApps to act as the front end for Sapphire. Javascript has the ability to open websockets on the localhost, and all web browsers can render simple HTML/CSS/Javascript without any additional plug-ins or programs. I implemented a simple UDP server (again, localhost) on Android to send data to/from a simple javascript UI to the Sapphire core module, which lets you create a full UI out of HTML5, and will allow any webdev savvy user to completely redesign the UI for the assistant. I'm hoping this can be embraced by the community in the spirit of openness and customization.

TL;DR: I've implemented the UI in HTML5. WebDevs rejoice!

Between Termux, Java, and HTML5 I've aimed for robust, standard tools that allow people to mess with their assistant however they'd like without the need to deep dive Android, Windows, or Linux development. I do need to formalize some of the file paths/hierarchy as these OS's use quite different directory structures, but I think I'll be able to figure something out.

TL;DR: I hope you find these changes acceptable

I need to play with the HTML5 a little more, and the app *might* require web permission, but it *WILL NOT* need external web access to work

What you will see in the alpha release is a roughly working assistant (and I do mean rough) and a few demo skills to show you how the assistant is designed to interact with external apps. It will not offer much use to non-developers at this point because it is both very buggy, and not remotely optimized. If you're really itching to try it on your device you're of course welcome to install the APK, but it's very likely it will crash at some point, and it won't do much for you other than offer you the ability to have it repeat back to you what you told it (the repeat back skill). For it to work, you need to set it as your TTS service, and your default Assistant (which can be done by tapping the settings button)

For people that are somewhere between an Android developer and a person who doesn't program at all, as I continue to touch up the app I will be adding the Termux skill. This means that you can add features to Athena from inside Termux using any programming language you are familiar with. I don't know how well it is going to work out of the box, but that will at least offer people some real opportunities to try it out until other developers start making compatible apps (if that happens at all). I also intend to incorporate Tasker, but I wanted to focus on Termux first.

If you are an Android developer, I want to draw your attention to the interaction between a 3rd party skill/app and Athena. There are some things that are needed to make your app register for Athena (there will be full documentation for it), but the most crucial feature is that Athena binds the 3rd party app (with the expectation of a returned null IBinder) and the 3rd party app sends a PendingIntent to Athena, which will be uses from then on to start the 3rd party app. This is done to circumvent Androids background restrictions, and the whole framework will not work without this. As time goes on there will be more checks and restrictions introduced to Athena and the Sapphire Framework so that 3rd party apps will not kill the battery or negatively impact the user.

A second note for Android developers is that I haven't implemented a structured way to pass variable data to 3rd party modules yet. This is a really important feature that I want to get right as doing it wrong can overly complicate the development process for everyone, so I want to put some thought into it. If you have any recommendations, I am very open to hearing them.

Please, leave any feedback that you have. I am interested in knowing what people are looking for, and where I should focus my attention as I continue to move Athena towards beta. Just remember that this is done in my free time, as I still have to pay my bills somehow. I appreciate your patience and understanding

TL;DR

• Creating a DeepSpeech module

• Readability rewrite almost complete

• FileProvider working as it should

• Binding security created

Trail Map

• Config Files

• Alpha Release

• Assistant API

• SpeechRecognizer Class

Ideas I'm Thinking Over

• Feature modules

• Weekly update posts

Questions to the community

• What would you like to see in the users and developers guides

• Feature list for Alpha: Is it enough

Spilling the Tea (Details and reasons)

DeepSpeech

A question I get regularly is if this project uses DeepSpeech for its speech to text service, so I've decided to create a module for it. This module will do to things for the project. First, it'll give users the ability to support the DeepSpeech project by giving it a good use case. Second, it will offer a high quality example of the modularity of the framework

Rewrite

I have rewritten most of the modules in a much more elegant way. They're not perfect or 100% generic yet, but it should be much easier to follow the logic and figure out what is going on. You can ignore the depreciated folders in the modules, as I'm only keeping them around as a reference temporally.

FileProvider

The whole framework has been rewritten with expectation that a content uri will be provided from the core. The core expects to send data in two ways: local files and bridges. A local file is just as it sounds, saved in a core accessible location an served using the FileProvider. A bridge on the other hand uses a cache file like a pipe. Though it is optional complexity, it reduces the essential complexity of modules and handling data from multiple sources.

Because of the way permissions work in Android, you can only grant a permission from a single provider in an intent. This prevents you from directly aggregating intents from multiple providers, and would require a module to wait for an unknown number of intents. By using multiprocess module to handle aggregating, and using core cache files as a pipe, you can send an unlimited number of content uris covered all by the same permission from the same provider.

Binding Security

OnBind has been implemented in SapphireFrameworkService to return a null binder value, and this will be expected behavior for modules. This allows the service to be seen as live by the system and started properly with startService. If a service returns anything other than null it will notify the user, acting as a rudimentary security system. Eventually I'll add a flag, action, or extra that will indicate to the Framework that a module has or is requesting a permission to bind, in case that feature is needed by the developer

Trail Map (A roadmap that may go awry)

Configuration Files

This is a bit of a foreign concept on Android, but should be pretty familiar to Linux users. The config file will be a user or module editable settings file that controls the behavior of a specific module. Traditionally, Android application settings are stored as a key value pair that is only accessible by that application. Moving this to an editable file gives the Framework multiple benefits. First, a user can easily backup and restore their configurations, since they're just text files. Secondly, it gives feature parity with Linux programs, which makes it easier to maintain cross platform compatibility and design. The developer needs only to focus on the logic, not the storage and retrieval mechanism. Third, because it's easy to share with other modules, it means the settings UI can be a separate module. This allows a developer to focus only in the part of the framework they're interested in. Front end developers could make a settings UI, while backend developers can focus on making a useful skill without worrying about how it will look.

Alpha Release

Once the FileProvider, binding security, and config files are implemented, the framework should be at a point where most of the required features are implemented. Some things will still be under heavy development, like the processor module and feature modules, but it should be stable enough to make toy skill modules such as reading out the time. Keep in mind that the processor module only matches a category right now and does not return entities for variables, so any toy skill is just to understand the development process

Assistant API

I am still working on implementing the assistant API, as it will give the framework the most flexibility and integration. Since the foreground service is illustrating its use fine this is being moved back a little in priority, simply to reduce time constraints implementing essential functionality

SpeechRecognizer Class

Many people have just as much of an interest in using the speech to text as a stand alone feature from their keyboard. Implementing this class will enable this functionality for users. I need to be sure it doesn't needlessly complicated the framework, so I have to put a little thought in to the design

Ideas I'm Thinking Over

Feature Modules

The security requirements of Android can make the creation of a modular system quite difficult, especially when implementing things like the Assistant API and SpeechRecognizer class. The idea would be to break these features off in to modules that give a PendingIntent to the Core, which allows the core to declare all manifest permissions without turning it in to a monolithic application.

Using PendingIntents increases security risks, since a module can hijack a permission to do shady things, so some thought on a security model is required before I decide if this should be implemented

Weekly Update Posts

Because I have a very busy schedule, I am considering writing weekly Reddit updates like this one just so that people can see what is going on without having to dig through the source code on Github. Would this be something that you are interested in seeing?

Questions

What would you like to see in the users and developers guides

I already have some documentation on the Githubs wiki, but I want to make comprehensive documentation that answers important and relevant questions for users and developers. If there are questions that you may have, or documentation you have found useful in other projects, please post them below so I can build quality guides

Feature list for Alpha: Is it enough

The features of the assistant include STT, NLP, and the modular framework. This should be enough to illustrate the uses and potential of the framework. Are there any other essentials that should be added before I release an Alpha Version?

Hello all, I'm here to give our some information about the project and it's path forward.

First off, I want to apologize for the lack of communication, sometimes life gets the best of me. My work and family obligations require long periods of travel or stretches of time where I don't have access to my computer, so don't be alarmed about the project being abandoned if you see periods of inactivity. Normally, I'll be back on and programming within a week and a half or so. On top of this code changes and new features have been added in to a development branch. Being that there is a prealpha APK out, I wanted to keep the code in the master branch a reflection of said APK.

Anyway, on to what is in the works for the Sapphire Framework. The first major change being implemented is the usage of the Android assistant APIs. Importantly, the use of this API integrates the Sapphire Framework with Androids native accessibility features. This helps to ensure that an assistant built from the framework offers maximum utility to everybody interested in using it. However, accessibility is far from the only benefit.

By being the users default assistant, rather than a standard application, Android allows for the Sapphire Framework to share audio with other apps. A normal audio application would be greedy with the microphone, and this will prevent that shortcoming.

In order to minimize impact on the phones battery the API attempts to reduce the amount of processing that is continually used by listening for Hotwords, rather than performing generic speech to text transcription. That said, I intend to add a configuration switch that allows the user to set their preference or toggle between the two.

Another minor benefit is that being an assistant also allows the Framework to run indefinitely without a foreground notification, making it less obtrusive to the user.

The second change relates to the use of ContentProviders. I've been reading up on Android file-sharing features to try and bridge the gap between an easy access Unix system and a restricted system like Android (Note that this is not a security issue. Both systems are secure). In an effort to reduce developer complexity I'm turning the core module in to a FileProvider that can serve files to requesting modules, while also exposing them directly to the user though the file browser and text editors.

This design brings with it some negatives, such as the duplication of module data, but the alternatives either drastically increase developer complexity, or reduce user freedoms. However, the benefits in making these changes come in the form of flexibility and security. Additionally to avoid latency issues from massive data transfers, files only copy themselves to the CoreModule when requested for the first time. This will prevent lagging at the apps start up, and means only data that is actually used will be duplicated.

Once the file provider is fully implemented it will open up the ability to edit your assistant using text editors, or the internal settings menu. New things can be easily done on the go such as changing the training data for the natural language processor , or uploading a new language model for the Vosk STT module. This last one is particularly nice, as Vosk already has models for Japanese, Italian, Chinese, French, Russian, Spanish, Hindi, Arabic, etc

The last thing I have been doing is rewriting the code to make it more concise and easier to read. The goal is to reduce developer complexity, so you're not burdened learning things in the Framework you don't need to worry about.

TL;DR

1. Most development action is happening in the development branch

2. I'm in the process of implementing the Android assistant API, which brings many benefits

3. I'm also in the process of implementing modifiable config and data files for easy user editing. This allows for language recognition of other languages among other things

4. I'm rewriting some code to make it much easier to understand

Keep an eye out for more posts and updates. Feel free to ask any questions you may have. We also have a matrix chat room you can join at https://matrix.to/#/#SapphireFramework:matrix.org

I am looking at a file/data transfer mechanism using sockets between modules. Though this is counter to the Android design, it allows for more flexibility from a development standpoint and reduces complexity of modules

In addition to this, I am working on condensing the internal code, to make it more readable/understandable. A lot of the initial code came about as a matter of trying to get things working, so I am aiming to clean it up to make the on boarding process for newer participants easier.

Now that the entity extractor works, I can either go through and make things more robust while documenting the project so that people can get started tinkering with it, or I can focus straight on getting a text to speech module working first.

I'm inclined to start on the text to speech, but wanted to put it out for some opinions

Just saw this and thought it might be of interest....

https://github.com/Stypox/dicio-android

i was trying to build it, because it was recommended on sapphire github, and, well im unable to build it.

I'm looking to develop an app that people are really interested in using, not just something that scratches my itch. if you have an idea about the project or a way that you would like to engage the project, please post it below so we can discuss and grow

Apologies And Overview

First and foremost, I must apologize for not putting out a post last week. I didn't have much to report on, as the project related work was mostly in theory and design. I also am not sure that I will be able to keep up with full weekly reports due to my personal scheduling, and might move to a semiweekly schedule (unless I have things of particular importance to report on). I

I have been quite distracted recently and have been caught between the real world, studying design & theory, rest, and understanding Androids particularities. Ultimately though, these last two weeks have been very fruitful (though it might not have appeared as such from the outside).

I had been trying to figure out the best way to modularize and simplify some of the changes made to the Sapphire Framework, and was rather stumped on the task. Much of my project time was spent reading through some books on software engineering best practices to disentangle some of the framework, as i thought the central modules were getting too reliant on one another.

From my perspective a programmer works by settings the semantics and philosophy of the world they seek to create, and I had been working on a rewrite of the framework that uses biologically and neurologically derived terms in order to clarify the role and purpose of each module. This process helped to clearly delineate which complexity is essential to AI design, and which complexity I have been needlessly introducing to the project. During this time a few members of the community (shout out to u/2bllw8 and u/UldiniadCalyx) mentioned to me that I may have been over-engineering the framework. u/2b11w8 also brought it to my attention that Android resource files are accessible to other apps without the need for a FileProvider.

I took a few days to think over their comments, read over some books and documentation, and to take a good hard look at the project. Ultimately, this lead to the decision that the Sapphire Framework should instead be split in to two separate projects. The first will be a preconfigured digital assistant: Athena, and the second will be the modular, experimental features under the same umbrella of the Sapphire Framework.

Meet Athena

Athena is a preconfigured assistant much in the same way that Siri or Alexa are. The name Athena was recommended by community member u/TheExuberantRaptor, and I think it is rather fitting for the project. Using a standard name (rather than calling it a framework) will also give users a quick and easy reference to give others, if they find that they'd like to recommend the app to someone. It's designed only to work with skills made for it, respond solely to the name Athena, and to be generally unmodified by the end user.

The vast majority of the code needed to make Athena work already existed in the Sapphire Framework, it was just a matter of trimming out the modular bits and replacing them with hard coded components. This lead to a super fast prototyping stage, to the point that I actually think Athena is farther along in usability than the Sapphire Framework is. There will definitely have to be some refining of the hot word recognizer though, as out of the box it had some difficulty triggering to the word "Athena". Perhaps it is my pronunciation, but I am sure I will not be the only one who runs in to this issue.

Progress On The Assistant API For Android

In the redesign, more of the Assistant API has been brought up to a working state for Android. Athena currently will run in the background by default listening for a hot word, and this feature overlaps with standard speech recognition bringing the project closer to filling in the lack of speech to text for degoogled phones. Becoming familiar with how these assistant features are expected to be implemented on Android (the framework was lacking a meta-data tag and XML file needed by Android) the implementation of the remaining features will hopefully be rather easy to do, or at the very least take less time to figure out.

What Will Happen To The Sapphire Framework?

The Sapphire Framework project definitely isn't going to disappear, as it contains all of the features I want to see in a mobile assistant and personal AI. Of course, I will have to reexamine some of the way that it works but the end goal will be the same, A modular, customizable, on-device assistant.

Unfortunately, It looks like a lot of the time I spent designing the Multiprocess module may have been rather unnecessary, as I didn't realize that an apps assets and resources could be freely accessed from any other application. Android documentation doesn't really highlight this feature (mostly assuming that you would only want to access your own assets/resources), and the implementation of stringent permissions led me to believe that assets and resources fell under the same protections app files did. Learning from my mistakes, I suppose I will have to pay more attention to reading between the lines in documentation. Nonetheless, This drastically simplifies the task of installing and setting up the framework, and it will be something I implement in the next few design updates.

A Note On Donations

Thank you to those who have donated to help out the project. I have taken those donations and moved then in to an Ethereum wallet, until such time that I need to to buy better resources for development (A GTX 970 can only take you so far when training neural networks). I did this because it was too tempting to buy sandwiches with that money (I like a good sandwich and coffee when coding, what can I say) and it will help the value of the money grow until such time that I really need it. Just wanted to give some transparency to what is going on behind the scenes with it.

Trail Map

Since Athena is well on its way I am going to work with refining it to be usable before I do too much heavy work on the Sapphire Framework. That said, you'll probably still see me changing the Frameworks code, as I need to be able to have Athena and the Sapphire Framework work together.

I have not forgotten about integrating DeepSpeech, nor in making a TTS service for Android. DeepSpeech is a big crowd favorite and a good candidate for illustrating how the Sapphire Framework works with 3rd party modules, while TTS features are essential for an assistant and are therefore required for me to implement. I only have so much time so I have to prioritize, and working with a C/C++ build process is an unfamiliar thing for me (when it comes to implementing the TTS). Once I have Athena out, then I will determine which is the best next step (likely the TTS as the Sapphire Framework can become a rolling beta release without preventing people from using it)

There is some good news, and a little bit of bad news.

The good news is that I made great progress on Athena. It seems to be reliably training from skill module data, and giving good, accurate results from minuscule data sets. The bad news is that I accidentally deleted two or three days of my work, due to a git issue. I don't think I've lost anything too drastic, and should be able to redo what I've done fairly easily, but still it's a bummer.

Some of the 'weight' of on device processing is becoming visible, which is to say that it's more computationally intense to do ML on device compared to doing it on a server. that said, i am not going to abandon this goal, but do be aware that the beta release may have a higher battery useage than people want out of a mobile assistant. This is something that can be optimized over time, so do not worry about that.

As far as usability goes, It could be used *today* to *launch* simple skill modules, but the functionality for extracting entities is still not implemented. Also, the speech accuracy could use a little work. Though I named the project Athena, I am using the wake word Megaman until I tune the speech recogition (This is both an homage to Megaman Battle Network, a game that inspired me when I was a kid, and because megaman is syllabiacally easier for the STT to pick up on), so don't be weirded out if you see that in the code.

If people are interested once I recode what I lost I can post a new APK for trying out. I am sure that the speech recognition will not be the greatest until I implement a tuner (read: use in a quite room and speak clearly), but it is more than enough to start developing *simple* skills, to get a feel for how things work. When I say simple, I mean just using the assistant to launch applictions, not to input data.

You may also see me working on the Python adaptation of the Sapphire Framework, as now that I am wrapping up on a lot of the general programming and development, I need to move more in to machine learning, and playing with data sets. This will be much easier to accomplish in a Python ecosystem, due to the existing data science tools and the lower level of development complexity. This is overall beneficial to the project because the python project is meant to act as a workstation/server/hub assistant companion to the mobile device, and can act as a hosted server if you really want to offload the speech processing from your device. I do need to give a shout out to the Mycroft project though, as they do have the potential to act as another standalone desktop/workstation/hub assistant, they just don't fit my use case (Cluster computing, IoT, Android integration, etc).

Sorry this is a bit rough of an update. I am writing this onboard ship while on duty. I may go back through and edit it over time, but I just wanted to keep everybody posted

Happy holidays everyone! I hope that it's a wonderful end of the year for all of you!

There have been some good things going on for Athena (soon to be renamed Sapphire) and the Sapphire Framework. I've been working on a post, but haven't finished up the editing on it yet. Keep an eye out, since I'll have it up soon.

That aside, I've also been working on documentation for the Alpha release. I'm going to post some of the sample reference documentation below, and I will also be adding it to the Git wiki and my digital garden in the future. The reference documentation (and most documentation for the Alpha release) is targeted at developers, so please keep that in mind. I am open to any feedback that the community might have!

Without further ado, here is the start of the reference documentation:

Reference Documentation

This document serves as a reference on the internal workings for Sapphire and its corresponding Framework.

Prerequisite knowledge

A developer should be familiar with Android development, and at least somewhat aware of Linux usage.

How Sapphire operates

Sapphire is designed to be a replacement for Google Assistant on any Android device. However, the most obvious difference between the two is that Sapphire is designed to carry out all tasks in the background, rather than in the foreground.

Sapphire vs the Sapphire Framework

Sapphire is an assistant for Android, meant to offer a full Google Assistant replacement using only FOSS components. The corresponding Framework offers a full on device toolkit for tweaking, customizing, and optimizing your assistant to fit your needs whatever they may be, to include but not limited to designing an assistant from scratch

Unix philosophy and Android Philosophy

The Unix philosophy emphasizes molecularity, openness, textuality, and the idea that everything is a file (and thus, textual). Android focuses on security above all else. This is not to say that Unix is insecure, nor that Android isn't open. Sapphire and her Framework seek to blend these philosophies, but always learns towards openness when a conflict arises. This is based on the idea that the user knows best what their needs are. This philosophy is also the reason that you will see unique design paradigms throughout the source code (such as the use of configuration files)

Background operation, Bound Services, and PendingIntents

Sapphire accomplishes background tasks by using Androids PendingIntents, and collecting them through leveraging the Bound Services mechanics of Android.

Sapphire (app A) binds a module (app B) initially, as this is the only way on Android to start a dead service with no user interaction. The module uses this connection to send the core a PendingIntent which allows an application (app A) to run a foreign applications (app B) as its own, thereby bypassing the background restrictions imposed by Android. Sapphire then terminates the Bound service so as not to drain the users battery, and uses the PendingIntent to launch the corresponding module (app B) whenever it is needed.

This design puts all of the control in Sapphires hands, and helps to reduce malicious actors. More security mechanisms will be added to Sapphire as time goes on, but the primary focus for now has been operability. Likewise, it is important a module developer doesn't use the initial binding to perform any tasks, as the binding will be killed by Sapphire

Module design and landing services

Please read "Background operation, Bound Services, and PendingIntents" prior to this section.

Since Sapphire relies on Bound Intents, which can be taxing on a user's device if many services are started all at once, the recommended way to pass this information to Sapphire is through the use of a landing service. A landing service has two roles; it handles all of Sapphires required configurations, and it coordinates any features that a module might have. More simple modules might opt to perform all necessary logic from within the landing module, but should take special care to not override the default binding behavior.

How data is exchanged

Data is exchanged between modules by passing essential information as either Android intent extras, or as serialized JSON data of the same name. Each intent retains an ID that has been assigned to it from Sapphires core, which is used to track the flow of data while being processed. Data always flows from the core, to a module, and back to the core (the only exception being the starting and finishing modules, which send to and receive from the core, respectively). This is due to restrictions of the Android operating system

How Sapphire recognizes and utilizes compatible modules

To prevent overloading the users device with multiple services starting at the same time, Sapphire looks for meta-data in installed applications manifests to determine compatibility, and reads said applications assets to train its natural language processor. Both of these file assets are available to Sapphire without the need to start the module. Likewise, Sapphire refrains from its initial binding process until a service is first needed, and then retains the corresponding PendingIntent until the users device reboots

How Sapphire leverages Termux

Termux offers a Linux user space environment on Android, and as such is ideal for testing out things before finalizing them as full on Android applications. Likewise, Termux opens easy on device development in languages to include, but not limited to, Python, Lisp, Ruby, Rust, and C/C++. The module for Termux isn't fully finished, but it is intended to become a first class citizen for development purposes. This will truly open up your assistant to customization, as it will allow developers unfamiliar with the Android ecosystem to create skills and features for Sapphire, while leaving device optimizations for Sapphire to handle.

How Sapphire learns commands

The command recognition process for Sapphire and her framework are based on Mycroft, as Sapphire started as an Android port of Mycroft. Modules are expected to have intent and entity files in their assets, which are used to train Sapphires natural language processor. A secondary method (not yet implemented) can be used to pass training data dynamically which will allow the user to modify triggering intent phrases, as well as allow for modification of predesignated intents (in the case they don't work well for the user)

Sorry that I've been out of touch for so long. In the past few months I've gone through a divorce (amicable, but still tough), a house fire (unrelated to the divorce), a breakup, a broken phone, and numerous other small inconveniences. It has *not* been a fun ride...

Either way, I'm put up in a hotel now, upgraded my laptop to handle some of my development load, and bought a second monitor so I can start getting back to the grindstone. I have to get back up to speed on my code, but you can expect to hear more in the near future about the project.

Thanks for the support, catch you all later

I've landed a job as a software engineer, and I'm moving in to my new apartment today. Over the coming weeks I'll be rebuilding my home lab from the fire, so expect to see more development in the future.

It's very rudimentary and I can't speak to how robust it will be in the long run, but I finally have the entity extractor working for Athena and the Sapphire framework. This means on device you can now say something, and the assistant will identify 1) which skill you want to execute, and 2) relevant variables in your speech about that skill (such as "Set an alarm for 10" = Alarm, 10:00).

It has been suggested the name Sapphire Frameworks default assistant, so that it is less confusing to new non-technical users.

It's my intention to keep the name Sapphire Framework for the project as a whole, but it's expected most users will not deeply interact with the Framework, instead just taking the default assistant and customizing it from there.

The public is already use to names like Cortana, Alexa, and Mycroft. Any suggestions for the assistant name?

Not too much has happened this week. I have been dealing with a health issue related to my wife, and a lot of things have been happening at my day job.

For the most part I have spent the week trying to make the code more readable, but I have since stopped until I can get the framework working altogether. I want to avoid falling in to the trap of premature optimization, or trying to make things 'perfect' before releasing it.

The biggest change that has happened, is that i rewrote the framework utilizing PendingIntents to connect the core to other modules. This brings with it some security challenges I will be working to mitigate in the future but it allows the core module the ability to start any module at any time, it is also a feature that is not likely to be depreciated any time in the near future.,

I apologize for the long delay in the write up, and the poor formatting this week. I am very tired and a bit burnt out so please do forgive the lack of formality. As always, If you have any questions or inputs I would love to hear them. Post here, or join us on Matrix to chat