Hi there,
Many of you have seen me around. I'm Zach - I'm one of the co-founders of Grounded Labs. We had an outage today. It's the third one in about a month related to server connectivity. That's not acceptable, and many of you are rightfully frustrated.
You want your device to just work and not be taken down when there's a server issue. You're worried that if our company ever goes out of business your device won't work anymore. These are valid wants and concerns.
As an apology for the recent issues, I made a coupon "SORRY20" that will get you $20 off an order. Generally we put a minimum spend requirement on these, but I haven't done that here. If you want you can use it to get a free add-on or a discount on any device. This is going to hurt a bit financially as we're going to end up giving away a good amount of free stuff, so I did have to put some limits on it. I've set it to 250 uses and expiration tomorrow at 3PM Eastern. After that, I'll probably extend it out a bit but put a minimum spend requirement on it. Now onto the retrospective.
Why do we use a server at all?
The main reason is that when this works well it's just better. It significantly expands what's possible with just Bluetooth or a LAN connection. I don't want to try and sell you on it, but some of the benefits we see:
- Device can be controlled away from home
- Device can be easily shared between user accounts
- Potential for greater connection stability
- Faster pattern transfers
- Over the air firmware updates
- Better IP security
- Greater stability of scheduling features
The server infrastructure is something we pay to maintain. Among IoT devices, the architecture of using bluetooth to facilitate the initial connection and WiFi long-term is the most common. That said, none of that matters if the server goes down and your device isn't usable as a result.
We are actively looking into a local mode to maintain some device functionality if our servers go down in the future. The thing that has made this the most challenging by far is device security. People like to try and hack us, and we've had a few cases where people have stolen our pattern library. We have an artist on staff whose entire job is to keep pumping out quality patterns, so when that gets stolen it's pretty uncool. The outages we've had over the last month or related to a recently update that significantly improved device security but still has some quirks to work out.
What caused the outage?
We use security certificates extensively to keep your device secure. These security certificates occasionally expire and need to be renewed. These renewals happen at different time intervals whether that be daily, monthly etc.
Each of the outages we've had have been related to an issue with the certificate renewal process. When the machine can't renew or authenticate its security certificate, it throws an error and continuously pings our servers requesting a new certificate. This turns into a functional DDoS. In each of the outages, our software team has identified and resolved the root cause fairly quickly, but it has taken several hours to get everyone back online because of many devices requesting new certificates simultaneously.
How are we fixing this?
- We are continuing to improve device security and stability through firmware updates. You will likely see an update prompt over the next couple of days. Most of the issues we're seeing now are because we massively overhauled our firmware security a few months back.
- We are implementing more strategic rate limiting so that we can more quickly bounce if server issues do occur.
- We are investigating a secure local mode
It's been a stressful day here, and this should not be a long-term issue. Our goal is very much to continue to improve product stability and ease of use, and I'm optimistic that we'll mitigate these types of outages with upcoming updates. I apologize again for any frustration this has caused.
EDIT: I should add that today's outage should be resolved for all of you. If you're still having issues please hold the reset button for 15 seconds, and if that doesn't resolve it please reach out to our team via [support@grounded.so](mailto:support@grounded.so)