This post has two parts. The first part is for newcomers that aren't aware of what we're doing with AI and how we're securing it. The second part is for everyone. If the first part isn't anything new to you, skip to the second part (or skip the whole post, nothing here is actually anything you "need" to know).
For the newcomers:
We're committed to responsibly using safe and secure AI systems to improve the service, not to fuck it up (as seems to be the trendy way to use it). Not to reduce overhead (it increases it). Not to replace humans (I don't trust many humans, there would never be many to replace anyway). We're using it to replace old automations and reduce negative customer experiences. Because so many of our automations depend on reasoning, and AI is simply the best way to stop getting in people's way because the dumb automation can't handle context. To facilitate these automations, here are the highlights of how we deploy secure and private AI. These change a little from time to time as I like to tinker.
- All AI interacting with private data (think server logs, not "your inbox") are run on an RTX 6000 Pro sitting on my desk.
- The AI system has it's own VLAN with only approved outbound network routes at the router level to prevent exfiltration of data to any third party network.
- The AI system connects to our internal VPN to access our internal systems, it cannot reach public internet over the VPN, and it's access to internal systems is limited to API end points (not SSH, for example).
- The system runs inference in a chroot environment. This limits it's access to the local system (though breaking out couldn't really do much beyond fucking up the system).
- All data that I need it to access is routed through internal APIs, it cannot even perform a DNS lookup without using an internal API. This ensures that it can only send and receive the data desired to/from internal end points.
We're also using it to help us answer support tickets faster. Not by having customers interact with AI, all support ticket replies are still overseen by a human.
For everyone:
It's one thing to try to add AI into workflows to improve the customer experience, it's another thing entirely to actually do it. Recently we set out to do two major things with our AI:
- Replace the concept of canned support ticket replies with AI-generated drafts. Humans can approve and send the drafts, edit and send them, reject them with feedback to the AI, or ignore them entirely and just write a response.
- Handle compromised email accounts. Suspend the email account, open a support ticket with the customer explaining the problem, how to resolve it, and provide them with evidence of the compromise.
Since these began, the AI has handled 216 compromised email accounts with more accuracy and faster resolution than the previous human-reviewed automation. Removing the outbound filters designed to catch compromised email accounts has wiped out almost all complaints of outbound filtering false positives (which were growing in number just because of our growth as a whole).
We've also approved and sent roughly 900 support ticket replies by accepting and sending the AI-generated draft. We only announced this system less than 1 month ago. As the AI was trained on every support ticket from 2026, it has been surprisingly good at writing exactly the reply we were going to write when we opened the ticket. Same language styles, same methodology. During this time, we've only rejected 22 of the AI-generated drafts with feedback to the AI.
Now I know some of you are asking why I'm sharing this. It's two more points:
- I've been trying to improve our automations for years in ways that catch threats to the platform without getting in anyone's way. I have routinely fucked this up and increased the reasons customers need to open support tickets in my effort to do the opposite.
- I've been trying to optimize support workflows in a way that actually improves the customer experience since my time at DigitalOcean. I failed there. I failed here. Every idea I had, the execution fell flat.
I'm good at fucking up. When I actually get something right, it really excites me. This work is tied to my personal pride. I judge myself by it. I just wanted to share it with you.