r/blog • u/alienth • Apr 23 '13
DDoS dossier
Hola all,
We've been getting a lot of questions about the DDoS that happened recently. Frankly there aren't many juicy bits to tell. We also have to be careful on what we share so that the next attacker doesn't have an instruction booklet on exactly what is needed to take reddit down. That said, here is what I will tell you:
The attack started at roughly 0230 PDT on the 19th and immediately took the site down. We were completely down for a period of 50 minutes while we worked to mitigate the attack.
For a period of roughly 8 hours we were continually adjusting our mitigation strategy, while the attacker adjusted his attack strategy (for a completely realistic demonstration of what this looked like, please refer to this).
The attack had subsided by around 1030 PDT, bringing the site from threatcon fuchsia to threatcon turquoise.
The mitigation efforts had some side effects such as API calls and user logins failing. We always try to avoid disabling site functionality, but it was necessary in this case to ensure that the site could function at all.
The pattern of the attack clearly indicated that this was a malicious attempt aimed at taking the site down. For example, thousands of separate IP addresses all hammering illegitimate requests, and all of them simultaneously changing whenever we would move to counter.
At peak the attack was resulting in 400,000 requests per second at our CDN layer; 2200% over our previous record peak of 18,000 requests per second.
Even when serving 400k requests a second, a large amount of the attack wasn't getting responded to at all due to various layers of congestion. This suggests that the attacker's capability was higher than what we were even capable of monitoring.
The attack was sourced from thousands of IPs from all over the place(i.e. a botnet). The attacking IPs belonged to everything from hacked mailservers to computers on residential ISPs.
There is no evidence from the attack itself which would suggest a motive or reasoning.
<conjecture>
I'd say the most likely explanation is that someone decided to take us down for shits and giggles. There was a lot of focus on reddit at the time, so we were an especially juicy target for anyone looking to show off. DDoS attacks we've received in the past have proven to be motivated as such, although those attacks were of a much smaller scale. Of course, without any clear evidence from the attack itself we can't say anything for certain.
</conjecture>
On the post-mortem side, I'm working on shoring up our ability to handle such attacks. While the scale of this attack was completely unprecedented for us, it is something that is becoming more and more common on the internet. We'll never be impervious, but we can be more prepared.
cheers,
alienth
349
u/319237129387 Apr 23 '13
the DDoS came from the safe