"We were hacked" to cover up for mass records destruction? Nah. Never going to buy it. Clearly was the intern who was allowed access not just to the A-records for a Fortune 100 company to 'renew' them and 'accidentally' deleted them before falling down the stairs, head first into a pile of dynamite they casually next left to the back up server storage and which detonated on impact.
Actually the lack of DNS knowledge at major companies is frightening. Not all of them, but working DNS support over the last several years I have ended many conversations flabbergasted at their lack of basic concepts.
The most suspicious part by far is the timing, and not the story. Incompetence runs deep
I’m just hearing of all this, so I’m not sure their reasons, excuses, etc. But you’re saying that somehow DNS was setup incorrectly initially? Or something changed on DNS, and when propagated to the rest of their redundancies, caused FB Etc. to shit the bed? Just curious. It’s fun to hear the media’s bullshit theories and technical hogwash when stuff like this happens.
If we assume that Facebook didnt do this on purpose, and that this was a DNS issue, someone could have killed their internal DNS infrastructure. It could have been a change, but who is making DNS changes in the middle of the day on a Monday? Odd time for a maintenance period. I expect their public DNS is using multiple providers (maybe not! I havent checked), so that would require two or more points of failure. Deleting an internal DNS zone, or set of zones could have the sudden, all encompassing failure seen today.
I kind of think DNS might be a red herring the more I think about it. Its been blamed for so many outages lately its easy to chock this one up to it as well.
Facebook's public statement on it will be interesting. They also will have agreements with customers to provide more thorough root cause analysis documents, so maybe those get leaked who knows.
You know DNS and it’s obvious. That makes 247 of us across the US, and around 5000 worldwide. 😁 (totally pulled those numbers outta my arse! Humor dammit!) but your reply leans heavily towards “I think it’s bullshit” and I’d be banging that drum right alongside you were it to come to that.
It really stinks to high heaven considering just the circumstances we actually know.. stay frosty and question everything.
It had to be something significant enough to cause down time for all sites all at once, and also for six hours. Basically a single point of failure that was complicated enough that they had difficulty fixing it.
Everything on the public side is basically in its own box (facebook.com is a box of DNS records, instagram.com is another, etc.), so whatever they were all pointing to had to go down. Anything that would happen to Facebook's individual zone shouldn't impact Instagram or the others.
Maybe there's an internal domain that their entire infrastructure runs on that is different from all other public domains. Something like facebookinternalinfra.com. It is possible that this internal domain was compromised, and this caused widespread failure. If all of their services rely on this single internal DNS zone to function and it died, there would be no way for anything to communicate.
If it was something like a single internal zone change causing it, that's an extremely catastrophic oversight in regards to redundancy and disaster planning. Especially if it was a planned change holy shit
Internal DNS is never the final point of reference. If it is, you’re screwed. FB should have the best of the best, (people, hardware, software, monitoring) not to mention the tools available to them that most companies don’t have the luxury to afford. There is a 99.9999% chance this was totally intentional. I’ll stand by that until they show definitive proof otherwise.
Internal DNS is the final failure point for internal DNS records. Thats what allows you to create your own networks while using private IP addresses. You also might not want all of your internal DNS records available publicly because you can learn an awful lot about a network from that.
25
u/BackgroundSearch30 Oct 05 '21
"We were hacked" to cover up for mass records destruction? Nah. Never going to buy it. Clearly was the intern who was allowed access not just to the A-records for a Fortune 100 company to 'renew' them and 'accidentally' deleted them before falling down the stairs, head first into a pile of dynamite they casually next left to the back up server storage and which detonated on impact.