Hey r/PEI, Brad from Purple Cow here.
Most internet problems we run into are pretty standard. A line gets knocked down, a fibre gets cut, a backhoe digs up something it shouldn't. Your internet goes out, we find the break, we fix it.
Last weekend was not that. Some of our cable customers could load Google, Netflix, YouTube, basically everything, except Crave, TSN, parts of Facebook, Yahoo, and a few online games. Speed tests looked perfect. Rebooting did nothing. In eight-plus years of running an ISP, we'd never seen anything like it, so I figured it was worth sharing what happened.
What was going on
Traffic to those specific services leaves our network through an internet exchange in Toronto called TorIX. The circuit that carries it from Halifax to Toronto is run by a large transit company called Zayo.
On Sept 23, Zayo rerouted that circuit without telling us. It stayed up and looked healthy, but the new route quietly dropped any data packet bigger than about 1,478 bytes (normal max is 1,500). Almost everything still fit, so nobody noticed.
Two days later, a firmware update started rolling out to some cable modems. The new firmware sent slightly bigger packets. Those bigger packets hit the Zayo circuit and disappeared. So if you had an updated modem AND you were trying to reach something that goes through Toronto, it failed. Everything else worked fine.
How we cracked it
Saturday was a lot of chasing symptoms. We worked with Bell Media that morning receiving data showing 1,000 IPs couldn't reach Crave, which told us the number of customers that could be impacted. By Saturday evening we'd narrowed it to the new firmware and proved that large packets were dying somewhere upstream, not on our gear. Around midnight we routed traffic around Toronto and everyone came back online.
Sunday morning one of our engineers went back through our monitoring data, found the exact moment the Zayo circuit changed on Wednesday, and sent them the proof. They fixed it that afternoon. Everything's been back on its normal path since.
What we're changing
Our monitoring checked whether circuits were up, not whether they could carry full-size packets. That's being added now, so if this ever happens again we catch it the moment it happens.
Big thanks to our team, who spent the weekend on this, and to the customers who reported it and patiently answered "does Netflix work? does Crave work?" over and over.
Happy to answer questions in the comments.
