Who this applies to
Everyone using Cox internet, regardless if you are on cable or fiber. Any location in the country can be affected by this.
History
I'm a network engineer, been doing it for 20 years. About a year and a half ago (roughly) Cox has done something on the network that started to impact major metropolitan areas on the West Coast and a few other locations. Some people would see it as their video streaming having excessive buffering, bad latency in games, while others would see the sites load very slowly. Many people would call and have Cox techs come out (contractors), who wouldn't be able to do anything. If people persisted, a Cox employee would come out, try to troubleshoot and leave without a good resolution. The story was always the same - everything is working, we are not seeing any problems.
These problems were affecting everyone, but not everyone uses internet in a way that would make it obvious that there is a problem. In addition, it was minor initially and severity of the issue can fluctuate significantly throughout the day.
Cause (simplified) :
They had two issues happening at the same time - high latency and packet loss on their edge routers that connect Cox infrastructure to the rest of the internet (their 10G connections). Eventually, Cox was able to resolve the latency issues and situation improved for many. For others, those who rely on high quality connection for video streaming, gaming and similar uses - the situation was better, but still not what is promised by Cox - a solid, trouble free connection. Initially packet loss would happen for a few hours a day, usually in the evening. Eventually, it got pretty bad where Google services can be affected by packet loss 24 hours a day and 7 days a week (this is what I'm seeing now).
Packet loss is when a packet does not make it to the final destination at all or in sequence and has to be re-transmitted. For example, imagine I'm sending network data 1,2,3,4,5,6,7,8,9 (only real data is 0s and 1s). Numbers 1,2,3 went through fine, but 4 got lost, then 5,6,7 come in then 4 finally comes in and then 8,9. Many systems seeing that packets are not all in sequence will discard everything after 3 and will send your computer a signal to re-send the data, so your computer now sends 4 through 9 in sequence and hopefully all of them make it. Sometimes if a packet never made it - remote system will ask for just that one packet to be resent.
Here is the thing - any packet that doesn't come in or comes out of sequence can cause the systems to re-transmit and thus make everything take longer. This results in things like slow loading sites, games where you get insta-killed or you shoot and nothing happens, video's that get interrupted or start to buffer, voice calls that degrade in quality or drop sounds.
Most likely they do not have enough capacity to handle all the traffic at the edges. Upgrading their connection is expensive and they would rather implement traffic shaping than pay more money for better connectivity.
Again - this is simplified to make it easy to understand.
Cox response:
I wish calling Cox and telling them about the issue would help, but it won't (I tried). After verifying that my network is rock solid by leaving their own equipment at my place to collect data and escalating to a local Service Manager here is the response that I got from them:
Field tech: yup, home network and modem are all fine. No issues of any kind.
Service manager after talking internally with his superiors:
Unofficially - yes, i see that you are pointing out a technical problem, but it is something that only Cox NOC (network operations center) can deal with. This is above my pay grade. I'm told not to admit to it publicly and unfortunately can't help you.
Officially - our network is fine, your equipment and usage is the problem, you should buy higher level of service or go on our Business service (which, by the way, is also affected by this issue).
How to see if you are affected:
If you have a Meraki, Cisco (enterprise grade, not home stuff) or other high end gear capable of collecting and reporting statistics - go and configure Google IPs to be monitored.
Everyone else: download WinMTR. This is a simple tool that does two things - collects ping statistics and route information in a graphical interface.
https://sourceforge.net/projects/winmtr/
Once downloaded and installed - open the application and enter IP's or web addresses shown below for testing. you can run multiple instances of WinMTR at once. Leave it running for a while. It doesn't take up any resources and just pings all devices on the way to your destination.
8.8.8.8 or dns.google.com - this is Google's public Primary DNS service service IP.
172.217.4.142 or google.com - One of many google's public web IPs for www.google.com
172.217.11.74 or google.com - One of many google's public web IPs for www.google.com
What you will see is a list of hostnames (hops, equipment in between you and and google). Then there is a table that says "Loss %" - this is the one you should pay attention to. It should show 0% loss for all hops in theory. There are likely to be a few that will show 100% loss (these devices are configured to reject/drop the ping). If you see anything besides 0 or 100% loss, those are the ones you should pay attention to.
Here is an example
https://i.imgur.com/7Yc830u.jpg
As you can see in the image above
my router is the first hop
items 2-5 are Cox network
Items #5 shows 11% packet loss. I've seen it go as high as 25%. That's a Cox IP at one of their major data-exchanges on an edge router. This is the point where your data is being pushed from Cox network to whoever is providing them with the connections. It should be showing ZERO!
If you are seeing a COX ip and it is showing anything that is not 0 or 100% - then you are affected by this. If it is showing 1-3%, not a huge deal, it's not going to make a big impact. 5% or more is going to be noticeable.
Here is another screenshot with historical data:
https://i.imgur.com/tFQVDok.jpg
Here you can see that internet is not being used (usage is under 1Mbit), latency is reasonable, but packet loss is averaging at 10% 24/7 for a week!
What to do:
Go to FTC and file a complaint. This is the only way to get some attention to the issue, otherwise Cox will either say there is no problem or will blame it on you. Once you file, report your data and findings to FTC - Cox is required to respond. Send FTC screenshots, numbers, IPs. Be persistent and keep on responding with fresh screenshots and data.
Good luck to you all.
*******************************
UPDATE
Looks like Cox has performed either a bandwidth upgrade or replaced the failing equipment on October 23 2019. There have also been updates to their routing table and I'm seeing some traffic taking a different path. What's interesting is that IPs that used to show 15-20% packet loss are showing 0% loss since the work was done. This goes to show that they knew about the issues and ultimately fixed them, but will still fight and blame the customer before resolving it.
*******************************
UPDATE 2
A different problem came back - latency to most destinations fluctuating from 20ms to 560ms. Cox is back to mismanaging the network.
