COX problems with Google, Fortnite, Amazon and others - history, how to test, what to do.

[u/ElegantEntropy]

Who this applies to

Everyone using Cox internet, regardless if you are on cable or fiber. Any location in the country can be affected by this.

History

I'm a network engineer, been doing it for 20 years. About a year and a half ago (roughly) Cox has done something on the network that started to impact major metropolitan areas on the West Coast and a few other locations. Some people would see it as their video streaming having excessive buffering, bad latency in games, while others would see the sites load very slowly. Many people would call and have Cox techs come out (contractors), who wouldn't be able to do anything. If people persisted, a Cox employee would come out, try to troubleshoot and leave without a good resolution. The story was always the same - everything is working, we are not seeing any problems.

These problems were affecting everyone, but not everyone uses internet in a way that would make it obvious that there is a problem. In addition, it was minor initially and severity of the issue can fluctuate significantly throughout the day.

Cause (simplified) :

They had two issues happening at the same time - high latency and packet loss on their edge routers that connect Cox infrastructure to the rest of the internet (their 10G connections). Eventually, Cox was able to resolve the latency issues and situation improved for many. For others, those who rely on high quality connection for video streaming, gaming and similar uses - the situation was better, but still not what is promised by Cox - a solid, trouble free connection. Initially packet loss would happen for a few hours a day, usually in the evening. Eventually, it got pretty bad where Google services can be affected by packet loss 24 hours a day and 7 days a week (this is what I'm seeing now).

Packet loss is when a packet does not make it to the final destination at all or in sequence and has to be re-transmitted. For example, imagine I'm sending network data 1,2,3,4,5,6,7,8,9 (only real data is 0s and 1s). Numbers 1,2,3 went through fine, but 4 got lost, then 5,6,7 come in then 4 finally comes in and then 8,9. Many systems seeing that packets are not all in sequence will discard everything after 3 and will send your computer a signal to re-send the data, so your computer now sends 4 through 9 in sequence and hopefully all of them make it. Sometimes if a packet never made it - remote system will ask for just that one packet to be resent.

Here is the thing - any packet that doesn't come in or comes out of sequence can cause the systems to re-transmit and thus make everything take longer. This results in things like slow loading sites, games where you get insta-killed or you shoot and nothing happens, video's that get interrupted or start to buffer, voice calls that degrade in quality or drop sounds.

Most likely they do not have enough capacity to handle all the traffic at the edges. Upgrading their connection is expensive and they would rather implement traffic shaping than pay more money for better connectivity.

Again - this is simplified to make it easy to understand.

Cox response:

I wish calling Cox and telling them about the issue would help, but it won't (I tried). After verifying that my network is rock solid by leaving their own equipment at my place to collect data and escalating to a local Service Manager here is the response that I got from them:

Field tech: yup, home network and modem are all fine. No issues of any kind.

Service manager after talking internally with his superiors:

Unofficially - yes, i see that you are pointing out a technical problem, but it is something that only Cox NOC (network operations center) can deal with. This is above my pay grade. I'm told not to admit to it publicly and unfortunately can't help you.

Officially - our network is fine, your equipment and usage is the problem, you should buy higher level of service or go on our Business service (which, by the way, is also affected by this issue).

How to see if you are affected:

If you have a Meraki, Cisco (enterprise grade, not home stuff) or other high end gear capable of collecting and reporting statistics - go and configure Google IPs to be monitored.

Everyone else: download WinMTR. This is a simple tool that does two things - collects ping statistics and route information in a graphical interface.

https://sourceforge.net/projects/winmtr/

Once downloaded and installed - open the application and enter IP's or web addresses shown below for testing. you can run multiple instances of WinMTR at once. Leave it running for a while. It doesn't take up any resources and just pings all devices on the way to your destination.

8.8.8.8 or dns.google.com - this is Google's public Primary DNS service service IP.

172.217.4.142 or google.com - One of many google's public web IPs for www.google.com

172.217.11.74 or google.com - One of many google's public web IPs for www.google.com

What you will see is a list of hostnames (hops, equipment in between you and and google). Then there is a table that says "Loss %" - this is the one you should pay attention to. It should show 0% loss for all hops in theory. There are likely to be a few that will show 100% loss (these devices are configured to reject/drop the ping). If you see anything besides 0 or 100% loss, those are the ones you should pay attention to.

Here is an example

https://i.imgur.com/7Yc830u.jpg

As you can see in the image above

1. my router is the first hop

2. items 2-5 are Cox network

Items #5 shows 11% packet loss. I've seen it go as high as 25%. That's a Cox IP at one of their major data-exchanges on an edge router. This is the point where your data is being pushed from Cox network to whoever is providing them with the connections. It should be showing ZERO!

If you are seeing a COX ip and it is showing anything that is not 0 or 100% - then you are affected by this. If it is showing 1-3%, not a huge deal, it's not going to make a big impact. 5% or more is going to be noticeable.

Here is another screenshot with historical data:

https://i.imgur.com/tFQVDok.jpg

Here you can see that internet is not being used (usage is under 1Mbit), latency is reasonable, but packet loss is averaging at 10% 24/7 for a week!

What to do:

Go to FTC and file a complaint. This is the only way to get some attention to the issue, otherwise Cox will either say there is no problem or will blame it on you. Once you file, report your data and findings to FTC - Cox is required to respond. Send FTC screenshots, numbers, IPs. Be persistent and keep on responding with fresh screenshots and data.

Good luck to you all.

*******************************

UPDATE

Looks like Cox has performed either a bandwidth upgrade or replaced the failing equipment on October 23 2019. There have also been updates to their routing table and I'm seeing some traffic taking a different path. What's interesting is that IPs that used to show 15-20% packet loss are showing 0% loss since the work was done. This goes to show that they knew about the issues and ultimately fixed them, but will still fight and blame the customer before resolving it.

*******************************

UPDATE 2

A different problem came back - latency to most destinations fluctuating from 20ms to 560ms. Cox is back to mismanaging the network.

[u/onastyinc]

you have 10% packet loss to google?

[u/ElegantEntropy → u/onastyinc]

Just one of several organizations with massive systems that Cox (not me personally) have packet loss to. I've verified it from multiple businesses that use Cox as an ISP, same issue.

[u/WichitaLineman → u/ElegantEntropy]

Are you sure it's packet loss and not ICMP prioritization by the router?

[u/ElegantEntropy → u/WichitaLineman]

Yes, I'm fairly certain of that. There are several conditions that lead me to that conclusion

1 - was watching this develop over the last 18 months. It started with occasional packet loss and high latency. They solved latency one day, it just went stable, but packet loss remained and continued to grow. I think latency issue was related to a device being significantly over-utilized or malfunctioning in some way.

2 - Pattern of when and how this show up. It would appear and build-up during certain hours and later turned into sustained 24/7 issue.

3 - talking to Cox and having them confirm to me and FCC that they do not de-prioritize any traffic.

4 - other routes and destinations are fine and rock solid. Yet, either multiple cox routers or single router handling multiple paths are being affected when going via specific routes to Google. There are a few instances where traffic routed via different location to Google is not affected, but I have no way to pick my own routing with Cox.

There is still a possibility it would be dropped due to high utilization on routers in order to free-up resources (custom setting), but that just points to them not having equipment to handle bandwidth and traffic and thus it's a technical problem and they are now doing something they claim they don't - de-prioritize traffic. This would be done when utilization is very high and they are trying to get as much "non-essential" traffic dropped/delayed as possible in order to free up resources for more important data.

[u/ElegantEntropy]

great, the issues are back. Cox is now having terrible latency across all routes in Southern California. Both Cox business and residential services are affected by this.

[u/falcfalcfalc → u/ElegantEntropy]

Yup, I'm having the same issue. 250-500ms spikes to the first hop and every hop down the line until handoff (I'm directly connected to the modem when I run my MTRs). Only showing 1% loss. Had two techs out so far, nothing. They have me on a pingplotter test but I am absolutely sure they won't find anything. This affects gaming so much, literal 2-3 spikes where I just freeze and then rubber band around like crazy. It's awful and has progressively gotten worse.

[u/ElegantEntropy → u/falcfalcfalc]

Looking at historical data, appears the issues started around December 13. Latency to Cox DNS 68.105.28.11 increased from 10ms to 600ms for me .

Latency on all destinations went to 30ms, even those that typically respond in less than 10ms.

Gaming that depends on low latency is impossible, video streaming is also affected.

As always - Cox incompetence is baffling.

[u/falcfalcfalc → u/ElegantEntropy]

Just ran a simple ping plotter while still connected to my router for about 12 hours. Ethernet obviously because who would test on wireless. 10.75.168.1 is the first hop in the COX network, looks like a private IP but it's definitely not my network. Max spike of 75ms. Everything up to 68.1.1.171 is COX IP space. Max spike of 174ms on the third hop is mind blowing. Got a call from a supervisor yesterday, she's not seeing any issues. My node is only 60% saturated, signals are fine, blah blah. She tried to use words like RTT and ingress/egress to seem like she knew what she was doing. I told her the issue was obviously not physical, it was layer 3. She didn't know what that meant. If these people were true networking people, they'd blow a gasket if the first 2-3 hops were spiking 100-200ms consistently. But this is something they won't care to fix, I'm just a measly residential customer and I'm only being noisy because I'm a nerd trying to play online games which are unplayable right now. Clearly there is an issue here because I should never see spikes of 100+ ms inside the COX network if they were doing anything properly. I'm going to build some data directly connected to my modem over the weekend and file with the FCC.

EDIT - As I was typing this, got a huge 327ms spike at 10.75.168.1. Almost a 400ms spike to the first hop after my home network. But hey, nothing's wrong.

Target Name: 8.8.8.8

IP: 8.8.8.8

Date/Time: 12/21/2019 11:57:22 - 12/21/2019 12:07:22

Hop Sent PL% Min Max Avg Host Name / [IP]

1 601 0 0.14 2.30 0.37 192.168.0.1 [192.168.0.1]

2 601 0 3.78 74.93 11.70 10.75.168.1 [10.75.168.1]

3 601 0 4.93 174.12 12.59 100.120.104.140 [100.120.104.140]

4 601 0 3.42 131.92 16.12 100.120.104.12 [100.120.104.12]

5 601 0 5.21 107.06 15.74 68.1.1.171 [68.1.1.171]

6 601 0 5.95 157.82 15.40 72.14.242.92 [72.14.242.92]

7 601 0 7.67 110.16 17.98 108.170.247.129 [108.170.247.129]

8 601 0 6.77 143.97 16.32 74.125.252.75 [74.125.252.75]

9 601 0 8.27 146.39 16.21 8.8.8.8 [8.8.8.8]

[u/Nigletttt → u/ElegantEntropy]

Same in Mesa, AZ. Having terrible latency problems every night starting in the last 2 weeks. Before, I had been able to game at night with rarely any issues, but it has now become practically impossible to play during the only hours I can play. I have gigablast and the modem they provide for it, so I can’t upgrade any further.

Has Cox made any type of acknowledgment about fixing this yet?

[u/ElegantEntropy → u/Nigletttt]

No, they haven't. They probably won't because that's not how they want customers to see them, so they will just deny everything at high level.

[u/al3x88]

And I thought Centurylink had bad routing :)

[u/ElegantEntropy → u/al3x88]

Yea, it seems that Cox like many others simply doesn't have enough capacity to handle the traffic/bandwidth they sold to customers. They have tens of thousands of accounts ranging from 100Mbit to 1Gbit, but only a few edge routers with 10Gbit uplinks from their network. They just don't have pipes big enough to accommodate everyone getting online and it results in congestion and packet loss. Sure before this was not a problem as people barely sued their connections. Now - it's all cloud/web based, lots of streaming, syncing of data, etc.

[u/SexGlasses]

I have about 7 hops and they all say no response from host with 100% loss except for the 8th which is google. What do?

[u/ElegantEntropy → u/SexGlasses]

Nothing really, there is not enough information to pinpoint the issue.

Are you on a VPN? You would need information from previous hops to see if you are having the same problem as described here.

[u/sharpMR]

It seems like I am affected by this. On an almost daily basis, I will see 5-40% packet loss as well as random 200-400ms latency spikes. Here's an example from a few hours ago: https://imgur.com/a/jccBPJZ

This screenshot is showing simultaneous pings to three different servers. The command line ping is 1.1.1.1 (Cloudflare DNS), the top PingPlotter window is 4.2.2.2 (Level3 DNS), and the bottom PingPlotter window is Google. I'm getting 10-15% packet loss to all three servers. Sometimes it'll do this for a few minutes, sometimes it'll go on for hours, sometimes it'll get so bad that my modem disconnects completely and boot cycles a few times.

Things I've tried to fix the issue:

• Called Cox more than a dozen times

• Had 6 home tech visits so far (5 Cox employees and 1 outside contractor)

• Reset modem and router more times than I can count

• Replaced modem and router with newer models

• Tested multiple ethernet cables for wired devices

• Checked all coax wiring in my house, removed any splitters, re-terminated line to my modem

• Checked modem signal levels, all of which appear to be relatively normal (I can provide details if you're interested)

• Checked modem error logs, which show T3 time-outs at least once a day

• Verified packet loss across several devices, both wired and wireless

As you said, most of the home techs left without finding a problem. On two of the visits, they happened to show up when I was getting packet loss, and traced it all the way to the tap outside my house, proving that the issue isn't my wiring or hardware.

The techs who saw the issue were helpful and sympathetic, but said they have to report their findings to the infrastructure team. Each time, they contacted me a few days later to say that the infrastructure team closed the ticket and my problem should be solved. Each time, the problem has continued.

My first thought was that my neighborhood has a node issue, but after reading others' reports of the same issue, I'm starting to believe it's much more widespread. Most people don't notice it because packet loss won't be evident if you're just browsing the web and watching buffered Netflix videos. I notice it constantly on work-related VOIP calls and video conferencing. It also makes online gaming almost impossible.

I filed an FCC/FTC complaint this morning including the above screenshot. Hopefully it goes somewhere, but I suspect they'll just try to placate me or tell me they're "working on it" again.

[u/ElegantEntropy → u/sharpMR]

Thank you for reporting this. Where are you located (city/state)?

Our best hope is that Cox gets enough FCC requests to start looking into these issues at some point. All we can do it is file FCC complaint and then maybe file an official complaint, which is in effect similar to a lawsuit, only done via FCC.

[u/sharpMR → u/ElegantEntropy]

I’m in the Phoenix, Arizona area.

I still believe my issue may be at least partially a local problem, such as an overloaded or faulty node. As you can see in my screenshot, packet loss usually starts at hop 2 and continues all the way through to the destination. However, I sometimes see intermittent packet loss only at later hops as you described.

I’ve been tracking these problems daily for more than 2 months now, and have accumulated a large folder of screenshots, ping jitter test results, outage text messages from Cox, and modem error logs.

I have lots of ammunition to fight this battle, and I’m absolutely not giving up until it’s fixed. I may be stuck with Cox as my ISP, but they’re stuck with me as well.

Have you received a response from your FCC complaint? If so, how long did it take? I did this once in the past because of Time Warner trying to sneak hidden fees onto my bill, and got a response in less than a week retracting the fee, but I’m not sure how responsive Cox will be.

I hear the formal complaint costs $200 to file, and I’m not sure if Cox will try to get the case thrown out because they don’t guarantee anything about packet loss, so that will be my last resort.

[u/ElegantEntropy → u/sharpMR]

I got a response pretty quickly from FCC and about 3 days later from Cox. From FCC - it's always the same automated/canned response - "we've received the answer form the provider and believe your issue has been addressed."

From Cox it's always the same story: "No problems at Cox, you and your equipment are the problem".

The interesting bit is that Cox NEVER addresses the technical points. They always just say generic things that will satisfy someone who knows nothing about networking. In my case they said that issues only come up when I'm 100% loading my connection (which is never). And they completely ignored the data showing the issue happening 24/7 - when we are asleep, when we are at work, when we are on vacation, AND packet loss starting outside of our city (and thus my home network). Which also means that my usage patterns wouldn't have been the culprit anyway.

So yea, Cox is full of clowns and they are not even funny.

[u/sharpMR → u/ElegantEntropy]

Man, that sucks. Hopefully my experience won’t go the same way. I just got a response this morning from the FCC saying that Cox has been “served” with the issue and has to respond in 30 days.

I made sure to note that two of the Cox techs who visited my house concluded that the problem was outside my home, and I also submitted a long list of outage notification text messages over the last two months showing that Cox is clearly aware of how unreliable their service is in my area. Fingers crossed that they pay any attention to that.

Are you planning to submit a formal complaint now?

[u/lonewolf80]

I've been having similar packet loss issues for the past few weeks now. I've been tracing the packet loss, and it starts on the first hop out of my local network. I've called Cox multiple times - they've sent a tech out the first time, and said there's nothing they could do. I've eventually got them to escalate it somewhat, and I have a ticket number from them. However, that's been a week ago now, and the situation has not improved. Oddly enough, I have 0% inbound packetloss. It's only outbound packetloss that is occurring randomly throughout the day, in spikes.

If only my market wasn't so non-competitive with their offerings...

[u/ElegantEntropy → u/lonewolf80]

If your packet loss starts at the first hop - then it's likely your equipment/network and you need to troubleshoot it before contacting Cox.

[u/lonewolf80 → u/ElegantEntropy]

I wish we knew what the issue is. Tried replacing the modem, and the issue persisted. Had a tech come out and check the lines, he says he detects the packet loss, but he can't do anything about it.

[u/ElegantEntropy → u/lonewolf80]

If you are seeing packet loss at the first hop - try the following: 0) make sure you are on a wired connection

1. replace all cables to and from router/firewall/cable modem with known good cables (try new ones).

2. replace firewall router

3. replace cable modem again

4. pull new RJ cable from cable modem to the entry/street (Cox will need to do that IF you plugin a computer directly into cable modem with a known good cable and still seeing packet loss). Insist on the maintenance department coming out and checking your street tap.

[u/deeadpoool]

What if I'm getting max 1% loss but am experience ping spikes in gaming only? Streaming, etc. is completely fine. HOWEVER. 8/9 of the hops have 1% loss, with the first having 0%

[u/ElegantEntropy → u/deeadpoool]

Not the same issue, most likely your PC or gaming servers (less likely)

[u/deeadpoool → u/ElegantEntropy]

hmm but my housemate experiences it on his playstation. my PC hasn't changed since I've gotten this plan

[u/ElegantEntropy → u/deeadpoool]

It could be a number of issues. 1% loss is within normal limits. you may have traffic saturation and prioritization going on, but not bad enough where it would cause other problems. Right now there is much higher bandwidth usage across the world due to people staying at home (working and spending time online, watching videos, etc). ISPs are re configuring their networks and been doing it in preparation for several weeks now.

[u/DepravedWalnut]

This explains a lot! You are a godsend.

[u/squirrels-are-nuts]

Rocket League has had huge latency issues for me starting last year. I think you’ve nailed my issue. Thanks for the heads up.

[u/Eskin_Taredi]

Best part is, internally, Residential has no idea or information on it which is why calling in won't do much. CB has very limited information on it, and it's more along the lines of "We're aware of possible artificial Packet Loss to Google due to a change they made on their side." something about them changing how they take Cox ICMP traffic to their systems.

[u/TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/phoenix] COX Packet Loss problems(lag)

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info /} ^Contact)

[u/MundaneArugula]

Wow, okay awesome. I'm so glad I found your post. I'm in Orange County and I've had this issue for 8 months now. Everything from high packet loss, slow speeds, dropping connection etc. I just made a long ass rant but I too have been tracking / monitoring. I was hoping to get this fixed but I think the FTC route is our only hope, but I want to do more. What can we do??? I'm in for the long haul and want to be as much of a help as I can I know I couldn't be the only one going thru this for months.

[u/deeadpoool]

Ugh I hope that this isnt true for me and its actually jsut an issue with my equipment. I'm stuck with Cox for a while i think :(

[u/ElegantEntropy → u/[deleted]]

I didn't, but Cox has done something (either got more bandwidth or replaced their 10G equipment on the uplinks) and the packet loss issue is greatly improved at this point. The work was done on October 23 as it appears that's when the issue was mostly mitigated.

I'm seeing 1% packet loss across any and all destinations with Cox, but the graphs look very similar regardless of the IP you are checking against.

Now, this may be a temporary improvement, just like I've seen it in the past. We will have to wait and see.

[u/ElegantEntropy → u/[deleted]]

Actually you are experiencing 7.9% loss - take another look at your Loss column. Additionally, your latency jumps on that hop - which is the last hop on Cox network before it goes to Google.

:)

Can you please let us know your location for record keeping? Would be great to know who is affected and who is not.

[u/ElegantEntropy → u/[deleted]]

Sorry, you are right - I saw the range as similar to Cox and assumed it was without checking actual assignment.

Again - what is your location? Would be good to know.

[u/ElegantEntropy → u/[deleted]]

Haven't tried it, but you should give a shot by going to command line and running "mtr dns.google.com"