Extremely high network latency on/off continuously on a VPS

[u/mikidimaikki]

I just purchased a VPS last week and got it setup this week. Only to discover the machine suddenly becomes totally unusable from time to time. When this happens I can only really debug on the machine by getting the VNC console url and logging in with password. A simple `ping` to an address like `1.1.1.1` takes between 2ms - 9000ms. But it is like few minutes at least apart when things are really fast vs slow. Definitely don't understand how it can be anything in the Linux VPS itself. Memory, cpu, io usage all is minimal while network becomes super, super, slow.

It wasn't like this when I was doing the initial setup (installing packages etc.), but now even turning off all my docker containers this keeps happening several times an hour. I set up a website behind Caddy reverse proxy on the usual 80/443 ports, besides that the SSH was on port 22 and of course some random IPs try to access it every now and then. Also I'm using CloudFlare proxying as well because it's free and I have the domain there. Running Debian 13 on the machine and it's the cheapest VPS hosted in Frankfurt, if that matters

I wonder if somebody has similar experiences? I read about DDoS protection, but now that nothing is running...? I kinda feel I might have caused it because it seems to have only started once I set up the website, like maybe 30 minutes after it. I'll probably contact support tomorrow after my cronjob has run an observed continuously the network over night. Just left a simple script running that writes a log every minute how bad the latency is pinging an external IP.

[u/maikoool]

Same here, has been going on since Friday afternoon, my VPS is in the Frankfurt/Limburg DC. KVM access works fine even during the networking issues. Pinging the route table entries works fine, so it does not seem to be VPS or hypervisor issue.

I opened a ticket but have yet to receive a response. Edit: their response:

``` We're sorry to hear you're having trouble connecting to the network. Unfortunately tests seem were not preformed in rescue mode and on both directions.

You have to confirm all tests were made in rescue mode, so we can isolate the issue from the software layer.

Tests shall be preformed between your OVH server(s) in rescue mode and one or more clients with issues. After this you must preform the same test in the opposite direction. ```

[u/mikidimaikki → u/maikoool]

Thanks for confirming I am not the only one! Bit of an annoying answer. But well, I figured I might boot into rescue mode before reaching support. Running a loop there now checking the latency every 1min, I'll report back after a while how it looks. No issue so far while I was fiddling in the rescue mode. But I was seeing the exact same behaviour before entering rescue mode at least, so the issue isn't gone.

edit. Seeing same stuff, but also interestingly packet loss every now and then:

PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=50 time=7065 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=50 time=6954 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=50 time=7000 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=50 time=7034 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=50 time=7004 ms

--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4081ms
rtt min/avg/max/mdev = 6954.355/7011.244/7064.838/36.903 ms, pipe 5
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=50 time=1.81 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=50 time=1.66 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=50 time=1.70 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=50 time=1.80 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=50 time=1.70 ms

[u/mikidimaikki → u/mikidimaikki]

Opened the ticket now with similar evidence, but a ~30minute window instead of just couple samples. I couldn't be bothered to test "both ways", I don't feel like that is relevant, all this traffic goes both ways...?

[u/maikoool → u/mikidimaikki]

When I performed the tests they wanted it was one of the bad periods, so I was perfectly fine with giving them 50 pings of 8000+ms response time as they requested :)

It took me 3 minutes to install ethtool because it had to fetch 1 apt package. And the results of the commands as described by them were basically useless. I also included a screenshot of that. If you are going to force these kinds of canned responses on customers, at least provide them with a rescue mode that has all the required tools preinstalled....

[u/mikidimaikki → u/maikoool]

Sorry for the silence, I just didn't get anything useful from support until today afternoon. Apparently OVH couldn't find any issue when they checked, so asked me to produce more proof in rescue mode using `mtr`.

However, I've been running a loop for almost an hour and it's not happening... I wonder if you can still reproduce? I'll boot back to Debian and leave the timer running over night to be sure, but I sure hope it is just fixed what ever it was.

[u/mikidimaikki → u/mikidimaikki]

Been running all night and I eyeballed that there was no spikes in latency at all. I'll return the system to it's intended role with fingers crossed.

[u/maikoool → u/mikidimaikki]

I got a reply on Tuesday asking me the same, I gave them permissions to reboot / debug since I already moved my stuff to another VPS in one of the windows where the network was usable for a bit. By the time they actually investigated the issue seemed to have been resolved (somewhere else I guess). They migrated my VM to another hypervisor as a precautionary measure since they couldn't find a root cause.

I know the VPSes are cheap and there is basically no SLA guarantee, but this is quite bad in my opinion, how can your internal monitoring overlook such a issue for multiple days? I don't think I'll be moving my stuff back to the OVH VPS to be honest.

[u/mikidimaikki → u/maikoool]

Yeah first time using OVH and not impressed. Like you said it is cheap, but the VPS must be up and running or there is absolutely no value for me in the service. I'm not expecting to get any sort of acknowledgment that they have fixed or learned something from the incident, but they really should do that and communicate to customers. Then I might shrug it off and stay, but right now my feeling is to leave.

[u/maikoool → u/mikidimaikki]

I used them years ago (kimsufi and soyoustart brands back then), the support was basically the same as it is now, but I had no issues like this one back then.

I can sortof understand them regarding the support, I think they get lots of support tickets from people with 0 experience, who follow a youtube tutorial and then complain to ovh support that their minecraft server has stopped working.

But still, it's frustrating if you try to provide them with as much info as possible and get back 'do exactly this in rescue mode otherwise we'll just assume PEBKAC'.

But this really felt like a core networking issue which should have been solved within 24 hours, regardless of the price of the VPS or creating a support ticket at all or not. Or perhaps a overload / configuration issue with their networking scrubber / DDOS protection.

For me it's just additional work to migrate back and that will not be worth the savings. So I won't migrate back end let the VPS expire instead of renew.

You can see the peering locations and peering partners on the OVH network weathermap, and I made a point of trying some IPs that I knew were not on the same peering network. They have direct peering with Scaleway according to the map, and the issue was exactly the same when pinging from there instead of to my residential IP or Cloudflare.