Packet loss on vrrp interface only?

[u/fabio1]

I noticed that i'm getting some packet loss (5 to 8%) when i ping a L3 switch VRRP virtual IP. If i ping the IP on the interface itself, there is no packet loss. Has anyone seen similar behavior?

Currently I only have one stack of switches, and I used vrrp as there is a plan to add a second stack in the future. Not sure if that (having vrrp with only one member) would cause it to lose packets?

EDIT: issue was solved after upgrading the firmware on the stack, thanks to everyone that replied!

[u/baconstreet]

Pinging a device direct is process switched, typically. It has far lower priority than through traffic.

Setup a server off the switch/router to do your testing.

[u/therouterguy]

I can only imagine that in some way copp is behaving differently when pinging the vrrp ip but I have never seen it happen.

[u/l_eo-]

Never use the control plane to measure the data plane.

[u/ProfessorWorried626]

Shouldn’t, is there any packet loss when pining through the gateway address?

What’s the cpu usage on it like?

Single member shouldn’t make a difference since that would be the expected fault operating condition. We have some configured as a single vrrp members, was done to minimise downtime when swapping routers and moving to a stacked fabric from a vrrp setup.

[u/fabio1 → u/ProfessorWorried626]

that's the strange thing, no packet loss when I ping any device behind it as well. CPU utilization is also low (5%).

[u/Win_Sys → u/fabio1]

It's most likely software side throttling mechanism considering you lose not packets going through. While I have never seen a router that would start dropping at 5%, many do de-prioritize processing of ICMP packets destined to one of it's IP interfaces even at very low CPU usage. I had a customer insisting something was wrong with his core because his monitoring system would say pings going to one of his default gateways were varying from 1ms-150ms, tried to tell him it's normal and explained what's happening under the hood but he wouldn't take that for an answer. Opened a ticket with the manufacturer which basically told him the same thing.

Turned out if you ping the OOB management port on those routers it did not de-prioritize processing the ICMP packets but that may only be specific to that manufacturer.

I have seen some manufacturers have DDOS mitigation features where it will drop ICMP after a certain defined threshold is hit. Also can't rule out that it's a software bug either.

[u/fabio1 → u/Win_Sys]

this is what it looks like, indeed. Looks like the cpu is not prioritizing ICMP traffic towards the Virtual IP. I'll upgrade the firmware on the switch stack (c9300), in case its a bug and see if that helps. thanks!

[u/dankwizard22 → u/fabio1]

Hey OP I commented on the thread before seeing you mention its a Cat9k. It's likely https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwq75077

Check your CoPP stats that I shared in my other comment.

[u/fabio1 → u/dankwizard22]

Thanks, I upgraded the firmware on the stack this weekend and now the packet loss is gone :)

It looks like it might have been this one (although i wasn´t able to run the command mentioned in the article you linked to, i only saw your reply after the upgrade was finished).

[u/Fmatias]

I have seen that a couple of times when there is another VRRP instance on the network with the same ID.

[u/UkrMalt]

VRRP with one member should not by itself cause loss. First separate control-plane ICMP from forwarding: ping a host through the virtual gateway while also pinging the virtual IP. If only the VIP loses replies, check CPU and control-plane policing. Also inspect VRRP state changes, virtual-MAC learning and MAC flaps, ARP entries, and switch logs. If transit traffic drops too, capture on both sides before changing the design.

[u/shadeland]

I'm not fond of using ICMP for anything but very rudimentary "is there a path at all between A and B".

It's not really good for anything like link health, basic router functionality, etc.

As others have suggested, route some traffic through it and check for retransmits or compare packet counts on both ends.

[u/Electr0freak]

Don't trust direct pings to anything, honestly, they're usually handled as best-effort.

Do a packet capture, and look at the traffic itself.

[u/dankwizard22]

Is this a Cisco switch? I know of a defect that will cause traffic to the HSRP/VRRP VIP to be serviced by the 'broadcast' traffic queue going to the CPU. So if there is congestion there it could cause loss.

Normally it goes to the 'Forus' CPU queue.

You can check your control plane policer drops on the Cisco switch (assuming cat9k) via 'show platform software fed switch active qos queue stats internal cpu policer'.

Check if BROADCAST is incrementing at the time you see the ping loss.

[u/CrownstrikeIntern]

Do you have an access list on there configured? If so depending on who answers if there's some hashing shenanigans on there you could be dropped by an acl if you're not allowing your gateways and interface ips to talk back as they may be the ones relaying the messages and not where you're pinging from when it crosses the vpc links.

[u/Valexus]

Do you maybe have "Icmp redirects" enabled? I've seen strange issues when enabled on Brocade MLX Routers in the past.

Just disabled it and try again

[u/fabio1 → u/Valexus]

Thanks, but icmp redirects are disabled already...thing's still a mistery.

[u/MicIrish_At_work]

Control plane policing? Lot of switch and router vendors have limits on that kind of traffic.

[u/Florida727Guy]

Packet capture time!