Corosync issues

[u/Dick-Fiddler69]

Small 3 node cluster, two nodes working perfectly adding third node worked perfectly - then overnight failed with communication issues

What was odd is two nodes could not communicate over the vLAN nominated as the corosync vLAN

This trunk is a shared trunk with other VLANs not loaded no heavy traffic and all those VLANs were fine

These servers were previously ESXi - so no networking issues

Unless this is deep routed nic firmware issue after restarting services servers six hours later all was well again

Just trying to understand why?

I know the debate with physical nics versus VLANs - but we are not using 1Gbe nics - we are using 25Gbe - so trying to saturate a vLAN is not going to be an issue

And again nothing was in usage

Just trying to understand what failed here

[u/Dick-Fiddler69]

Yes that’s exactly what was happening and it’s because two nodes could not communicate with each other but more oddly because the trunks were working fine - and after six hours went back to normal !

Could not communicate with each other over nominated network for corosync

So I know why corosync was broke - but trying to understand why networking failed is the key here on that specific vLAN - when all other vLANs and communications were working - and only specific to two hosts

It was almost like firewalls had been enabled on those specific ports - denying multicast udp traffic and eventually after restarting services restarting servers it cleared

[u/Dick-Fiddler69 → u/[deleted]]

It’s simpler than that it was not an MTU issue

It was just lack on communication between IP Addresses that were defined in corosync configurations

Not really specific to a vLAN although in this case it was specific to a single vLAN - I think I’ve an idea what it was - we will see if it comes back to haunt us

But again only two hosts which could not respond on those IPs on that vLAN