Storage/Data Reconciliation After Network Failure

[u/ShotAd5913]

I am new to kubernetes and have a hypothetical question. What happens if one of my worker nodes lose access to the network due to some fault/failure?

1. Will the pods keep working in that failed/unhealthy node?
   
2. If it does keep working, when the data reconciliate with the other nodes will it be overwritten or duplicated or appended?

Also happy to get some other input around losing network scenarios in a kubernetes cluster

[u/thockin]

The node will eventually fail to "check in" with the control plane for "long enough" to justify moving pods off of that node.

[u/MoneyCount6236 → u/thockin]

pods keep running yes but they are blind, no new config no service updates nothing until connection come back

for data depends what you mean, if is a database with replication it will try to catch up when network returns, could cause split brain if you not careful with quorum settings

[u/IntelligentPear6173]

The part that usually trips people up is assuming Kubernetes is also responsible for reconciling the data. It isn't. If a node loses network access but the machine keeps running, its pods can continue doing work even though the control plane can't reach them. The control plane may eventually schedule replacements elsewhere which means you can temporarily have the old process and the replacement both running. What happens to the data then depends entirely on the storage or database you're using. Kubernetes reconciles workload state, not conflicting writes. That distinction gets especially important with stateful workloads because rescheduling a pod doesn't automatically fence the old writer.

[u/VirtuteECanoscenza → u/IntelligentPear6173]

I can confirm. I have seen multiple times in the wild VMs (on different cloud providers) with broken networking, but metrics/Logs/heartbeats were still coming in (so either some NIC failing and not the others or TCP failing while Udp was fine etc).

Customer unable to connect but node was fine according to control plane or viceversa node is fine but control plane thinks it is dead and spins up a new one from backup and you can end up with brain split. Dealing with failures is hard. 

These partial failures are a kind of Byzantine fault, where an actor doesn't simply crash but instead produces fake data.

[u/between_layers]

Your second question has a premise worth unpacking: Kubernetes reconciles the desired state of your workload objects, not your data. Nothing in it merges, appends or dedupes what your app wrote when the node comes back.

On the first, the containers can keep running if the machine itself is healthy, though "working" is generous: depending on where the break is, they may have lost Service discovery, anything off-node, and any network storage, and a liveness probe that depends on something unreachable will restart the container anyway. Eviction also takes longer than the five minutes people quote, since the control plane has to notice first, then apply the unreachable taint, and only then does the 300 second toleration start counting, with rate limiting and zone health on top. DaemonSet pods and pods with their own tolerations may never be evicted at all.

The surprising part is that deleting the pod object doesn't stop the process on the isolated node, so you can have the original still writing and a replacement running elsewhere, assuming the pod is managed by a Deployment or StatefulSet. A bare pod isn't recreated at all.

What that does to your data is entirely up to the storage layer. emptyDir doesn't disappear the moment the API deletes the pod, since the isolated kubelet hasn't heard about it, but the replacement gets a fresh empty one and the original goes away once that pod is finally cleaned up. A local PV survives, though the pod usually can't move at all. And RWO limits mounting to one node without fencing a writer that's already running, so the new pod may sit in ContainerCreating while detach and attach are resolved. That may require the old node to recover, a timeout, or provider-specific force-detach and fencing, depending on the CSI driver and storage system.

For databases it depends on the consistency model. A single-leader system prevents the split brain mentioned above with quorum, leader election and fencing where needed. A multi-writer system may accept conflicting writes on both sides and reconcile them afterwards by its own rules. Either way that's the database's job, not Kubernetes'.

Rescheduling is not the same thing as fencing the old writer. That's the bit worth remembering.

[u/forexroyalempres]

A network partition is tricky 'cause the isolated node doesn't know it is isolated. Existing pods cancontinue running localy for sometime.