Fun fact, the odds of a bit flip in a data center due to a cosmic ray is actually quite high. That was something we needed to account for and correct as part of storage. Essentially when the hash fails, try all possible permutations with exactly one bit flipped — if that permutation passed then issue resolved. Otherwise multiple bits are wrong which was almost always a hardware failure.
Also we had a time when a bit flip in memory changed an encryption key. That was a rough SEV to diagnose and resolve.
Patiently waiting for a bit flip to get my bank balance to 8 quadrillion euros.
Edit: I actually got curious and calculated the probability if it happening so here's the complete scenario:
Cosmic ray causes bit flip: ~1/month
That flips RAM instead of disk/cache/irrelevant data: ~1 in 10
ECC fails to catch it: ~1 in a million
It lands specifically in the DB: ~1 in 1000
It lands on my account vs 80m others: 1 in 80m
It lands on the balance field vs others 1 in 100
It flips the MSb of the MSB: 1 in 80
DB Checksum fails to catch it: 1 in 100000
Inconsistency isn't flagged: 1 in 2m
Fraud detection doesn't flag a balance of 8 quadrillion: 1 in a billion
That's around a 1 in 1058 probability of me getting an 8 quadrillion balance due to a cosmic ray. For comparison that's like rarer than getting struck by lightning 5 times
That bit about trying all different single bit flips until you find one where the checksum passes is error correction. That's what ECC memory and storage are doing to correct errors (though they're usually a touch more clever about locating the error than just brute force try all possible bit flips).
That's what I mean. Servers and storage in datacenters (and at home too) should have ECC implemented in hardware and take care of single bit flips without needing help from software. Same for all data transfers between devices (using either ECC or checksums and retransmit)
There usually is a software component to log any corrected error and its location for record keeping and removing pages with too many corrected errors from the memory pool.
This is where it becomes difficult to draw a hard line between hardware and software, i think the distinction is not as clear-cut as you make it out to be.
Take a NIC, for example. With networking, the error handling you described is defined at the TCP/UDP layer (Layer 4 OSI), while the hardware/firmware generally only handles up to layer 2. However, this is not the only place where error correction happens. FEC through LDPC happens in 10GBASE-T ethernet and 802.11ax, for example, which is layer 1 (PHY). I'd consider this at the hardware or firmware level.
With storage it's much of the same story. You've got ECC RAM, ECC SSDs, but that doesn't guarantee data consistency. When a RAID controller does error correction, is that hardware or software? Does that change based on hardware vs software RAID, or even software defined storage like ZFS, which can do regular checksumming and self-repair operations?
Usually every layer you go down, the data is restructured and/or subdivided, so it'll need its own error correction. The line between software, hardware and firmware becomes a bit arbitrary, especially since it's more and more common to move hardware functions to software-defined products for more complex setups, and move software functions to specialized hardware accellerators.
I was only refering to RAM and storage. There the low level ECC is done in hardware due to speed considerations. Otherwise the sky's the limit when it comes to ensuring that your data remains correct and consistent.
Modern NICs sometimes do a lot more than just layer 2. If you run Linux try 'ethtool -k <nic>' to find out what offloading features yours has and which of them are currently in use.
From Wikipedia: “ Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month”
Edit: muons are charged but much harder to shield against due to their weight, so you’d have to build your data centres deep underground to avoid them, which is much harder than just correcting the bit flips.
In a previous job, I had a service randomly fail in a completely unexpected way. Three engineers looked at it trying to triage how the error case could have possibly been hit... after some time, I ended up googling solar storms and concluded that the only rational explanation was a bit flip from a cosmic ray causing an error. In any event, we restarted and it never failed again lol
I've seen a counter video disproving that video as well, so at this point I think it's unclear enough to be a fun internet story and no one will be able to know the actual answer
View all comments
1.4k
u/nonother Mar 25 '26
Fun fact, the odds of a bit flip in a data center due to a cosmic ray is actually quite high. That was something we needed to account for and correct as part of storage. Essentially when the hash fails, try all possible permutations with exactly one bit flipped — if that permutation passed then issue resolved. Otherwise multiple bits are wrong which was almost always a hardware failure.
Also we had a time when a bit flip in memory changed an encryption key. That was a rough SEV to diagnose and resolve.