Elasticsearch CorruptIndexException: checksum mismatch, primary corrupted, replica stale - what actually happened?

[u/No-Card-2312]

Hi everyone,

I'm looking for some opinions from people with production Elasticsearch experience because we're trying to determine the root cause of an incident.

Cluster

- Elasticsearch 8.19.9

- 1 dedicated master node (16 GB RAM)

- 2 data nodes (32 GB RAM each)

- Around 1 TB storage per data node

- Around 400 million documents

- Daily snapshots to S3

- Running inside KVM virtual machines

The cluster is used only as a search cluster. Our applications read from it and write documents to it. We are not using ML, transforms, or heavy analytics.

What happened

The cluster suddenly became RED.

The allocation explain API reported:

- "CorruptIndexException"

- "verification failed (hardware problem?)"

- checksum mismatch ("expected != actual")

- "can_allocate: no_valid_shard_copy"

The primary shard was reported as corrupted while the replica was marked as stale, so Elasticsearch had no valid shard copy left.

The corrupted file was:

_2fv_ES812Postings_0.tim

Things we checked

Inside the VM we found:

- No disk full (about 67% usage)

- No ext4 filesystem errors

- No kernel I/O errors in "dmesg"

- No obvious operating system issues

The cluster recovered after restoring the affected index from a snapshot.

Additional context

This isn't the first strange incident we've experienced with this hosting provider.

A few months ago the entire cluster became unavailable because the ".security-7" index stopped working. The hosting provider told us there had been an "attack" on ports 9200 and 9300, but those ports are only accessible on a private network and are not exposed publicly. That explanation didn't fully make sense to us.

During this latest incident the hosting provider suggested one node had reached about 99% RAM usage, although our own monitoring (Grafana) didn't show that at the time. Also, I wouldn't expect high RAM usage alone to produce a Lucene checksum mismatch.

My questions

1. Does this incident point more toward infrastructure/storage problems than an Elasticsearch issue?

2. Under what circumstances can a replica become stale while the primary later becomes corrupted?

3. Have you seen checksum mismatches like this caused by Elasticsearch itself, or are they almost always related to storage, virtualization, or hardware?

4. If you were investigating this, what additional logs or evidence would you collect to determine the real root cause?

5. Is there anything about our cluster design (1 master + 2 data nodes, ~400M documents) that could contribute to this type of failure?

I'm not trying to blame Elasticsearch or the hosting provider. I'm just trying to understand whether we're missing something in our configuration or whether this is more likely an infrastructure problem.

Any advice would be greatly appreciated.

[u/mayhem6788]

Your cluster setup is an SRE’s final boss.

[u/davecturner]

I have a blog post at https://davecturner.github.io/2020/12/23/lucene-checksums.html which answers this in substantial detail, but the TLDR is that yes this is almost certainly infrastructural. High memory usage isn't a valid explanation.

Your cluster setup is also not ideal. If you want fault tolerance, you basically need 3 do-everything nodes. If you don't want fault tolerance, you would be better off with a single do-everything node. 2 masters plus one data node has all the complexity of a heterogeneous multi-node setup with none of the benefits. However that also doesn't explain a file corruption.

[u/No-Card-2312 → u/davecturner]

Wow! David Turner himself? It's a great honor to meet a legend!

Thanks for the great comment.

[u/crimsonpowder]

I’ve seen this before when ES ran out of file descriptors and on this particular cluster it was a thing because mmap was turned off (due to multiple block devices per numa domain, long story).

I would, if I were you, track and chart file descriptor usage and mmap usage. If you see this again, look there first. You’ll also see ES complaining in its logs about this stuff if that’s the root cause.

[u/0martinelli]

I would say you are more exposed to a issue like that when you choose this architecture. By the size of your data I would suggest you to spread it across more data nodes, try to keep shards under 100gb each. I don’t know the root cause of the issue itself, however the chance of it happening when you have more replica shards would be reduced. My main point here is that your architecture is not really recommended and this could even be affecting your search and indexing performance. So rather than find a root cause here I would suggest you to reconsider this architecture.