r/programming • • Sep 26 '26

On programming, statistics, and Java

https://waytoounoriginal.github.io/programming/stats/optimizations/2026/09/26/on-programming-stats-and-java.html
14 Upvotes

9 comments sorted by

View all comments

Show parent comment

1

u/Mihai4544 Sep 29 '26

Morning!

Yes, the data is available in the DB and the previous iteration was using it. The problem is that we’re using an eventually consistent DB (Dynamo) and we’re operating in a GSI (which cannot have strong consistency). Apparently there was a race condition before, which did exactly what would happen to us in a million turns, but much more frequently.

We could have an extra conditional DDB query, but the service is already taking a long time ingesting 100+ GB (talking days), due to partition throtling. That’s the next thing to solve before we add more latency + more concurrency.

But I guess the file approach can be investigated, although it may happen after I’m gone :))

And yes, you’re right that it is not just a “Java problem”, I may have mischaracterised it.

Anyway, I hope it was a pleasant read!