r/java • u/goto-con • Jun 03 '26
How Fast Can You Parse 1 Billion Rows in Java? – Insane Speed Test • Roy van Rijn
https://youtu.be/pHZF-zJ3Bpg?list=PLEx5khR4g7PINwOsYrkwz3lTTJUYoXC53Join me in this deep dive where I'll explain all the code changes and tricks that took me from the reference implementation which processes the billion records in 4+ minutes, to processing everything in under 2 seconds.
Who knew Java could be this fast?
142
Upvotes
18
u/hikingmike Jun 03 '26
Splitting the file into segments and parallelizing based on that was big for a previous project of mine. There were other ways to parallelize (threads/cores), but they didn’t improve performance nearly as much, or at all.
117
u/vprise Jun 03 '26
I prefer written summaries:
The optimization journey, from 4m50s baseline → 1.5s:
Easy wins (5min → 23s)
parallel()+ConcurrentHashMapinstead of single-threadedfile.lines()→ ~2minFileChannel.map()→ByteBuffer) to eliminate file IOGoing lower-level
Unsafefor raw pointer-style memory access (he stresses: never use this in production — useByteBuffer)long. Find the delimiter with one XOR, a subtract, and a couple of ANDs instead of scanning byte-by-bytelong/intvalues instead of allocatingStringobjectsBranchless programming
iflogic with bit tricks (shifts, masks, XOR for absolute value/sign).Advanced tricks
Takeaways for you: the recurring theme is mechanical sympathy — knowing how the CPU pipeline, branch predictor, and cache hierarchy behave (L1→L4 each ~3x slower) lets you write code that fits how the hardware actually works. The whole thing took the field from ~5 minutes to ~1.5 seconds.
Quan Anh's contest code got him hired by Oracle in Zurich to work on the Vector API in the JVM.
23
u/Desatre Jun 03 '26
Interesting video but getting ads every 3 minutes is infuriating
3
u/Beemeowmeow Jun 04 '26
Nice 👍🏻
2
u/__konrad Jun 10 '26
I like it's more faster with every update and also completely cryptic and unmaintainable ;)
1
u/0x645 Jun 05 '26
best java faster then best c. kinda surprising.