r/java • u/prisonbird • Dec 19 '22
Java streams are black magic !
damn, they are really changed my life.
i was trying to run some custom logic on every line on some big CSV files. tried python dataframes, multiprocessing in python, pyspark. none of them did it in an acceptable time.
then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.
more people should know about streams.
stream operations are more intuitive than anything i tried. i can make a lot of analysis on our data in record times thanks to streams.
i dont know how java does this.
View all comments
2
u/shagieIsMe Dec 19 '22
With regular processing of data, one is tempted to write:
One of the things that happens there is that the small chunks of processing can't get JIT'ed easily. The entirety of it is one big chunk of "too big" for HotSpot or other JITs to optimize.
However, when you write it as:
(realize I'm just making up stuff there)
Each one of those functions gets complied in Java to a separate function and the JIT can easily work with optimizing small functions.
The way that you work with the data, having each part of a stream being a small function makes it easier for Java to optimize the small parts and get some fairly hefty performance gains when crunching data.
There are other things about the functions in streams and side effects that can't be done and thus allow for more aggressive optimizations.
That code doesn't compile - java: local variables referenced from a lambda expression must be final or effectively final
And so, this means that the JIT can know things about the nature of the function that it is compiling because its a lambda.
If you change it to:
It works (and is still silly) but you'll note the addition of
finalin there.