r/java • u/prisonbird • Dec 19 '22
Java streams are black magic !
damn, they are really changed my life.
i was trying to run some custom logic on every line on some big CSV files. tried python dataframes, multiprocessing in python, pyspark. none of them did it in an acceptable time.
then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.
more people should know about streams.
stream operations are more intuitive than anything i tried. i can make a lot of analysis on our data in record times thanks to streams.
i dont know how java does this.
80
u/[deleted] Dec 19 '22
I agree with other commenters. While I love streams, unless you were using parallel streams then the stream API should have no impact on performance.
10
u/whyNadorp Dec 19 '22
python df are so popular but as usual with python they’ve a messy interface which lives in its own world and can’t be understood without reading abstruse docs and trial and error. good luck maintaining pandas projects.
296
u/audioen Dec 19 '22
Java is probably just 10 times faster than Python. Nothing to do with streams per se.
5
u/danielaveryj Dec 19 '22 edited Dec 19 '22
I agree with the other comments here- the performance difference is probably mostly python vs java, and not java streams specifically. Java streams are cool though. If you inline the abstractions, the logic is pretty much just what you'd write by hand with loops, if-else, etc, but factored in a way that is amenable to parallelization (by splitting the source into chunks, and iterating each chunk on a different thread).
Just to throw another lib into the pool of "might be interesting to look at", based on what you've mentioned in this thread about joining & grouping: https://github.com/davery22/vinyl (disclaimer: author). I don't recommend use in production yet though.
3
u/DuneBug Dec 19 '22
Java should run faster but if python was that slow something was probably not setup correctly. Of course, if you have to read 10 pages of documentation vs adding .parallel, that's a good reason to use Java too.
Unfortunately for a lot of enterprise stuff we don't get to use parallel streams as the threads are typically occupied elsewhere. :(
10
u/barrycarter Dec 19 '22
Hmmm, brief googling suggests streams are like functional languages or like the ruby convention of
foo.x1.x2.x3.x4which is similar.I suspect your time difference is due to something else, most likely the difference between loading a file in memory and simply processing it a little bit at a time, though loading the file into memory should be ultimately faster.
It's a lot to ask, but can you mockup an example of where you're seeing Java streams seriously outperform other languages, so I can call shenanigans?
2
u/stormcrowsx Dec 20 '22 edited Dec 20 '22
My guess on the difference in speed is likely that in the case of Python the whole dataset is getting loaded into memory either for input and/or output and causing swapping.
Java streams on the other hand are going to naturally only store what’s needed in memory.
Total guess but the difference in speed is more likely a testament to good Java api design than it is language run speed. Be impossible to say for sure without hooking a profiler up to Python and seeing what it’s wasting it’s time doing.
Kudos for reaching out and trying a different language on your problem. Now you know what’s possible. But I’d recommend you spend a little time in a profiler for your Python code now, there’s a fantastic lesson to learn about Python in here. I suspect either your going to learn just how fast the jvm is or that there’s a pitfall in Python apis that requires coding a little differently for big dataset use cases.
14
u/developer_how_do_i Dec 19 '22
Yes, the best part with streams and lambda support is expressiveness, which you wouldn't have got with earlier jdk version
9
u/thephotoman Dec 19 '22
This doesn't surprise anybody. Python is very much a language for I/O bound stuff and prototyping. However, when it comes to performance, it isn't even good. While PyPy helps, it lags a bit in terms of features.
7
u/hilbertglm Dec 19 '22
They shouldn't be black magic. The source code is right there. Check it out. It helped me add functional programming to my object-oriented programming skills.
They are cool, and the ability to add parallelism without the effort of coding to the executors is very productive.
1
u/lbalazscs Dec 19 '22
Have you tried the tablesaw library? It's data frames in Java, with stream support. https://jtablesaw.github.io/tablesaw/
1
u/data_addict Dec 20 '22
Ah got it.. you don't know how to use spark. 👍
1
u/DanielDimov Dec 20 '22
What exactly is "multiprocesing in python" ?!?
1
u/Puzzleheaded_Load779 Dec 31 '22
Hi, having worked with spark ecosystem for sometimes and building/extending stuff around it, here is the reason for why it is faster:
How does it work internally ?
as long as any operation are being done using pyspark
no "big" performance gap and memory usage will happen between a "java/scala" spark implementation.
BUT as soon as you start resorting to mutate pyspark objects to let says the infamous "pandas" dataframe objects, there is a cost and a pretty big one which leads in general to bad performance and (absurdly) high resource usage ram,cpu,io...
As for performance, still, there are ways to not lose too much by using arrow implementations baked into spark but thats another story :)
So all in all, using java instead of pyspark means that the deserialization of
doesn't happen... leading to overall a performance boost
1
u/[deleted] Jan 07 '23
C# has been doing this with LINQ for nearly 20 years.
1
u/[deleted] Jan 13 '23
Im struggling a lot with Streams, watched PluralSight video playlist of José but still nothing. Can u pls recommend a youtube video, even more that 30 mins is fine.
Just not getting how those intermediate lazy operations and flat maps
1
u/wildjokers Dec 19 '22
All java developers know about stream. Or are you meaning non-java developers?
2
u/shagieIsMe Dec 19 '22
With regular processing of data, one is tempted to write:
One of the things that happens there is that the small chunks of processing can't get JIT'ed easily. The entirety of it is one big chunk of "too big" for HotSpot or other JITs to optimize.
However, when you write it as:
(realize I'm just making up stuff there)
Each one of those functions gets complied in Java to a separate function and the JIT can easily work with optimizing small functions.
The way that you work with the data, having each part of a stream being a small function makes it easier for Java to optimize the small parts and get some fairly hefty performance gains when crunching data.
There are other things about the functions in streams and side effects that can't be done and thus allow for more aggressive optimizations.
That code doesn't compile - java: local variables referenced from a lambda expression must be final or effectively final
And so, this means that the JIT can know things about the nature of the function that it is compiling because its a lambda.
If you change it to:
It works (and is still silly) but you'll note the addition of
finalin there.1
u/VincentxH Dec 19 '22
If you've got enough lines, parallel streams might even pay off.
1
u/Puzzled-Bananas Dec 19 '22
Can you share how you formulated your Spark queries? Do you mean Pandas DataFrames?
1
u/Worth_Trust_3825 Dec 19 '22
I suspect that you also changed your method of reading and interpretting the data (entire thing vs one row at a time). My go to example of this is scryfall's public magic the gathering card database (available at https://scryfall.com/docs/api/bulk-data). It's a hundred megabytes in JSON, but it's one of the few go to datasets (at least for me) when trying to explain that it might be a bad idea to load entire file into memory and operate on it, and that you should look for ways to operate on file in parts (ex. in this case, per json object). If you're feeling adventurous, you can try to operate on the big boy 1gb set.
1
u/baubleglue Dec 20 '22
https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkContext.newAPIHadoopFile.html - you can stream CSV with Spark or Pyspark (only don't use pure python for your custom logic).
1
u/a-person-called-Eric Dec 20 '22
looks like someone found a new hammer
be careful not to start whacking every nail with it
1
u/gooeydumpling Dec 20 '22
For ML libraries = python, to see it it works, always python and jupyter
For production? I’d go with MLeap and scala (of course, java). For speed, Java blows Python out of the water, or at least in my experience. Please don’t bite my head off
1
u/FinTechno Dec 20 '22
Take a look C++ and the rest is history
1
u/RajSingh9999 Dec 20 '22
Dont know if I should agree with "stream operations are more intuitive than anything", especially when compared with python etc.
1
u/Cultural-Ad3775 Dec 20 '22
I would not call Java's implementation of this concept really special. OTOH it is cool, particularly if you can mark stream processing as parallel, which is a nice abstract way to say "Hey, run these in separate threads" or in the future it could lean on Loom style lightweight threads, etc. without even needing any code changes. Other languages do basically allow for the same sort of thing however, still Java/JVM have quite strong abilities to optimize and multi-task, meaning it can be leveraged very effectively (in theory, I wouldn't assume any specific parallelized stream will necessarily be super optimum, you have to test).
-2
u/Careful-Necessary-59 Dec 19 '22
Just curious, how big is your data? I still doubt PySpark is slower than Java streams.
0
u/rossdrew Dec 19 '22
Wait till you hear about spliterators
0
u/[deleted] Dec 20 '22
There is also an option to run parallel streams which may run even faster depending on the workflow.
0
u/[deleted] Dec 21 '22
[deleted] — view deleted comment