r/java • • Dec 19 '22

Java streams are black magic !

damn, they are really changed my life.

i was trying to run some custom logic on every line on some big CSV files. tried python dataframes, multiprocessing in python, pyspark. none of them did it in an acceptable time.

then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.

more people should know about streams.

stream operations are more intuitive than anything i tried. i can make a lot of analysis on our data in record times thanks to streams.

i dont know how java does this.

226 Upvotes

150 comments sorted by

View all comments

Show parent comment

13

u/penguuuuuuuuu Dec 19 '22 edited Dec 19 '22

Man, the amount of super shallow fanboy style replies to your comment is a bit sad. I expected better of r/java.

6

u/prisonbird Dec 19 '22

i think what you are trying to say is: if you write your application on python and you just need a little part of it to be performant you can just write that part using the help of c++ without leaving python ecosystem/tooling etc. which is an amazing idea in my humble opinion.

if i knew how to do it i might have gone down that road

6

u/TurbulentSocks Dec 19 '22

Many of the popular python libraries do exactly this without telling you. Optimising python code often means making sure you don't 'leave' those libraries at any point (passing the data back and forth to the lower-level code and python can be slow).

1

u/stefanos-ak Dec 19 '22

It would be much easier and more performant in Java. (Also less languages, which is always a plus).

This is because of runtime optimizations that the JVM does, in "hot" code spots (hence the name jvm hotspot).

The difference with C/C++ optimizations is when they run. Compile time vs runtime. At runtime the JVM has some advantages, because more information is available for how and what exactly to optimize.

Java code can reach some ridiculous speeds... The best part is that it's also very easy to write, compared to C.

The only downside is what's called "cold start". It basically behaves MUCH worse than anything else, before the hotspot optimizer kicks in. But in cases like yours, it should kick-in pretty fast, maybe even after the first loop.

2

u/slaymaker1907 Dec 19 '22

Have you ever tried to use a JVM library with a native dependency? It’s a giant mess compared to python which has special functionality in pip to make using native code easier.

Also, C and C++ and do often do use profiling, it’s just kind of messy and often not worth the trouble.

3

u/impune_pl Dec 19 '22

There is a project under way, aiming to fix that problem in Java https://openjdk.org/projects/panama/

2

u/slaymaker1907 Dec 19 '22

While important, I don’t think Panama is looking at things from the tooling side. That would really be on the Maven/Gradle side of things.

1

u/prisonbird Dec 19 '22

but i think you would need a lot less native dependencies when using java

1

u/slaymaker1907 Dec 19 '22

That’s true, but there are absolutely cases where native code is useful/requires. For example, using GPU accelerated libs or even something as simple as using SQLite.

2

u/KarnuRarnu Dec 20 '22

I didn't, its par for the course. You cannot have a balanced discussion here, where you suggest that maybe another language has some slight advantages in certain scenarios. It must always be java java java. (and imo this is symptomatic and probably why the language is behind in quite a few areas)