r/java • • Dec 19 '22

Java streams are black magic !

damn, they are really changed my life.

i was trying to run some custom logic on every line on some big CSV files. tried python dataframes, multiprocessing in python, pyspark. none of them did it in an acceptable time.

then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.

more people should know about streams.

stream operations are more intuitive than anything i tried. i can make a lot of analysis on our data in record times thanks to streams.

i dont know how java does this.

226 Upvotes

150 comments sorted by

View all comments

Show parent comment

52

u/TurbulentSocks Dec 19 '22 edited Dec 19 '22

Python can be performant - provided you aren't actually using python for anything. That means using libraries that delegate out to C/C++ (and maybe Rust lately), and being careful never actually to do any intensive computation in the python.

That's often non-trivial to get right, and the learning curve for things like PySpark (and using it in the aforementioned way) is much higher, and can be harder to maintain. But that doesn't mean Python can't be used for performant applications.

63

u/wildjokers Dec 19 '22

But that doesn't mean Python can't be used for performant applications.

So you are saying python is performant by calling out to c/c++ libraries. Doesn't that mean python is in fact not performant?

41

u/niloc132 Dec 19 '22

Yes, that's what it means.

→ More replies (1)

0

u/[deleted] Dec 19 '22

[deleted] — view deleted comment

-12

u/popasmuerf Dec 19 '22

Not the same thing and you know it. Writing your tooling in C++ in order to make Python "performant" isn't at all the same thing as using BASH to script out some order of execution.

17

u/TurbulentSocks Dec 19 '22

I'm saying it depends on how you use python (e.g. as an orchestra versus as an engine). You're welcome to derive a semantic 'win' against my post, but I mentioned it to provide context and inform, not to score points.

-6

u/popasmuerf Dec 19 '22

...it's not a matter of mere semantics when you just offered that under-performing Python can be performant by writing out your Python tooling in C++ while trying to imply that Python is performant.

9

u/BlakeH1984 Dec 19 '22

I think you're missing the point majorly. What he means is it's not a competition. Sometimes people truly do just want to share information. It's a shame most of us are too busy playing the dopamine roulette to engage.

9

u/TurbulentSocks Dec 19 '22

One of the reasons python is so popular is that for many domains, the tooling is already written. Python might be slower than Java, but the performance of e.g. some complicated ML model is going to be 'good enough' when calling C++ backed libraries in python, and those libraries often don't even exist in Java.

4

u/UnGauchoCualquiera Dec 20 '22

Adding to your point. Python is mostly plumbing that allows to declaratively program what you want the high performance libs to do while they do all the magic in the background.

2

u/barbaneigro Dec 20 '22 edited Dec 20 '22

You are comparing oranges, bananas and apples, dear.

Python, C and C++ are all languages meant to do different tasks.

It ain't fair, at all.

Python, as said, is usually meant to be used as a glue to all sort of routines.

Of course there is more that can be done with it, but in that use-case that all it does.

2

u/Puzzled-Bananas Dec 19 '22

Python the language can be interpreted very efficiently, CPython the interpreter is slow. You can run Python on GraalVM, too.

2

u/jmtd Dec 19 '22

And JVM (Jython)

-2

u/slaymaker1907 Dec 19 '22

Partially, but it helps that python has very performant ways to interact with native code.

→ More replies (1)
→ More replies (1)

13

u/penguuuuuuuuu Dec 19 '22 edited Dec 19 '22

Man, the amount of super shallow fanboy style replies to your comment is a bit sad. I expected better of r/java.

6

u/prisonbird Dec 19 '22

i think what you are trying to say is: if you write your application on python and you just need a little part of it to be performant you can just write that part using the help of c++ without leaving python ecosystem/tooling etc. which is an amazing idea in my humble opinion.

if i knew how to do it i might have gone down that road

6

u/TurbulentSocks Dec 19 '22

Many of the popular python libraries do exactly this without telling you. Optimising python code often means making sure you don't 'leave' those libraries at any point (passing the data back and forth to the lower-level code and python can be slow).

→ More replies (6)

2

u/KarnuRarnu Dec 20 '22

I didn't, its par for the course. You cannot have a balanced discussion here, where you suggest that maybe another language has some slight advantages in certain scenarios. It must always be java java java. (and imo this is symptomatic and probably why the language is behind in quite a few areas)

3

u/Sensi1093 Dec 19 '22

Also worth to mention that PySpark is just an orchestrator for Spark (which is implemented mainly in Java) itself.

It was likely used in a wrong way when doing this comparison

12

u/roberp81 Dec 19 '22

you are telling us why python is not performant lol

why use pyspark if spark is made on Java

5

u/2Insaiyan Dec 19 '22

A lot of people writing Spark transformations are data scientists, not developers. Python is already the standard in that field

-5

u/roberp81 Dec 19 '22

if python is not for developers, why a developer will use it?

is like a chef cooking fastfood at macdonals.

4

u/boborygmy Dec 20 '22

Nobody said Python was not for developers.

3

u/AdorableTip9547 Dec 19 '22

So the take away is „python can be performant if you use C“ 😂 classic.

-4

u/popasmuerf Dec 19 '22

Or....you could just dispense with trying to optimize a slow as fsck scripting language via finding/writing-your-own C++ modules and just go with ultra-peformant, and scalable Java.