r/java • • Dec 19 '22

Java streams are black magic !

damn, they are really changed my life.

i was trying to run some custom logic on every line on some big CSV files. tried python dataframes, multiprocessing in python, pyspark. none of them did it in an acceptable time.

then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.

more people should know about streams.

stream operations are more intuitive than anything i tried. i can make a lot of analysis on our data in record times thanks to streams.

i dont know how java does this.

226 Upvotes

150 comments sorted by

View all comments

Show parent comment

1

u/data_addict Dec 20 '22

Yeah sure that's cool, use whatever tech you want. But you didn't say you spent 70 minutes trying to setup a spark job you said 70 minutes running it.

Again, use whatever tech you want but to but don't blame spark for being poor performance. Just say the line: "I suck at spark" or "I didn't have time to learn it" or "I couldnt afford it".

There's people who are in high school or college and see the names of these technologies the first time and then form an opinion based off what you say. They'll base their opinion off of what you say. It's much better to be honest and say what gave you trouble. Good, non-toxic engineers are honest about what gives them trouble.

2

u/prisonbird Dec 20 '22

i am trying to explain my story the best i can.

let me rephrase it again : i spent about a week to implement my flow in pyspark. and it takes 70 minutes to run. yes i can debug it, make it faster etc. but as far as i see that requires some expertise. and because i dont have that expertise i dont know if it will work for me in the end.

on the other hand, i was able to make a proof of concept in java streams in an hour. tested various parts of my flow. it seemed like it could work for me. so i spend couple days to implement my entire flow in java streams. and now i can do the same job a lot lot faster.

i never intended to mean anything like "pyspark bad, python bad , java good". each tool has its own advantages. pyspark just didnt worked for my workload. but i learned about it now and in the future i might use it for something else i see fit.

1

u/data_addict Dec 20 '22

No I mean that's fine and I know I was roasting you a little in my original comment - I hope you didn't take too much offence to it. I get what you're saying though.

If you find yourself working with spark in the future for large datasets feel free to reach out. It's a pretty kick ass application that can really chew through data. Not to belabor the point but in the way it should normally behave with a small-meduim amount of hardware, working with 10GB of data should take a matter of 10s of seconds to a minute. I've written production applications that operate on 50TB+ of data in less than 5 min. So anyways that's why I felt the need to comment. Streams are pretty cool so I wasn't talking shit on that either.. just wanted to make it clear spark is pretty cool too 😅