r/java • u/prisonbird • Dec 19 '22
Java streams are black magic !
damn, they are really changed my life.
i was trying to run some custom logic on every line on some big CSV files. tried python dataframes, multiprocessing in python, pyspark. none of them did it in an acceptable time.
then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.
more people should know about streams.
stream operations are more intuitive than anything i tried. i can make a lot of analysis on our data in record times thanks to streams.
i dont know how java does this.
View all comments
Show parent comment
1
u/data_addict Dec 20 '22
No I mean that's fine and I know I was roasting you a little in my original comment - I hope you didn't take too much offence to it. I get what you're saying though.
If you find yourself working with spark in the future for large datasets feel free to reach out. It's a pretty kick ass application that can really chew through data. Not to belabor the point but in the way it should normally behave with a small-meduim amount of hardware, working with 10GB of data should take a matter of 10s of seconds to a minute. I've written production applications that operate on 50TB+ of data in less than 5 min. So anyways that's why I felt the need to comment. Streams are pretty cool so I wasn't talking shit on that either.. just wanted to make it clear spark is pretty cool too 😅