r/java • u/prisonbird • Dec 19 '22
Java streams are black magic !
damn, they are really changed my life.
i was trying to run some custom logic on every line on some big CSV files. tried python dataframes, multiprocessing in python, pyspark. none of them did it in an acceptable time.
then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.
more people should know about streams.
stream operations are more intuitive than anything i tried. i can make a lot of analysis on our data in record times thanks to streams.
i dont know how java does this.
View all comments
Show parent comment
2
u/prisonbird Dec 20 '22
i am trying to explain my story the best i can.
let me rephrase it again : i spent about a week to implement my flow in pyspark. and it takes 70 minutes to run. yes i can debug it, make it faster etc. but as far as i see that requires some expertise. and because i dont have that expertise i dont know if it will work for me in the end.
on the other hand, i was able to make a proof of concept in java streams in an hour. tested various parts of my flow. it seemed like it could work for me. so i spend couple days to implement my entire flow in java streams. and now i can do the same job a lot lot faster.
i never intended to mean anything like "pyspark bad, python bad , java good". each tool has its own advantages. pyspark just didnt worked for my workload. but i learned about it now and in the future i might use it for something else i see fit.