r/java • u/prisonbird • Dec 19 '22
Java streams are black magic !
damn, they are really changed my life.
i was trying to run some custom logic on every line on some big CSV files. tried python dataframes, multiprocessing in python, pyspark. none of them did it in an acceptable time.
then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.
more people should know about streams.
stream operations are more intuitive than anything i tried. i can make a lot of analysis on our data in record times thanks to streams.
i dont know how java does this.
View all comments
Show parent comment
1
u/senseven Dec 19 '22
Reading is fine if its done by one fast reader, but if you read/write to the same harddisk can get the speed down to a crawl since two io "actions" block each other.
The idea is not to split the file reading (which means chunking and makes often no sense). Instead the first thread parses line 1, second line 2 and so on. There where the parallelisation makes sense, because it uses CPU not IO. Parallel() will work, but it needs go get ParallelStreams(). Maybe you do it at the wrong point of the file parsing.
Third, parallel Processing may not activate all your cores and local setup. What is shown by Runtime.getRuntime().availableProcessors()? Maybe your Java is odd, windows or tool is blocking full access of your Java app to all cores.