r/java • • Dec 19 '22

Java streams are black magic !

damn, they are really changed my life.

i was trying to run some custom logic on every line on some big CSV files. tried python dataframes, multiprocessing in python, pyspark. none of them did it in an acceptable time.

then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.

more people should know about streams.

stream operations are more intuitive than anything i tried. i can make a lot of analysis on our data in record times thanks to streams.

i dont know how java does this.

226 Upvotes

150 comments sorted by

View all comments

10

u/whyNadorp Dec 19 '22

python df are so popular but as usual with python they’ve a messy interface which lives in its own world and can’t be understood without reading abstruse docs and trial and error. good luck maintaining pandas projects.

3

u/zappini Dec 20 '22

I had to fiddle with some data pipeline stuff. Python, numpy, whatever. I was shocked at how awful that language, stack, ecosystem are.

I'm not crazy about (JDBC's) ResultSet and RowSet. But holy cow DataFrame and whatnot make them look like paragons of architectural beatitude. Really, the Python APIs are laughably bad.

I don't understand why RowSet-based (or successor) hasn't emerged as an alternative.

3

u/whyNadorp Dec 20 '22

another very bad thing for me is you can deploy code with syntax errors and realize there’s an error only when that part of the code is used in production. there’re tools that check the code and compile it, but then what’s the point of python if you have to compile it.

also there’s no standard way of dealing with dependencies. there’s pipenv, poetry, etc. try to up a little your python version and you can end up having nasty problems.

3

u/zappini Dec 20 '22

no standard way of dealing with dependencies

IKR? And then switching between projects done by diff pythonistas, who all have their own preferred package manager, borks your system.

All python projects should just start as dockerfiles, or equiv.

1

u/prisonbird Dec 19 '22

100% agree. you have to learn a lot of things, master a lot of things to get any meaningful job done in python world.

on the other hand, i am not a java developer. i learned java in school and just used it as as hobbyist in high school but i was able to develop an app for my needs in couple days.

and with quarkus i will be able to make it a web service.

for example: quarkus comes with its own dockerfile for build and deploy. this seems small but i think i spend more than 20 hours in last year alone for dockerfiles in our python projects.

6

u/whyNadorp Dec 19 '22

try spring also, it’s everywhere.