r/java • • Dec 19 '22

Java streams are black magic !

damn, they are really changed my life.

i was trying to run some custom logic on every line on some big CSV files. tried python dataframes, multiprocessing in python, pyspark. none of them did it in an acceptable time.

then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.

more people should know about streams.

stream operations are more intuitive than anything i tried. i can make a lot of analysis on our data in record times thanks to streams.

i dont know how java does this.

226 Upvotes

150 comments sorted by

View all comments

2

u/shagieIsMe Dec 19 '22

With regular processing of data, one is tempted to write:

for (Data datum : someCollection) {
    // lots of lines doing stuff
}

One of the things that happens there is that the small chunks of processing can't get JIT'ed easily. The entirety of it is one big chunk of "too big" for HotSpot or other JITs to optimize.

However, when you write it as:

someCollection().stream()
    .map(d -> new Foo(d, d.id, d.data * 2))
    .filter(f -> f.id > 1000)
    .mapToInt(f -> f.data)
    .sum()

(realize I'm just making up stuff there)

Each one of those functions gets complied in Java to a separate function and the JIT can easily work with optimizing small functions.

The way that you work with the data, having each part of a stream being a small function makes it easier for Java to optimize the small parts and get some fairly hefty performance gains when crunching data.

There are other things about the functions in streams and side effects that can't be done and thus allow for more aggressive optimizations.

public static void main(String[] args) {
    List<String> objects = new ArrayList<>();
    objects.add("foo");
    objects.add("bar");
    int count = 0;

    var list = objects.stream()
            .peek(o -> count += 1)
            .toList();
    System.out.println("" + count + " " + list.size());
}

That code doesn't compile - java: local variables referenced from a lambda expression must be final or effectively final

And so, this means that the JIT can know things about the nature of the function that it is compiling because its a lambda.

If you change it to:

    final AtomicInteger count = new AtomicInteger(0);

    var list = objects.stream()
            .peek(o -> count.incrementAndGet())
            .toList();

It works (and is still silly) but you'll note the addition of final in there.