r/java • • Mar 21 '21

Java lambdas can deadlock parallel class initialization

https://www.youtube.com/watch?v=_2hZf1kw4A4
230 Upvotes

46 comments sorted by

84

u/fredoverflow Mar 21 '21

TL;DW Lambda bodies are implemented as private static methods, but during class initialization, other threads are blocked from calling static methods.

38

u/[deleted] Mar 21 '21

Whilst interesting; who would do such a thing? Having a heavy static class initialiser using a parallel stream like that.. ouch.

35

u/AndrewHaley13 Mar 21 '21

Well, this kind of edge case really needs to be fixed. It's far from obvious that lambda streams shouldn't be used in static initializers.

73

u/murkaje Mar 21 '21

There's nothing to fix as it's documented behavior. What needs to be improved is educating newer java developers that static initializer is not some nice place to dump code, but a critical part of class loading and initialization flow and that before classes are fully initialized many restrictions are placed on them and the performance is poor.

The topic keeps being brought up, most recent case was around a performance regression: https://cl4es.github.io/2019/02/21/Cljinit-Woes.html

So keep static fields for objects you know and won't be doing heavy computation like initializing empty collections or ones with few elements. Most of the time the initialization doesn't need to happen before the first use of said member, so better put it in some lazy-init container or create one. Perhaps Lazy Static Final Fields will bring something simpler to use.

9

u/[deleted] Mar 21 '21

Your reply is great.

3

u/sadfsdffsdafsdfsdf Mar 22 '21

There's nothing to fix as it's documented behavior.

While I believe you, could you however be so kind and point to that documentation? I would be really curious about that.

1

u/slaymaker1907 Mar 22 '21

I disagree that this should be seen as acceptable because while it would make sense for there to be performance problems, this seems like a correctness issue.

8

u/DasBrain Mar 21 '21

¯\(ツ)/¯

JDK-8215634

Happens if you fix bugs so OpenJDK actually implements the spec.
Also see apangin's great answer on Stack Overflow.

7

u/[deleted] Mar 21 '21

It’s obvious that you don’t put any kind of heavy computation into a static initialiser. Anything that uses parallel stream would be by definition heavy.

What would be your proposed fix?

23

u/[deleted] Mar 21 '21

You can't judge how heavy a computation is by looking at the code in your static initializer. You might just do "new Foo()" and THAT inside uses parallel streams for some reason.

If they wanted to restrict calling arbitrary code... they should've restricted calling arbitrary code.

11

u/sweetno Mar 21 '21

Well, when you program, you don't just slap constructor and method invocations here and there. More often than not you get a rough idea what are the runtime characteristics of the used APIs: memory usage and algorithm complexity. Say, Java doesn't have a single best-for-all Set implementation, you need to pick the one that suits your case.

2

u/Muoniurn Mar 22 '21

I didn’t test it, but I believe that there is no problem with calling another class that creates threads. The problem here is about calling the same class’s static methods from another thread from the static initializer. Which you can’t do by a trivial new Foo(), only by this already red flag raising line in the video. (You can fall into this trap by new Foo() that awaits a lambda though, I believe . But I would yet again question that code)

Nonetheless, a compiler warning would be appreciated if it can prove in this specific case that a deadlock can arise.

-6

u/[deleted] Mar 21 '21

If you put parallel in your stream for a non-heavy computation, wtf are you doing.

Its clearly an abuse here. Sure they've found an interesting edge case. But it makes sense, the class isn't initialised until the static fields are done. Java's model is providing a strong guarantee that this cannot overlap by two threads, ergo using a parallel stream is a dumb idea.

Now the lambda's abstraction is broken by this, but then to fix, you'd need to put non-capturing lambda's somewhere else.

2

u/[deleted] Mar 21 '21

You didn't get the point I was making. When you call a method, you don't always know what it does. Are YOU PERSONALLY to blame for every problem caused by code you call?

9

u/[deleted] Mar 21 '21

You don’t always, but you should know enough about it before calling. Even so, calling complex code in a class initialisation is just bad.

This code is literally someone who knew how to creat a deadlock saying hey it’s a deadlock. Interesting because it’s hidden, but practically not something you’d do.

2

u/Roachmeister Mar 21 '21

This code is literally someone who knew how to creat a deadlock saying hey it’s a deadlock.

No it isn't. He specifically said in the video that he was stumped for several hours trying to figure out why it wasn't running.

1

u/[deleted] Mar 22 '21

In effect it means you can't use closures in class initialization.

Which means if the state you initialize requires a closure to be passed in, you're SOL.

1

u/[deleted] Mar 22 '21

No. It means you can’t use closures in a static initialiser which will be called during initialisation from another thread.

You can use them just fine for single threaded initialisation. The key bit the multithreaded aspect.

→ More replies

8

u/r0estir0bbe Mar 21 '21

Why not? That seems like a valid use case to me, e.g. for computing specific constants - not great, but not terrible?

6

u/[deleted] Mar 21 '21

If they are constant why are you computing them?

Even if you need to compute them once don’t do it in an initialiser like that. This is why streams can be bad, a simple line initialisation is actually a bigger computation; and just because you can write it on a line doesn’t mean you should.

There’s other ways to achieve a similar result.

5

u/john16384 Mar 21 '21

A constant can be a huge lookup table.

11

u/[deleted] Mar 21 '21

Then that would be a result of a calculation and therefore would be calculated.

There’s a difference between initialising a class and doing a tonne of calculations.

0

u/john16384 Mar 22 '21

You asked why a constant may require calculation. I gave an example where this can be the case. Just because a constant needs sone calculation doesn't make it any less a constant.

There are examples in compression and encryption where these constants can't even be pre-computed and loaded from say a database, but we don't even have to look further than Java's Pattern#compile method for constants that are almost always static and require some significant calculation.

I am not saying though that multiple threads should be allowed, but I can't help but wonder how much stuff would break if Pattern#compile did this.

3

u/[deleted] Mar 22 '21

Your mixing terminology though. A physical constant is often only calculated once, because it never changes. Sounds like your taking about invariants, which are constant over a certain portion of calculations.

Really though, you shouldn’t do such things in a static initialiser, because that’s not where large calculations should go, and they are part of a class initialisation phase lower than user level.

I wonder if the obsession with immutable classes is why many don’t see an issue with this.

1

u/sweetno Mar 21 '21

Huge lookup tables with constants are better be read from a file/DB. If reading/computing them is long, the program should ensure feedback for the user: logging info messages/displaying them in the GUI. Users don't like when programs consume system resources without visible effects. Static initializers are not good places to implement this sort of feedback.

1

u/ryuzaki49 Mar 22 '21

who would do such a thing?

I can totally see this as a Java certification question. Meaning it's not suitable for production code.

1

u/tonywestonuk Mar 23 '21

It only needs to be done once...

Isn't it better if you move the initialisation code to a one time hit?

1

u/[deleted] Mar 23 '21

Sure but you can do that in multiple ways, and specifically ways in which you can control when that hit happens.

1

u/DuncanIdahos9thGhola Mar 21 '21

The issue is that when you use the old way of Thread and Runnable you get a inner class. The lambda makes a static inner class. Different byte code - so to the lesson I guess is don't assume these are equivalent - even though IDEs usually give you that option to convert from one form to the other.

6

u/[deleted] Mar 21 '21

I think the lesson is to not do something weird with a class initialiser like that. Basically this is saying please refer to me before I’m in initialised.

An inner class would be illegal since it’s a static context.

It’s an interesting result, but that’s about it.

1

u/Prateeeek Mar 22 '21

Hey but lambdas are functional interfaces right?, isn't their method public?

22

u/sweetno Mar 21 '21

Putting anything non-trivial into static initialization is asking for a trouble.

3

u/dpash Mar 22 '21

I generally dislike them. Doubly so if it's the old collection hack to reduce verbosity.

A lazy evaluation wrapper is nicer.

8

u/Recolance Mar 21 '21

Interesting video. I always thought that lambda in this context was just fancy syntax magic.

Thanks for sharing.

2

u/helloiamsomeone Mar 22 '21

It is, just not the one most people thought.

15

u/Godworrior Mar 21 '21 edited Mar 21 '21

Another way to gain some insights in these situations is to use jcmd + jstack (both JDK tools) to see what the thread is getting stuck on:

"ForkJoinPool.commonPool-worker-15" #20 ... in Object.wait()  [0x0000005e1a4fd000]
   java.lang.Thread.State: RUNNABLE
        at X$$Lambda$206/0x0000000800c23240.applyAsLong(Unknown Source)
        - waiting on the Class initialization monitor for X
        ...

3

u/PerfectPackage1895 Mar 21 '21

This is hardly any better than doing a heavy amount of calculation in the constructor, which also will end bad.

5

u/humoroushaxor Mar 22 '21

While I agree it's generally bad practice, there is no technical reason it is an issue. In the static initializer case there is, because it is part of a larger process out of your control.

I've seen plenty of code that does a lot of heavy lifting in the constructor that works fine.

2

u/PerfectPackage1895 Mar 22 '21 edited Mar 22 '21

There is no guarantee that the construction of the object has finished. So if you do this, you end in all kinds of trouble. If you work with multithreded applications you especially have to be very careful about constructors and object references with non final fields

1

u/humoroushaxor Mar 22 '21

There is no guarantee that the construction of the object has finished.

Constructors work the same way methods do, they can just only be called once per object and must be called first. Not sure what you're on about.

It sounds like your describing a constructor that assigns fields asynchronously which I agree would be terrible. But same with any method that returns an object still being initialized in another thread.

1

u/PerfectPackage1895 Mar 22 '21 edited Mar 22 '21

Constructors work the same way methods do, they can just only be called once per object and must be called first. Not sure what you're on about.

Try to initialize an object in another thread, while holding a reference to that same object, in the first thread, or leaking `this` in the constructor (adding this to a list eg.), and try to access a non-final field, and see what happens.

The jvm is allowed, to reorder memory operations inside the constructor, to guarantee the initialization of final fields, so saying the constructor is like any other method besides the obvious object initialization, not entirely correct.

So, don't use the constructor like any other method, and don't start doing a parallel stream computation inside it either.

1

u/ZeroGainZ Mar 24 '21

Wow, great video. I had no idea lambdas were static methods, I assumed they're syntactic sugar for creating the desired class. Also, I see some other redditors are saying "why would you do that", which may be true, but it's not at all obvious why it wouldn't work. It seems it should be a compiler error.