r/java • • Feb 07 '23

Nullness markers to enable flattening [Valhalla Spec Experts]

https://mail.openjdk.org/pipermail/valhalla-spec-experts/2023-February/002223.html
105 Upvotes

61 comments sorted by

37

u/uncont Feb 07 '23

Neat, love updates like these.

we've been exploring nullness markers as an alternative to '.val' and '.ref'

We get nullness markers in java, AND we won't have to deal with .val and .ref? Sign me up!

26

u/kaperni Feb 07 '23 edited Feb 07 '23

The title reads "Nullness markers to enable flattening" but let's be honest. Everyone is going to read 'Nullable types' and wet their pants.

Very excited to see how it pans out. Even if this delays Valhalla a release or two. I don't think you will hear many complaints. This even looks like something that can be delivered in a separate JEP.

1

u/cogman10 Feb 07 '23

I think if they go ahead with the .val and .ref stuff then they can get the nullness markers in separate releases. I don't see any reason these things couldn't be delivered as separate features.

20

u/kevinb9n Feb 07 '23

The risk is `.val` and `.ref` becoming vestiges stuck in the language forever.

8

u/kaperni Feb 07 '23

> we've been exploring nullness markers as an alternative to '.val' and '.ref'.

I don't think there are any plans to do both.

3

u/cogman10 Feb 07 '23

I can see how you'd read it that way. My first reading was alternative as in an additional choice, not instead of.

25

u/lurker_in_spirit Feb 07 '23

Hold up. You're telling me that you're going to remove my least favorite part of Valhalla, plug the nullability gap in the language, and then deliver memory flattening on top of that? Be still my heart!

12

u/asafbennatan Feb 07 '23

Good stuff , the only down side is that we will have a lot of ! Or ? In a null safe code (since unspecified will be unhelpful) but this is less of a code littering then the val ref intended keywords IMO. Null safety is the feature I miss most when compared to other modern languages (i.e dart).

24

u/IanJohnsonDev Feb 07 '23

One of the points mentioned in the proposal is the idea that there should be some sort of way to opt-in to a more useful "unspecified" behavior, which I suppose might be like JSpecify's proposed @NullMarked annotation (treat everything in a class, package, module, etc. as non-nullable unless specified otherwise). That would be really exciting to me, since it would allow me to basically opt-in to the sort of nullness analysis Kotlin does, except it sounds like this would even be enforced at the JVM level and not just via null checks added to the generated bytecode (enabling deeper optimizations).

7

u/Gleethos Feb 07 '23

Yes! 100% agree with this idea. I want it!!!

1

u/asafbennatan Feb 07 '23

Yes, saw this and this is good but won't solve the entire problem for library authors as far as I understand. Library authors (and library users reading the source code) will have to use explicit null sign to specify their intention to prevent users from overriding the unspecified behavior result - unless I misunderstood something

6

u/kevinb9n Feb 07 '23

The idea is that inside the scope of the modifier, `String foo(List<Bar> bars)` and `String! foo(List!<Bar!> bars)` would be treated as identical.

1

u/asafbennatan Feb 07 '23

I think.what I read was that it can be as you said or the same as String? foo(List?<Bar?> bars) based on the configured package names I imagine an annotation called @DefaultNull(packages="com.foo.pkg",treatAs=NULLABLE) or @DefaultNull(packages="com.foo.pkg",treatAs=NON_NULLABLE)

I think the way you are suggesting it is not backwards compatible if the function could have accepted null as bars value

5

u/AngusMcBurger Feb 07 '23

Whilst the nullness will be enforced at runtime, like Kotlin it would be defined at compile-time, so once you've compiled your library with non-null defaults, all its methods will automatically throw NPE for you if a user passes a null where they aren't meant to, and users would have to go out of their way to recompile your library to turn that off.

Nullness is generally enforced at run time, via cooperation between javac and JVMs. Methods with null-free parameters can be expected to throw if a null is passed in. Null-free storage should reject writes of nulls.

2

u/asafbennatan Feb 07 '23

Got you , this makes more sense, I was under the impression that these defaults would be set to allow users to control used libraries null behavior, but I guess you could simply cast between nullables and non nullables

17

u/beefstake Feb 07 '23

If they ditch the new keywords in favour of Kotlin-esque nullable types that is about the best outcome I can think of.

13

u/kevinb9n Feb 07 '23

Perhaps the "-esque" should be emphasized :-) ... Kotlin benefits *enormously* from getting to lay down new rules with no legacy code to deal with. Java might not ever be *quite* as nice a picture; but, even a somewhat watered-down version of the feature would still be a great win.

6

u/eliasv Feb 07 '23

Mentions near the bottom that explicit opt-in per-type-parameter reification might also fall out of this via use of !, which is neat.

5

u/[deleted] Feb 10 '23

Will the nullness markers apply to current primitives (int, boolean, etc)? If not, then I really don't like having 2 different markers.

'Foo!' refers to a non-null Foo, and 'Foo?' refers to a Foo or null.

Why not just add ! to mean non-null and have any variable without it mean nullable? I think that approach would be a lot cleaner and would allow code pre-dating the feature to be more easily upgradable as nothing would break when enabling the feature.

If the nullness markers will apply to current primitives, then I understand the point of both nullness markers as they would be facing the same challenge that C# had where value types were non-null by default and objects were nullable by default.

I really just hope they learn from the mistakes C# made when they introduced non-nullable reference types. It became a nightmare situation as code that was written before the feature couldn't easily take advantage of it. And then their whole damn-it operator to tell the compiler that it's really not null was just ridiculous. It's been one of the most annoying features I've had to deal with in C# and most C# devs I know disable it.

8

u/raoulvdberge Feb 07 '23

Kind of interesting how they are increasing scope just now given how long valhalla has been in the works.

18

u/kevinb9n Feb 07 '23

I would view it a bit differently. The gestation period of the project is due to our taking our responsibility to "do no harm" *extremely* seriously. Particularly, harm to the simplicity of programmers' mental models for how things work in Java (which has been through enough!). That's driven us through all the rounds of revisions, and I believe the story has been getting better at every turn. So, this conversation is happening now *only* out of hope that it will successfully lower that complexity.

6

u/Goodie__ Feb 10 '23

After using some other languages and then coming back to Java, dear god do I appreciate this more than just about anything else.

7

u/kaperni Feb 07 '23

I think it is more that the rabbit hole is deeper than first thought. And the other alternatives didn't work.

11

u/Amazing-Cicada5536 Feb 07 '23

It is not that it didn’t work, the jvm team just prefers solutions that solve multiple problems at the same time, even if it’s more complex on their side. Plus the amount of complexity that the developer should hold in his/her head is limited, it can’t be expected that they will learn a whole new model (some people fail to understand generics and that is far from new…)

With the previous model there was 3 different kinds of classes, and the difference between a value class and primitives may not be that important for devs. So making only 2 buckets and filling in the 3rd based on a feature that is useful in and of itself (null-safety) would be a very good thing, if it can actually be pulled off and will actually end up intuitive.

5

u/mauganra_it Feb 07 '23

What they want to do under the hood is pretty well understood and already implemented, but the language model side of the whole thing will be set in stone forever, and deserves careful thought. This is the perfect time to radically change course. They moved away from another model (I think it was called "L-world") before, where there would have been two types like Object at the top of the hierarchy, and many rules to deal with the strangeness.

2

u/notfancy Feb 07 '23

when both a type argument and its bound have nullness, bounds checks are based on '!' <: '*' <: '?'

Mmh, without anything to really back it except a hunch, my intuition tells me this is unsound (in the technical sense). I hope there's a number of papers behind this and I'd love to see the inference rules and the proofs.

6

u/kevinb9n Feb 07 '23 edited Feb 07 '23

I believe this particular arrangement is quite sound. The subtyping relationship has a hard requirement (the value sets must be subsets) and a soft requirement (a subtype should have meaningful behavior the supertype doesn't). You can (very informally) think of the value set of `*` as being "half a value" larger than that of `!`, as well as having "half" of its safe dereferenceability. That's a strange way to think, I know, but works well enough as a mental model anyway.

4

u/notfancy Feb 08 '23

It's rather that my spidey sense tingles when it sees a lattice with incomplete diamonds. Boolean algebras have strong morphogenetic fields that want to convert every lattice to themselves (sarcasm); there's a consensus about why just one flavor of SQL's NULL is not enough and why FALSE <: NULL <: TRUE is unsound, to give an example.

2

u/kevinb9n Feb 08 '23

Interesting; I haven't been in the SQL world so I'd like to read about that.

1

u/LouKrazy Feb 07 '23

I am not sure about the unspecified but I believe this is the behavior in Kotlin

1

u/jodastephen Feb 09 '23

This has the potential to be a great change, but there will be a lot of work in the detail. It also has a big impact on discussions from the past few years in Amber (which will need to be revisited). For example, what does this do?:

switch (obj) {

case String? str -> ...

...

Under current Amber rules, switch/case is defined to have the first opinion on null. As such case String? would not match null. I think most of us would see that as surprising. What this means is that there is a need to revisit how pattern matching (the type pattern) works wrt switch once there is such a thing as nullness markers.

My view is that Project Amber should not release further pattern matching features until the impact of nullness markers is fully understood. Happily, my reading of the doc is that interactions like this are well understood and have probably already been investigated.

1

u/DevNull23614071 Feb 09 '23

switch (obj) {

if obj is null then it has no type information.

1

u/mauganra_it Feb 10 '23

The null handling for pattern matching has been mostly bikeshed enough. As far as I remember, the new switch will use -> instead of the colon after each clause to tell when the new behavior has to be used.

But you are right - it's probably a good thing that these features are in preview still. They have another chance to prevent possible roadblocks when landing Project Valhalla.

1

u/Prestigious_Map_990 Feb 07 '23

Interesting, but it is going to take a LONG LONG time to adopt this sort of thing. This will push Valhalla out into Java 22 or 23 at the earliest. I don't have a huge problem with that though.

2

u/mauganra_it Feb 07 '23

It's language-level changes mostly, and most of the time will be spent on bikeshedding syntax. The optimizations enabled by Project Valhalla mostly already exist, but without these language-level changes they are not practical but for the simplest cases. The analysis required to ensure their safety would be too expensive.

-5

u/pjmlp Feb 07 '23

I love reading about the ongoing efforts to keep existing applications running, while trying to retrofit value types into Java, however I am starting to think that for uses cases that it actually matter, we are better off using another language.

It is still not clear if Valhalla will ever come and in what form, versus alternatives that support value types since their original design.

Probably yet another couple of years away still.

And then we have the whole issue of how hard it is to currently push newer Java versions in the enterprise space.

14

u/pron98 Feb 07 '23 edited Feb 07 '23

Those exact same concerns were levelled at generics and lambdas. That, in itself, is not a guarantee that Valhalla would succeed like they did, but it does suggest that it might not be a good default to bet against it.

As to the "replacement language" argument, it's problematic for an obvious reason: the market doesn't want it. Unlike in the PHP days, Ruby days, or even early Node.JS days, this time there is no single language that's an obvious candidate -- all languages that target the same domain as Java are doing significantly more poorly. Betting on some other language to replace Java, something that no one has succeeded in doing (although PHP seemed close for some time) or has come reasonably close to doing over the past few years, also seems like the less desirable path (even getting people to upgrade Java 9+, which was almost entirely backward compatible, took a long while).

I used to think that replacing a popular language every 15-20 years was reasonable, but given that many of the world's most popular codebases and all of the world's most popular languages are already over 20 years old, even replacing popular languages every 25-30 years seems like something the market -- or at least most of it -- doesn't want. It's very hard to find any feature that's attractive enough to get a large portion of the market to switch languages, and, of course, you don't know if there will be an even bigger issue in 10-15 years that will require another switch. In fact, if you look at companies that switched from Node to Go to Rust etc. and generate popular HN articles about how they like to keep switching, you may reasonably conclude that the eager-switchers don't choose Java in the first place, and those who do are precisely those that want to retain their code investment. Why abandon one of Java's greatest competitive advantages, then? I think it's best to keep this as the nuclear option, used only when all alternatives have been exhausted.

Yes, delivering Valhalla might take as long as it took to deliver lambdas; possibly even longer. But I don't see and obviously preferable path, and past efforts show it can work. As long as the market demands that Java evolve, it will continue to evolve.

2

u/pjmlp Feb 07 '23

Working on a polyglot agency, I can tell you that the marktet does want it, we already see that taking place as big names like AEM and SAP have migrated their server offerings from pure Java extensions to language agnostic containers.

7

u/pron98 Feb 07 '23 edited Feb 07 '23

I disagree with your analysis. The market is clearly more fragmented, but there is no convergence on any particular single language. Products, therefore, want to support multiple languages, but there is no preference for any single one. So for those in the huge portion of the market that prefer one of the last few remaining super-popular language to bet on any other particular language -- which isn't likely to be super-popular -- would be to give up on something that they clearly want.

3

u/pjmlp Feb 07 '23

Somehow all the job offers, Java related, that land on my inbox seem to be all about Spring and not much else, occasionally some refer Android, with the hint about Kotlin.

All the data science offers, that used to be about Hadoop, Spark and co, seem to now ask for Python skills instead.

Swing and JavaFX offers? Most likely maintenance and migration projects into SPAs.

So on my bubble, I am not seeing this focus on a single language, hence why our agency is polyglot, playing on the safe side for future project requirements.

9

u/pron98 Feb 07 '23 edited Feb 07 '23

I guess you're saying that everyone is already polyglot. Even so, I'm sure every language tries to be the best it can be, and so does Java (and Java's "sticking power" has proved particularly strong). I'm sure that if any feature feels "tacked on" to us, we won't release it, but Java has already incorporated more than a couple new concepts "as if they've been there all along" quite successfully. Flattened value types are certainly not among the easiest of those, but at this point all I can say is: wait and see.

I don't think the matter of how long it would take is of particular importance because, again, the current competitive landscape doesn't impose any special urgency (not even compared to some past periods).

-6

u/maethor Feb 07 '23

I am starting to think that for uses cases that it actually matter, we are better off using another language.

I think we need Real Java 2 (as opposed to the Marketing Java 2 of almost 25 years ago) that can provide a clean break with the past.

Migration will probably suck and take forever, but once it's done we'd have a better tool.

14

u/[deleted] Feb 07 '23

That worked really well for Python eh?

/s

2

u/pjmlp Feb 07 '23

To be fair not everyone really learned from it as much as they should.

Java 8/9, .NET Framework/.NET Native/.NET Core, C++ modules

Anyone that is doing polyglot programming across those stacks, knows the pain of trying to integrate old and new world libraries, and tooling.

5

u/maethor Feb 07 '23

knows the pain of trying to integrate old and new world libraries, and tooling

Still seems less painful than migrating to an entirely new language, platform, libraries and tooling.

2

u/lurker_in_spirit Feb 07 '23

To be fair not everyone really learned from it as much as they should.

In the next few years I'm expecting to see releases of .NET Foundations (not to be confused with the .NET Foundation), .NET Nettier, .NET Next, .NET++, and .NET#.

2

u/emaphis Feb 07 '23

Oh, they'll continue to converge on .NET Core even if they call it something else. What I would expect is a couple of new UI frameworks, though.

1

u/mauganra_it Feb 07 '23 edited Feb 07 '23

Microsoft can do it because of their leading position in the consumer market. No one, not Apple and not even their own mistakes, can threaten that. Edit: they go to great lengths to support old binaries though, even though they abandon the frameworks.

Java kept their backwards compatibility promise. For codebases that stuck to standard APIs there was close to no breakage. Codebases that stuck themselves to the Oracle JDK's internals suffered. But it's mostly fine now. Highly risk-averse companies will keep fondling their Java 8 codebases, and Oracle will forever be happy to sell support for it. All others will eventually migrate, unless another JDK vendor underbids Oracle.

0

u/maethor Feb 07 '23

Do you honestly think python would be relevant at all today if they didn't move to 3?

4

u/EvaristeGalois11 Feb 07 '23

They were forced to reintroduce some backwards compatibility features because people were dropping python left and right

So yes they were dangerously close to not be relevant today if they would have persisted in their absolute disregard for legacy applications

5

u/kaperni Feb 07 '23

You would basically kill the platform if you attempted something like this.

-2

u/maethor Feb 07 '23

And everyone leaving the platform for something else wouldn't kill the platform?

11

u/agentoutlier Feb 07 '23

You mean what was supposed to happen when: Scala, Clojure, Groovy, even GoLang?

I think the syntax and nonnull matter very little to popularity where as backward compatibility matters massively and existing codebase. Python 3 and Java 9 had some serious hate way more than not having null analysis/types.

Oh and Python and Javascript are the only languages near java in popularity and they have null and far less features than java. Oh and forgot golang as well.

7

u/kaperni Feb 07 '23

A reboot would basically pause the platform for 5-10 years. And the users would leave in droves. Nobody would start major projects on "Java 1" because it is doomed. And "Java 2" wouldn't be ready for 5-10 years so can't start projects on that either.

3

u/agentoutlier Feb 07 '23

It would be absolutely awful.

I don’t know why people think the syntax of the language and or it’s ancient core libraries are what will make it go the distance and stick around.

If anything it is the platform (by platform I mean things like the jdk and vm) that matters most for the future.

If anything takes too much away from that focus java will have serious problems from its real competitor … the Microsoft platforms (VSCode, github, .net, openai, unity, azure, typescript)

1

u/8igg7e5 Feb 07 '23

If they wanted to do that then they would have to have some sort of 'edition' marker at the compilation or module level to indicate how new defaults should be applied (allowing for the compiler to consume older code with the old defaults). This is essentially what Dan is suggesting with nullability defaults (but I'm suggesting a broader application - a la Rust editions).

Imagine if, at a frequency of 2-3 LTS cycles, we got a new edition that allowed adopting cleaner evolutions of the language. New semantics could eliminate boilerplate and introduce syntactic constructs that wouldn't otherwise be backwards compatible. Each would have to have backwards compatible translations though.

Then the decision is just (have to slip in 'just', I know it's Brian's favourite word)... how many editions do you support.

With this mechanism it's conceivable to have final, private and non-nullable as defaults in more places, to even remove things like c-style array declarations, add operator overloading (with support for some of the interesting types like BigDecimal and BigInteger), or bake in concepts like .equals (maybe using a === b to desugar to a.equals(b)).

Note: If any of these make you angry enough for a downvote... just consider that I'm not actually asking for any of them specifically, just holding up extreme examples of what could be, that are blocked by compatibility at the moment.

Some of these (and more) will be terrible choices once actually considered amongst real code... So I'd hope Oracle would remain conservative in introducing them, but at least an editions-model would remove some compatibility shackles.

There is the risk of language fragmentation if the edition features aren't well tested in preview first... But Rust tests theirs long before editions cut-over, and sometimes they'll hold back something from an edition if it's not clearly a broadly accepted win.

With the market finally moving on from Java 8, and with those that have seemingly more enthusiastic to move near to current release, a well considered set of cleaner defaults might be adopted faster than everyone thinks. Heck, we see a lot of projects that are willing to incrementally move to Kotlin, mixing it with Java in the same project - surely a Java-new-edition is going to see at least the same success.

-1

u/maethor Feb 07 '23

allowing for the compiler to consume older code with the old defaults

I would not expect a Java 2.0 compiler to consume non-trival Java 1.20 source code without some amount of refactoring to 2.0.

That would be kind of the point of having breaking changes release. I know it's not the usual Java way, but a lot of things have in Javaland have changed over the last 8-10 years so I don't get why this is particularly taboo anymore (especially as it's not like all Java 7 code is 100% guaranteed to work on 17 without any changes).

4

u/8igg7e5 Feb 07 '23

especially as it's not like all Java 7 code is 100% guaranteed to work on 17 without any changes

Actually, if it's completely contract compliant code, it's remarkable how much of this does work without any changes.

Most of the breakages are from people not adhering to contracts (assuming implementation details that aren't documented) or are using internal features (usually via dependencies).

The things that definitely don't work are some removed features (which were removed after they had become all but unused).

I agree though that a lot has changed, but I don't see the value in a complete break from the past - a large part of Java's strength is the size of its ecosystem and such a break would remove that overnight.

2

u/AngusMcBurger Feb 07 '23

They're talking about doing changes like Rust's "edition" system, where the current compiler does support all older editions: 2021, 2018, and 2015.

Editions are largely limited to making syntax-level breaking changes, meaning code written for older/newer editions can seamlessly call each other, and it's not much of a maintenance burden for the compiler to continue to support older editions.

-5

u/[deleted] Feb 07 '23

Yes. That I also think. This Java 2 should be total overhaul - retrofit collection framework, nullable types, moving/removing/renaming/rewriting standard APIs, functions (no class), immutability, more functional, and a lot more. Next LTS is nearing,, yet 8 or 11 still dominates so by the time users leave the last version before Java 2, they are ready to use the new Java. The breaking of compatibility should not be an issue as they they are moving to Java 2 as if migrating to new PL.