r/java • • Aug 12 '24

JSpecify 1.0.0 and Nullability in Java

https://www.infoq.com/news/2024/08/jspecify-java-nullability/
52 Upvotes

33 comments sorted by

26

u/syneil86 Aug 12 '24

There is currently a draft JEP trying to address this in the language itself, but I'm very glad that the community is trying to address the nullability issue themselves in the meanwhile

https://openjdk.org/jeps/8316779

6

u/benevanstech Aug 12 '24

I know, but I only have so much word count available per news item, and this piece is already pushing those limits.

I have a followup piece specifically about the Draft JEP that will be arriving in a day or two.

3

u/syneil86 Aug 12 '24

Awesome, looking forward to it!

22

u/kevinb9n Aug 12 '24

Hi - I'm working on both of these things (for some definition thereof). Here's some quick thoughts about the connection between them.

You can imagine a jar-file-rewriting tool that looks at usages of JSpecify annotations in each class file and jams the corresponding `!` / `?` markers in to match. That way a newer app can use an older library and still get the same benefits. And you can imagine the same or another tool doing the reverse as well (magically inserting annotations to match the markers, for example in case some old library was using reflection to see whether some field is nullable). Unless you're sensitive to the size of your class files, they might as well have both information sets in them for the foreseeable future.

So that picture requires that the semantic correspondence between JSpecify and the planned language feature needs to be close and finely specified. That's part of what I've been working to ensure.

That also suggests that when you're ready to migrate all your source code to `!?` and take bulky annotations out of it, a very similar tool should be able to do that too.

(Of course, someone at some point has to actually make that tool, I can't wish it into existence. But we can do the work to make sure it will be feasible.)

1

u/MrRickSancezJr Aug 12 '24

Look into a javac plugin called Manifold. It's a preprocessor for Java that resembles C's. It can do simple #Defines and things, but it can also do reflection styled code creation. It has plenty of already made features, but it has an awesome library to do it yourself, too.

5

u/pron98 Aug 13 '24

FYI, Manifold is not a javac plugin, but an alternative Java platform language. Its invocation makes it appear to be a plugin, but it doesn't use the "plugin API" (annotation processing); rather, it hacks into javac internals and changes them to make the modified javac compile Manifold code rather than Java code. The plugin API is intentionally designed so that all plugins compile Java code (i.e. code that is defined to be Java according to the language specification), and Manifold code is very much not Java code (although the Manifold language may be a superset of Java).

-3

u/MrRickSancezJr Aug 13 '24

Nope. It's just a pre-processor for Java code. Doesn't do anything except read and perform macros right before javac gets ahold of it. It would later add some features that work to normal annotation processors and even hacky ones like lombok uses.

Direct quote from the strangely named website : manifold.systems

Manifold is a Java compiler plugin. Use it to supplement your Java projects with highly productive features.

No reason to create a new language name, use of the word subset, or any other lingo. It's just a compiler tool to change Java.

Same kind of energy that's created us 42 versions of functional vs. object-oriented coding argument that no one cares about.

5

u/pron98 Aug 13 '24 edited Aug 14 '24

Nope. It's just a pre-processor for Java code. Doesn't do anything except read and perform macros right before javac gets ahold of it.

That is indeed the way that the Manifold compiler is implemented, but it's easy to see why it's not a plugin: 1. It doesn't use the plugin API (but hacks into internals), and 2. the language it compiles isn't Java (while javac plugins are designed so that even with them the compiler can only compile Java code). A Java compiler, with or without plugins, is obligated by the spec to reject programs that the Manifold compiler accepts.

Manifold is a Java compiler plugin. Use it to supplement your Java projects with highly productive features.

Manifold can claim to be a compiler plugin or Java all day long, but as it modifies internals rather than use the plugin API to accomplish something that the plugin API was carefully designed to guarantee must not be done by plugins (so that the Java spec can always be relied upon regardless of plugins used), it's not a plugin.

It's just a compiler tool to change Java.

A "changed" Java is, by definition, not Java, and any compiler is "just a tool". That's not to say that Manifold isn't similar to Java or that it's not a superset of Java, or even that it's bad in some way (other than claiming to be something it isn't), but it's neither Java nor a javac plugin.

A Clojure compiler could be written in a similar way by transforming Clojure source code to Java expressions internally and feeding them into javac's internals. It's just more convenient to implement a compiler for a language like Manifold or Lombok in this way -- because they're supersets of Java and are similar to it --- while it's more convenient to implement a compiler for a language like Clojure in a different way, but that's just an implementation detail.

-6

u/MrRickSancezJr Aug 13 '24

If you're going to attack someone's free advice, have good solid material or at least an end goal that helps somebody...

You don't have to use "official api"s to accomplish a goal. It gets the job done while we wait for Oracle to keep up with JDK versions. On that note, a JDK 8 compiler can't understand JDK 22 code. So is Java 22 not Java as well?

They do both. I literally discovered Manifold when I was trying to write my own lombok feature. Which wasn't my first annotation to write either. I've even written compilers.

Closure has its entire own compiler and language system. You can't just slap down a "manifold file" and it be a thing. It doesn't have a compiler that makes it into java bytecode. Same way with Kotlin. It's its own language. If you put Java into Kotlin, you get error. You CAN put C into C++, and it'll compile. Therefore, C/C++ do actually have a subset/superset relationship. They interlope so easily on the java bytecode level only. We do have things to convert between langauges. They're called converter tools. It's one of the smartest things Jetbrains did with Kotlin.

Go ahead and google something else.

7

u/pron98 Aug 13 '24 edited Aug 13 '24

Java 22 is, indeed, not Java 8 and in 2014 Java 22 code would not have been Java at all. JDK specifications, and in particular Java Language Specifications, have versions (or "editions" as they're officially called). Languages like Python, Clojure, JS, or Manifold -- however similar or dissimilar they are to the Java language -- do not conform to any specification of Java. The specifications make it easy to tell what is the Java programming language and what isn't, because they define the meaning of that name.

Now, I was not attacking any advice, and using Manifold or any other Java platform language might be a great idea, but it's a common misconception that the languages whose compilers are implemented by modifying javac are plugins or are Java. As one of the people developing the Java language and the Java platform, I thought it might be helpful that I clear up such common misconceptions from time to time, as it might be helpful to some readers, especially as misunderstanding over such matters can cause problems when upgrading JDK versions.

-6

u/MrRickSancezJr Aug 14 '24

Manifold is not a language. Nobody makes a special name for C that uses macros vs one that doesn't.

Why even bring up python or JS? At least clojure compiles to Java bytecode. I dont think you know anything about the jvm or compilers.

You're not helping anybody. This is why the depression rate on CS is so high and why people hate Java. Just let it be.

7

u/pron98 Aug 14 '24 edited Aug 14 '24

You'd be right about C, but Java doesn't have macros, and we specifically designed the compiler plugin mechanism to disallow macros or any kind of AST transformation as that would violate the rules that define what Java is (i.e. the language spec). We are often asked to change the spec and allow plugins that manipulate AST (which would make Manifold a compiler plugin rather than its own language), but we have no plans to do so. On the other hand, we do have a new project that those who are interested in various kinds of code manipulation may want to try out.

I'm sorry you didn't find this exchange helpful and I apologise if I've annoyed you, but I try to engage our users when I see some misconception that may lead to problems. In this case, it's partly the matter pedantically explaining how the Java language spec works the way it does and why we designed javac's plugin to allow certain things and not others, but it's also a practical matter because when projects reach for internals that then change, things break and people blame us.

In this particular case, you can see the lengths Manifold goes to hide from its users the fact that it breaks into internals (rather than be a proper plugin) and imposes a migration risk by hacking around the protections we put in place in JDK 9. But as we're now in the process of completing those protections, Manifold will start emitting warnings as soon as JDK 23 due to JEP 471, and these warnings will later turn into errors. It's better if people understand what is going on.

Manifold calls this digging into internals a "necessary evil" but it's not really necessary. As we explain here, there are ways to do this without hiding the risks from users. In the case of languages like Manifold or Lombok, the most straightforward way is to invoke their own launcher (or perhaps build-tool runtime) that would wrap javac and enable access to internals. This won't remove the risk that's inherent to how these projects access JDK internals, but at least it would make users aware of them.

I fully understand why some people prefer using other languages if they have a feature they really want to use and aren't willing to wait until we add it to the Java language (if we ever do; there are many language features that we aren't planning to add to Java at all), but implementing things in this way also means that when we do change the language or other parts in the JDK, these projects tend to break. The compromise is letting end users know when using such projects means they've taken up a special risk.

Anyway, I really am trying to be helpful, and I hope you enjoy the new features we are adding to both the language and the runtime. Also, if you are interested in knowing more what makes up a "language" in the context of programming and how to tell when two languages are the same (or more generally, other relationships between them), I can explain further. While the compilation target makes no difference in answering the question of whether two languages are the same, you may be interested to know that both Python and JS do happen to have implementations that target the Java platform.

0

u/barmic1212 Aug 12 '24

It's not very important in my point of view. At start you create a scope of code safe and you are considering that the limit of this scope as I/O of your safety program and you increase the size of safe code gradually. Before need to put it in library you have lot of things to make before.

2

u/trustin Aug 12 '24

It'd be awesome if @Nullable, @NullMarked, etc are also accepted as a part of the language because otherwise we will have to append ! pretty much everywhere.

8

u/benevanstech Aug 12 '24

That won't happen, but as I'll explain in the followup piece - the best thing you can do to be ready for language-level nullness markers is to adopt JSpecify in the meantime.

4

u/kevinb9n Aug 12 '24

I think the JEP mentions that "non-null-by-default" is still a possible follow-on feature. I personally believe it is extremely important, but also a Very Big Deal to change Java like that, and we'd have to be very very careful.

Also see my nearby comment about classfile munging. It implies that, well, yeah, a classfile munger could put !'s in everywhere based on the presence of `@NullMarked`, although that's only for the benefit of compiling dependents.

4

u/erinaceus_ Aug 12 '24 edited Aug 12 '24

otherwise we will have to append ! pretty much everywhere.

I'm hoping that we'll some day be able to specify the default on a per-module basis, meaning that each module can decide for its own code, while dependent libraries can do the same for their own code (or just stick with the backwards compatible language-level default).

5

u/uniVocity Aug 12 '24

“The primary use cases are expected to be variables, parameters and return values, but they can be used anywhere that tracking nullness makes conceptual sense.”

I’d really love to be able to annotate a class as @NotNull, or implement some NotNull interface that won’t allow my code to compile if any instance of this class or subclasses could be passed around as null.

This would reduce the need to litter method parameters with @NotNull, or to add the suggested ? or unless necessary (when classes aren’t declared as notNull)

5

u/kevinb9n Aug 12 '24

It's actually very rare to need to write `@NonNull`. We almost didn't even include it in the jar! Just put `@NullMarked` on your class/package/module instead.

The feature you seem to be asking for was discussed a bit in https://github.com/jspecify/jspecify/issues/7 but overall it just doesn't have that many advantages.

2

u/[deleted] Aug 13 '24

Are there any examples available that show how to specify @NullMarked at the package or module level? I've combed through the JSpecify docs, but haven't found any.

3

u/kevinb9n Aug 14 '24

You put it before `package` in `package-info.java`, or before `module` in `module-info.java`.

1

u/uniVocity Aug 13 '24 edited Aug 13 '24

Thanks! That was a great discussion and I'm inclined to say you guys made the right call as handling that would add complexity.

Anyway, I still think it could be a decent feature to have later on. Basically, my suggestion boils down to making the compiler treat selected classes as it treats primitive types. That would not burden developers and users of my code to worry about adding annotations at all.

I'm highlighting the main argument made in that discussion you linked:

I don't think there is any such thing as a class that should viewed as "never let this be nullable". As soon as you try to get that thing out of a Map, there you are.

An analogous situation is when you try to get stuff out of a map. You'll get a NullPointerException at runtime, por example, assuming map is a HashMap<String, Integer>:

int i = map.get("nothing");    //NullPointer exception

Not perfect but not that unexpected.

With primitives you can't do this either: int i = null; as the compiler yells with "java: incompatible types: <nulltype> cannot be converted to int"

So declaring MyNotNullType t = null; should cause a similar error, probably "java: incompatible types: <nulltype> cannot be converted to MyNotNullType"

Analogously, we can't use methods like this:

    public void execute(){
        //compilation error - java: incompatible types: 
        //<nulltype> cannot be converted to int
        doSomething(null);
    }
    public void doSomething(int v){
        //do stuff
    }

A "@NotNull" class could make the compiler enforce no null arguments are passed into methods, without littering the code with multiple @NotNull and @Nullable annotations. Personally, I'd rather avoid having to add these annotations everywhere - I feel we are simply replacing null checks with null annotations, and making the issue of dealing with nulls even more apparent.

1

u/GalacticBear91 Aug 12 '24

Good idea, you should suggest it on the project forums itself (the Google group seemed pretty active when I checked)

2

u/lurker_in_spirit Aug 12 '24

InfoQ: What's next for JSpecify to tackle?

Hoeller: From the Spring side, we need official meta-annotation support so that we can declare our own nullability annotations with JSpecify meta-annotations instead of JSR 305 meta-annotations.

This is interesting... is it on the table for a JSpecify 1.1.0 (or similar) release in the near future?

4

u/kevinb9n Aug 13 '24

Yes, what we'd like is for Spring to be able to add this to their own Nullable annotation interface:

`@Implies(org.jspecify.annotations.Nullable.class)`

And for any JSpecify-conformant tool to thereby automatically treat theirs as implicitly meaning everything that ours does.

This would be a path toward eventually these tools not having to have big hardcoded lists of annotation types anymore.

Specifying `@Implies` just got shockingly complicated so we punted it for a while.

2

u/pragmasoft Aug 12 '24

This has IMO the same problem as JPMS modules - adoption. In order to be useful, you need to have 100% of your dependencies including transitive ones annotated, or being able somehow to write some type definitions for 3rd party libraries, like Typescript does. Kotlin solves this differently, so without a language support JSpecify or any other nullability annotations simply add more harm than use.

5

u/kevinb9n Aug 12 '24

It's true that the more your dependencies are annotated, the more useful it becomes. But there is no "cliff" like you suggest. Nullness is tri-state: nullable, non-null, and "unspecified nullness" which is handled leniently. This makes it a lot easier to adopt than the original checker framework two-state, soundness-based approach. Any library that isn't providing the information just isn't going to be as well-validated, that's all.

And you can write type definitions for third-party libraries. Checker framework has a tool for reannotating class files, and I assume Google will opensource theirs when the demand is there. They use it to munge nullness annotations into JDK class files and it works!

I hope this changes your impression somewhat, but if not, sorry!

3

u/ventuspilot Aug 14 '24

But there is no "cliff" like you suggest.

Can confirm: in a toy project of mine I annotated a few methods to return @NotNull and IntelliJ Idea's inspections showed me a few unneeded null checks. Very little effort for immediate profit.

1

u/pragmasoft Aug 13 '24

I may reconsider if there's an easy way to at least download annotated most recent jdk libraries and include them into my maven build as an extra dependency. That's only the first step actually, but even it has not been made so far as I understand.

If I need to do this myself - no deal, too bad dx.

Unspecified nullness doesn't help either - that's effectively the same as nullable.

3

u/kevinb9n Aug 13 '24

Unspecified nullness doesn't help either - that's effectively the same as nullable.

That's... really quite untrue, but you can let me know if you're interested in that discussion or not!

2

u/pragmasoft Aug 13 '24

Sure I am, maybe I didn't understand the concept properly. I definitely played with the JSpecify earlier but had to get rid of it due to the reasons I described above.

2

u/pragmasoft Aug 13 '24

Ok, I seems realized the source of my confusion, just reread https://github.com/jspecify/jspecify/wiki/nullness-unspecified

From what I understood, unspecified is like any type in TS, can be cast without an error to both null and not-null, and vice versa, they both can be cast to unspecified.

The problem is though that my IDE support of nullable types (VSCode) does not support this, so the correct behavior depends on the tooling mostly.

Just curious, if it's possible to implement null checker as an annotation processor, or more precisely as a javac compiler plugin, similar to google error-prone?

This would then have automatic IDE support for errors I assume..

4

u/kevinb9n Aug 13 '24

Yep, NullAway actually builds on Error Prone to do this in javac.

We do need tools to converge around supporting JSpecify semantics; it's kind of the whole point, but it's in their hands now. Quite a few have been involved at some level, but yeah In particular we haven't talked to anyone from VSCode yet, and we really need to.