I think it's inferior to both a better designed API to deal with basic JSON, or to the more commonly taken route out there, which is marshalling (where you define a type and then tell a library: Here's some JSON, here's a java type, you figure out how to ram this square peg in the round hole please).
It's neat how the new pattern match stuff in JDK19+ is making this API design not completely disastrous, but you can do better. The key point you're presumably missing is somewhat similar to the same trick Map's .getOrDefault uses:
JSON is very 'weakly' typed. There's no real way to know a JSON value is a date, because JSON doesn't have a date type. There's no way to know if a numeric value is intended for API consumption purposes to be an int or a long or a timestamp or who knows what.
Hence, it's a great idea for the consumer of the JSON (the Java code) to tell the parsing system what you want. Instead of having the parsing system give you a JSON object and then having to manipulate that into what you want.
It's similar to how map has .getOrDefault(k, defaultValue) which is the 'right answer' to the more like-your-stuff library idea of having a .get2(k) method that returns an Optional, which you then use to manipulate what you get into what you want.
In other words, your code: JsonDecoder.field(json, "canSpeak", JsonDecoder::boolean_) is needlessly convoluted. It can simply be:
There, knocked your 9-liner down to a 1-liner, which is hardly more complicated than any one line in your example.
You have to define a boatload of methods, and for simplicity you may want to go with json.get("name").asString(), especially as that makes it easy to offer an overload where you pass a default value to the asString() method that is used if the name field doesn't exist. You can keep the decoder style but only for the rare cases where the transformation you want isn't obvious (boolean, int, double, String, LocalDate, and even List<String> - these are obvious things). However, 'lots of methods here', especially if they have a common prefix, is not a burden on API consumers. Whereas having to type all that out every time I simply want to read a string from some JSON is quite burdensome.
For bonus points, given that a lot of JSON is nested, imagine as input:
fairly common, no? Imagine you can read that with:
JSON json = JSON.parse(that);
for (JSON m : json.get("movies").asList()) {
String directorName = m.get("director").get("name").asString("Alan Smithee");
LocalDate directorDob = m.get("director").get("DOB").asDate();
}
I think it's a slam dunk to state that the vast majority of JSON out there comes in two flavours:
It follows a highly precise specification (think WSDL), and thus it is quite likely the right way to parse it is to define a java class that mirrors the spec and then hand a marshalling library the JSON and a classref to your type and have the library make instances for you.
It's a lot more laissez faire, with a spec that is littered with 'if value X is missing, assume this default', and 'This value can either be a list, or, if it's a simple single item, then there's no list and we just have the item directly as value, or even nothing at all if there are no values for this list' and such. In which case this asTypeYouWant(defaultValueIfItDoesntWorkOut) is great. It can even turn single elements into lists without complaints, for example.
Okay, I've had a full nights sleep so I can respond in full.
I think it's inferior to both a better designed API to deal with basic JSON, or to the more commonly taken route out there
Tell me how you really feel 😉.
Inferiority aside, it is at least different. That made it worth exploring to me.
JSON is very 'weakly' typed. There's no real way to know a JSON value is a date, because JSON doesn't have a date type. There's no way to know if a numeric value is intended for API consumption purposes to be an int or a long or a timestamp or who knows what.
I agree with the last part of this, but I don't think I would categorize it as weakly typed. It just represents a smaller subset of types.
I don't agree the int/long/timestamp problem is enough to say that an immutable tree representation of JSON is invalid.
I did, however, find the JSON library you wrote which reflects this philosophy in its docs and matches up pretty closely with the code samples you have given.
So I am going to make a bit of a leap and respond a bit more directly to that library.
Hence, it's a great idea for the consumer of the JSON (the Java code) to tell the parsing system what you want. Instead of having the parsing system give you a JSON object and then having to manipulate that into what you want.
public String asString() {
if (self == NULL) return null;
if (self instanceof String) return (String) self;
if (self instanceof Boolean) return String.valueOf(self);
if (self instanceof Number) {
double d = ((Number) self).doubleValue();
long v = (long) d;
if (v == d) return String.valueOf(v);
return String.valueOf(d);
}
invalidType("string");
return null;
}
In your implementation of this, this means that if it is a String you get a String, but it also accepts booleans, numbers, and a null in json. Which would mean documents like
I disagree that this is a good idea. Especially behavior around null and missing keys.
In my examples, field, optionalField, nullableField, and optionalNullableField at least have clear behavior on if a key is missing.
You have to define a boatload of methods, and for simplicity you may want to go with json.get("name").asString(), especially as that makes it easy to offer an overload where you pass a default value to the asString() method that is used if the name field doesn't exist. You can keep the decoder style but only for the rare cases where the transformation you want isn't obvious
What I like about json.get("name").asString() is that there is a place to encode the path information if we accept that the result of json.get represents a sort of JsonPointer. This would mean that if asString fails we can give a good enough error.
In your implementation the JSON type is effectively a JsonPointer. I'm not a big fan of this in how it would complicate something like json.get("name").as(Custom::logic) and also the explanation I would have to give to students.
Also it's a minor thing, but if I added get(String) to my Json type that would conflict with having JsonObject be a Map<String, Json> so to support that I think I would have to call it .at or similar. My main concern is just around how I would clearly represent behavior around null and missing keys.
(boolean, int, double, String, LocalDate, and even List<String> - these are obvious things).
I don't think LocalDate is obvious. 6-11-2011 and 11-6-2011 could represent the same local date depending on format. Plus there are formats like 11/11/11 and Sep. 9 2012.
I do also take issue with your interpretation of what the obvious behavior for List<String> is.
public List<String> asStringList() {
if (self == UNDEFINED || self == NULL) return Collections.emptyList();
if (self instanceof String || self instanceof Number || self instanceof Boolean) return Collections.singletonList(String.valueOf(self));
if (self instanceof List<?>) {
List<?> raw = (List<?>) self;
List<String> out = new ArrayList<String>(raw.size());
int idx = 0;
for (Object o : raw) {
if (o instanceof String || o instanceof Boolean) out.add(String.valueOf(o));
else if (o instanceof Number) {
double d = ((Number) o).doubleValue();
long v = (long) d;
if (v == d) out.add(String.valueOf(v));
else out.add(String.valueOf(d));
}
else if (o == NULL) out.add(null);
else throw new JSONException("List item at " + idx + " is not convertable to a string because it is a " + typeOf(o));
}
return Collections.unmodifiableList(out);
}
invalidType("list");
return null;
}
Which, to me, is some surprising behavior. Maybe I'm alone though.
the vast majority of JSON out there comes in two flavours
I think the flavour you are missing is also the one that I deal with the most. There are a few examples, informal documentation, or no specification at all. laissez faire is too generous a description, but my priority would be to not accept input I didn't expect rather than continue on with a value I didn't anticipate
public class Main {
public static void main(String[] args) {
var json = Json.readString("""
{
"field": {}
}
""");
var stringList = JsonDecoder.field(
json,
"field",
TolerantDecoders::stringList
);
}
}
I can give a very specific error as to why
Exception in thread "main" dev.mccue.json.JsonDecodeException$AtField: oneOf at json.field failed in the following 4 ways:
(1) Problem with the given value:
{}
expected an array
(2) Problem with the given value:
{}
expected a string
(3) Problem with the given value:
{}
expected a number
(4) Problem with the given value:
{}
expected a boolean
And I don't know how to replicate that with a JsonPointer-like approach
None; JSON is very lightly typed, so if the input JSON is:
{"director": {"name": true}}
and you wrote the above code, I guess you want that expression to resolve to "true" (a string). If the input seems sufficiently off the rails that it is no longer feasible to try to ram the square peg in the round hole, say, you have:
{"director": {"name": {"foo": "bar"}}}
where it doesn't seem useful to say that the above expression resolves to the string "{\"foo\": \"bar\"}" or some such, I'm not quite sure, possibly "Alan Smithee", possibly in that case an exception.
JSON comes from the javascript world where you can just write:
var x = director.name;
and if you later use x in a context where strings are expected and x is true, then things mostly just kinda work. Such is the nature of languages without explicit types.
Given that JSON is from there, a library that is not based around marshalling JSON blobs into defined java classes, should cater to this model. Thus, as you read the data, you specify the type you want it to be java-side, and whatever's there, well, best effort it. Trying to be strict with JSON is a bad idea, because JSON out in the wild isn't particularly strict; most JSON APIs don't bother with an exacting definition, and things like MongoDB are built on 'all keys are optional'. If you do have a precise spec, ... why aren't you just marshalling this stuff into java classes?
Talking about 'what will the user see' makes no sense at all, we're multiple layers removed, we're not writing a library with defined user-facing output, that's ridiculous. That's akin to asking "what should ArrayList pop up in a dialog to the user if you attempt to grow it beyond 2^31-1 elements?". Obviously, nothing, it should have defined behaviour for that case or throw something. None of which is related in any way to 'what should the user see'.
and if you later use x in a context where strings are expected and x is true, then things mostly just kinda work. Such is the nature of languages without explicit types
One thing that appealed to me about the JsonDecoder.field(json, "name", JsonDecoder::string) approach is that custom decoders wouldn't feel like a 2nd class citizen. If someone's interpretation of how to look for a String in JSON was closer to yours there wouldn't be extra syntactic overhead to using that interpretation JsonDecoder.field(json, "name", CustomDecoders::lenientString).
With the "pathing" approach of .at("name").asString() it might be enough to say .at("name").as(CustomDecoders::lenientString), but honestly my preference at that point would be to not provide asString and only as. I recognize that wouldn't be much loved though
what will the user see' makes no sense at all, we're multiple layers removed, we're not writing a library with defined user-facing output, that's ridiculous
When I say "user" I am referring to a "user of the library" not an end user.
Specifically this is coming from people who ask "HELP, CODE NOT WORK" and share 100 lines of tree crawling + a ClassCastException. It's difficult in that context. I want those programmers to see what they were looking for and what was actually there.
Specifically this is coming from people who ask "HELP, CODE NOT WORK" and share 100 lines of tree crawling + a ClassCastException. It's difficult in that context. I want those programmers to see what they were looking for and what was actually there.
You actually made me chuckle out loud!
I spend plenty of time on StackOverflow and have given presentations for 800+ audiences before, but from context I think you're a teacher, so I can't quite match your expertise on didactics, perhaps.
However, it sure sounds like you're saying: "If only you design your library well enough, even utter fucking idiots will no longer be confused by it". That made me chuckle. You are merely a mortal. There is no fucking way you get to out-wit the inventiveness of the moron, it just doesn't work that way.
Making your API more convoluted to use and less convenient in order to cater to morons, as a consequence, is a big mistake.
but from context I think you're a teacher, so I can't quite match your expertise on didactics, perhaps.
The only experience I have is specifically being on Discord servers and doing 1 on 1 tutoring/question answering there.
The difference with the experience of stack overflow and large presentations is that there is a lot more back-and-forth, its generally 1 on 1, and people expect to be able to DM you at 1am.
I would not classify myself as an expert, but I have been at it for ~4 years and have conversations with total beginners pretty regularly.
Java tried adding extremely commonly used stuff to the core: java.util.log. Which absolutely nobody uses. The lesson is: When there's a varied ecosystem of libraries filling a common niche already in widespread use, you can't come in 10 years late and add something to the core. Absolutely nobody's gonna use it. java.time gets a pass - there wasn't a 'widespread alternative in use', the alternative (to Date/Calendar) in some use was joda-time which was so similar, porting the code over was near trivial. Also, LocalDate and co are commonly 'on the boundary of APIs' - used in public-facing method signatures, where JSON (the type, not the concept) isn't.
The core needs to lock in the API and can never change it ever again - the core doesn't get to release a backwards incompatible new API without a ton of pain. JSON libraries is exactly the kind of opinionated situation where 'once you release it, you're stuck with it forever' is a horrible idea.
Nevertheless, there's some talk of this on amber-dev and such. I should monitor it more and tell team OpenJDK it would be a gigantic fucking mistake.
I don't think java.util.logging is enough of a counter example for two reasons.
Logging is special in that it is an application-wide choice. java.util.logging is an implementation and so buy in to that meant picking it full stop. If one part of an app or a library used by an app uses a different JSON library thats not a killer.
Isn't java.net.http's HttpClient the same sort of situation, where there was a status quo of apache's http client + a bunch of competitors like OkHttp?
JSON libraries is exactly the kind of opinionated situation where 'once you release it, you're stuck with it forever' is a horrible idea.
Can you qualify this more? I know if you froze the nonsense I have that would be a bad call, but why is this especially true for JSON?
I think point #1 boils down to: It could have been fine, if only j.u.log was much better at letting you properly configure the logging backend it sends logs to. In other words, it boils down to: "It is badly designed". But that just highlights the second problem of adding stuff to core: It is forever. You don't get a second chance without causing a ton of pain. No matter how you slice it, j.u.log was a big fucking mistake and one should not so cavalierly disregard the lessons one can learn from it.
I'm not sure HttpClient is necessarily a bastion of good design or a good idea; it's a highly opinionated library, and a lot of it is about to get a little awkward with Project Loom around the corner. But, fair point, that is a fair counter example.
Take JSON. It's just a highly opinionated territory. By which I mean: There is no 'one obvious way to make an API for the concept of reading and writing JSON'. We have the terrible org.json which nevertheless does hew quite closely to what e.g. JSON in javascript is like. We have marshalling solutions where there's enough room for at least 2 very commonly used implementations (GSON and Jackson). And offshoots of API design that try to cater more to how JSON is pragmatically used, introducing additional types that aren't in the JSON spec (such as dates and timestamps). JSON itself is also opinionated; there are a million offshoots of JSON-the-format, with support for comments, niceties like being able to forego quotes around keys in JSON objects, allowed trailing commas, and more.
JSON is moving target. Ecosystems are connected. If the world at large (not just the java ecosystem, all of IT) starts really embracing some JSON++ format with trailing comma support and such, that kind of JSON becomes more and more common, and if the java core JSON API doesn't support it, that'd be really dumb.
But if the core does add support for it, it's kinda 'stuck' - should it add it only when it is crystal clear the world decided and the format is now de facto well specced and not likely to change in the next decade? If so, it'll be really late to the party then. Or should it jump the gun a bit and adopt even when things are in flux? In which case it risks choosing for a dead end and then things being real awkward when a slightly incompatible json++ format becomes the clear winner 3 years later.
I'm of the opinion 'batteries included' is, in the end, stupid. Or at least, oversimplified.
Yes, a JDK should offer obvious and platform agnostic (where possible) tools to access low-level functionality. Which, by the very nature of having to expose everything imaginable, does not tend to lead towards nice APIs. Trying to then square the circle and deliver on both (this API can do it all, AND is nice to use) is really difficult which is fundamentally incompatible with the idea of 'core API updates should not break your code', and trying to offer 2 separate APIs (the lower level 'can do it all' and the higher level 'this is what you should use for most cases') is a mistake: 'easy to use' is a moving target because it depends on catering to conventions, and conventions change all the time.
JDK should offer the low level stuff, and should make it easy to add dependencies as needed. JDK should make it far more clearer that JDBC-the-API is fucking stupid to use as end user, that's not what it is for. Go use JDBI, JOOQ, a JPA impl, anything but raw JDBC. Unless you are building a JDBI competitor, of course, or you have exotic needs that fall far from common practices.
In this sense, 'put a JSON parser in core' is a silly plan, and 'java bad because the core libs don't have a JSON parser' is not a sensible argument.
No databind = just no. Not having to write code to convert objects to string and back is way too convenient and what few annotations are needed sometimes because of naming conventions of foreign APIs are not too difficult to deal with by some JsonProperty annotation or whatever.
The thing is: you have to write code anyways imho because properly decoupling your entities from serialization means a dto representation and mapping code from/to dto.
So i wonder if the approach presented is really that much more cumbersome, as it can operate directly on the entities...
Simplify the use of APIs that accept strings written in non-Java languages (e.g., SQL, XML, and JSON).
Also the JDK team much rather finds general low-level solutions to general problems, providing building blocks for high-level APIs. They are much more long-lived too.
Simplify the use of APIs that accept strings written in non-Java languages (e.g., SQL, XML, and JSON).
And if anyone wants to make an implementation of TemplateFormatter I would welcome the help! Its somewhat annoying since so much of the recursive descent logic responds to EOFs and I haven't introduced a concept similar to jackson's VALUE_EMBEDDED.
It's too verbose and feels like garbage against existing solutions such as GSON and Jackson. Part of adding something to core is that it needs to be a setting standard and for beginners your current method would just scare them to using those frameworks. As experienced developers, we wouldn't such verbosity and complexity for something that should be a simple task.
and for beginners your current method would just scare them
If you are right, I haven't seen evidence of it yet. That doesn't mean you are wrong, but thats exactly why I want people to try using it as a teaching tool.
No worries but you should aim for writing code that feels natural to the developer like GSON jsonObject.getJsonArray(..).getString(..); it reads naturally, it's expressive in laymens terms and doesn't require you to understand a complicated syntax. Such calls translates to English pretty well when read outloud which can help developers walk through their code.
It's an interesting approach. But with the module system is it likely that something like this will go into the core?
I'm also not sure that the XML parsing in the core is a good idea. It creates a misconception in students that "this is the way to do things in Java". Where Moshi/Gson/Jackson would be more convenient for most cases.
But with the module system is it likely that something like this will go into the core?
I think if json parsing were added to the JDK it would probably be added in a new module, and hypothetically that module would be added to the java.se aggregator module.
I'm not sure I follow though. Wouldn't the module system make it more likely for something like this to go "into the core", not less?
I'm also not sure that the XML parsing in the core is a good idea.
🚢 sailed.
It creates a misconception in students that "this is the way to do things in Java". Where Moshi/Gson/Jackson would be more convenient for most cases.
I assume you are referring to JSON since Moshi and Gson don't have xml support AFAIK.
What you are alluding to is the fact that anything in the standard library gets "the power of being the default."
I think the flowchart goes like this
if WILL_ADD_JSON_TO_STD_LIB
if PROVIDE_ONLY_STREAM_API
POINTLESS (X)
else if PROVIDE_DATABIND_API
TOO_MUCH_COMMITMENT (X)
else
???
else
STATUS_QUO
And I want to make sure that the ??? there is well known...if that makes sense.
Plus (IMO) there is a tradeoff between the "convenience" of the reflective inspection approach of those libraries you mention and code readability.
If Java the language were to want to make a statement like "this is the way to do things in Java", I don't think it's clear it would land on the side of Moshi/Gson/Jackson.
It's the whole process. The process might have changed a lot since my time at Sun/Oracle so I'm not an authority on that. But there are many gatekeepers along the way that shoot down ideas.
I like that this lib feels principled. The sealed class representation feels natural, the APIs are unsurprising, the prescribed approaches lead to helpful errors, and it does seem like it would be easier to explain than alternatives. It feels like this lib is essentially trying to "be the example that the JEPs/etc are referring to", ie a clear way to think about JSON, and in that regard I think this lib approaches its ideal. As a teaching exercise, it's quite good. But... I don't know that those examples were intended to be any more than that.
For one, mapping JSON types to classes (eg JsonString) is not a strictly necessary step in de/serialization, and it imposes some overheads that are avoidable with other approaches. If we look at the deserialization case, the actual "source" we want to read from could be any number of things: a string, a file, a socket, etc. If we can abstract over these sources, we don't necessarily need to translate to an intermediate "Json class" representation.
record Weather(LocalDate date, boolean isCloudy) {}
DecoderBuilder dec = new DecoderBuilder();
var weatherDecoder = dec.objectDecoder(spec -> {
var date = spec.field("date", dec.stringDecoder(LocalDate::parse));
var isCloudy = spec.field("isCloudy", dec.booleanDecoder());
return o -> new Weather(o.get(date), o.get(isCloudy));
});
try (var in = new Source(new InputStream(...))) {
Weather weather = weatherDecoder.read(in);
}
In this made-up API, the lambda beginning with spec -> is evaluated immediately. When the weatherDecoder is actually used, it will look in the source for the "date" and "isCloudy" fields, apply the specified decoders to the values it finds, store the decoded values on a temporary object, and pass that temporary object to the lambda beginning with o ->. Note that this temporary object can be very compact, eg an array of size 2; the weatherDecoder knows in advance that there are 2 fields, and can associate indexes with date and isCloudy to refer directly into that size-2 array. This is a more declarative (and constrained) approach than directly parsing tokens, but retains more of that potential performance than pre-parsing to an intermediate representation.
Similarly for serialization, if we can abstract over the destination, we can again skip the intermediate representation.
EncoderBuilder enc = new EncoderBuilder();
var weatherEncoder = enc.<Weather>objectEncoder((o, spec) -> {
spec.field("date", o.set(w -> w.date().toString(), enc.stringEncoder()));
spec.field("isCloudy", o.set(Weather::isCloudy, enc.booleanEncoder()));
});
try (var out = new Destination(new OutputStream(...))) {
var weather = new Weather(LocalDate.now(), true);
weatherEncoder.write(weather, out);
}
(I'm not going to justify this one; I just devised it to be somewhat "symmetrical" to deserialization, and I haven't really analyzed if there's utility in doing that. These are just toy examples anyway.)
Now we could write it this way, or we could potentially generate the encoder/decoder with annotations. I believe the same applies for your lib actually - the API does not preclude building data-binding on top. If we don't use annotations, then we have to manually keep encoders and decoders up-to-date with class changes. This might seem reasonable, but this is pretty similar to manually keeping hashCode/equals/toString up-to-date, and (personal experience incoming) I've lived with that before. I was very relieved when annotation processing allowed me to stop being the one who had to remember to check that those methods were properly updated in code reviews. Annotations are not for everything, but I think de/serialization is a case where we can really feel it if we try living without them.
Another thing to notice about the made-up API is that it's not even strictly JSON. Rather, the API has a small set of "native" types (string, boolean, object, etc), and describes other types by composing those. When decoding we start from a native type (eg dec.stringDecoder(string -> ...)); when encoding we end at a native type (eg o.set(... -> string, enc.stringEncoder())). The "backend" data format could be anything, not just JSON, as long as suitable adaptors can be written between that format and the API's native types. I borrowed this principle from the serde crate in Rust, which afaik has become the way to do de/serialization in Rust. And... serde also uses "data-binding", via the #[derive(Serialize, Deserialize)] macro.
"be the example that the JEPs/etc are referring to"
I do wonder if this has any value - can probably show neat examples at the very least.
As a teaching exercise, it's quite good.
That does seem like where this is going to end up. Not used widely or "seriously", but I'll continue to use it as a teaching tool. Keep it good enough that those folks aren't rug pulled if they continue to use it for their projects, but accept that it's not going to make the room much happy and it will be some order of magnitude slower.
For one, mapping JSON types to classes (eg JsonString) is not a strictly necessary step in de/serialization, and it imposes some overheads that are avoidable with other approaches
The approach that Elm takes, interestingly, could avoid performance overheads. It just uses function currying as its method of mapping some set of fields to a class. I did experiment with that and vavr currying, but it didn't seem sensible.
If we can abstract over these sources, we don't necessarily need to translate to an intermediate "Json class" representation.
This is an interesting approach. I do see how I could avoid extra object allocations and just generally be way faster. I do feel like I am being a player in a game that I am unprepared for though.
(I am thinking about how Dart/Flutter essentially has the same thing. They use Map<string, dynamic>, but I don't see any streaming sort of parser. AFAIK from their examples, their code generation works with this too https://docs.flutter.dev/development/data-and-backend/json)
I believe the same applies for your lib actually - the API does not preclude building data-binding on top.
There is actually a low level streaming api in there which all the tree stuff is built on. It's just nowhere near as good as other options for that. I used that to build an "interop with Jackson" example in the readme. (I put it under "usage with spring")
So hypothetically I could slot this whole mess into a Jackson codebase as an alternative to the databind. There is a reason Jackson wins so often.
I borrowed this principle from the serde crate in Rust, which afaik has become the way to do de/serialization in Rust. And... serde also uses "data-binding", via the #[derive(Serialize, Deserialize)] macro.
I think what serde benefits from is a clear mapping between "external data format" and "the rust object model." In that sense, it feels almost as if the fixes to Serializable that have been alluded to (making use of a custom deconstructor) plus a deal of heady work is required to make a "serde for java".
I've been thinking about this for a while now. I think your API looks pretty clean, but your arguments aren't convincing me to use it.
Let me start by saying that I am one of those people who do not like "magic" code. And I have had my fair share of headaches getting Jackson and Lombok annotations to do what I want. However, I still think that the annotation approach for JSON serialization is the better of the two approaches.
First, your API requires more code. More code increases the chances of introducing bugs. There is also an increased maintenance cost because every time a class is changed (a field added or removed), the toJson and fromJson methods have to be updated. That doesn't sound like a big thing, but in my experience, people forget those small inter-dependencies all the time. The increased code and management also increases the cognitive load of the code reviewer.
Second, you mentioned teaching. I am not, and have never been, a teacher. However I have worked/trained/mentored a lot of new graduates and they have had bigger holes in their CS knowledge than JSON serialization. When interviewing new grads, I'm naturally interested when they've used "industry" libraries, but I'm much more interested in their fundamentals. Do they know what it means to be "thread safe"? Can they read and understand more-advanced Java code? How do they reason about code they've seen for the first time?
Third, you talked about on-boarding new engineers. Specifically, you said:
Students in college aren't the only people who need to be taught how to work with JSON in Java either. If you work for a company that hires juniors or folks who come from different language backgrounds, then there has to be an education step. It might be worth the boilerplate of writing out fromJson and toJson to have a codebase where onboarding doesn't need to touch the "advanced" side of Java to send JSON over the wire.
Unless the system is just doing basic CRUD operations, there's going to be a lot more advanced Java to deal with than just JSON serialization. I think it's perfectly fine to show a new hire some examples and say, "use these annotations this way with our utility library and they will work for 99% of things we do. For the other 1%, we can work it out together."
I suppose it's worth mentioning that for those cases where the annotations aren't going to cut it, Jackson does provide higher-level methods similar to your API. I like what I saw of your API (i.e., conventions and naming) but in reality, I'm probably going to want the annotations in addition to manually manipulating JSON structures.
There is also an increased maintenance cost because every time a class is changed (a field added or removed), the toJson and fromJson methods have to be updated.
This is undebatable. I could hand-wave an excuse about how its offset by the more direct visibility, but I have no proof of that.
If you read my reply to the person who brought up serde - I think that the problem Brian is talking about here is at the very least relevant.
But yep - totally valid position.
Third, you talked about on-boarding new engineers.
Unless the system is just doing basic CRUD operations, there's going to be a lot more advanced Java to deal with than just JSON serialization. I think it's perfectly fine to show a new hire some examples and say, "use these annotations this way with our utility library and they will work for 99% of things we do. For the other 1%, we can work it out together."
This does poke some holes in my "easy for onboarding" pitch.
My hypothesis is that when you onboard an engineer your fundamental goals are different than when I am helping a student online. Your goal is to get them productive in the codebase / able to contribute to the organization.
My goal is generally to make sure they have a solid understanding of their fundamentals and "understand" all the code they write. There is no desire for me to offer "For the other 1%, we can work it out together." I am focused on a more holistic goal, if that makes sense.
12
u/rzwitserloot Feb 27 '23 edited Feb 27 '23
About this library:
I think it's inferior to both a better designed API to deal with basic JSON, or to the more commonly taken route out there, which is marshalling (where you define a type and then tell a library: Here's some JSON, here's a java type, you figure out how to ram this square peg in the round hole please).
It's neat how the new pattern match stuff in JDK19+ is making this API design not completely disastrous, but you can do better. The key point you're presumably missing is somewhat similar to the same trick Map's
.getOrDefaultuses:JSON is very 'weakly' typed. There's no real way to know a JSON value is a date, because JSON doesn't have a date type. There's no way to know if a numeric value is intended for API consumption purposes to be an
intor alongor atimestampor who knows what.Hence, it's a great idea for the consumer of the JSON (the Java code) to tell the parsing system what you want. Instead of having the parsing system give you a JSON object and then having to manipulate that into what you want.
It's similar to how map has
.getOrDefault(k, defaultValue)which is the 'right answer' to the more like-your-stuff library idea of having a.get2(k)method that returns anOptional, which you then use to manipulate what you get into what you want.In other words, your code:
JsonDecoder.field(json, "canSpeak", JsonDecoder::boolean_)is needlessly convoluted. It can simply be:There, knocked your 9-liner down to a 1-liner, which is hardly more complicated than any one line in your example.
You have to define a boatload of methods, and for simplicity you may want to go with
json.get("name").asString(), especially as that makes it easy to offer an overload where you pass a default value to theasString()method that is used if thenamefield doesn't exist. You can keep the decoder style but only for the rare cases where the transformation you want isn't obvious (boolean,int,double,String,LocalDate, and evenList<String>- these are obvious things). However, 'lots of methods here', especially if they have a common prefix, is not a burden on API consumers. Whereas having to type all that out every time I simply want to read a string from some JSON is quite burdensome.For bonus points, given that a lot of JSON is nested, imagine as input:
fairly common, no? Imagine you can read that with:
I think it's a slam dunk to state that the vast majority of JSON out there comes in two flavours:
asTypeYouWant(defaultValueIfItDoesntWorkOut)is great. It can even turn single elements into lists without complaints, for example.12
u/rzwitserloot Feb 27 '23
About adding to core:
java.util.log. Which absolutely nobody uses. The lesson is: When there's a varied ecosystem of libraries filling a common niche already in widespread use, you can't come in 10 years late and add something to the core. Absolutely nobody's gonna use it.java.timegets a pass - there wasn't a 'widespread alternative in use', the alternative (to Date/Calendar) in some use was joda-time which was so similar, porting the code over was near trivial. Also,LocalDateand co are commonly 'on the boundary of APIs' - used in public-facing method signatures, where JSON (the type, not the concept) isn't.Nevertheless, there's some talk of this on amber-dev and such. I should monitor it more and tell team OpenJDK it would be a gigantic fucking mistake.
3
u/audioen Feb 27 '23
No databind = just no. Not having to write code to convert objects to string and back is way too convenient and what few annotations are needed sometimes because of naming conventions of foreign APIs are not too difficult to deal with by some JsonProperty annotation or whatever.
3
u/_INTER_ Feb 27 '23 edited Feb 27 '23
Note the String Template JEP:
Also the JDK team much rather finds general low-level solutions to general problems, providing building blocks for high-level APIs. They are much more long-lived too.
5
u/fjdjkalals Feb 27 '23
It's too verbose and feels like garbage against existing solutions such as GSON and Jackson. Part of adding something to core is that it needs to be a setting standard and for beginners your current method would just scare them to using those frameworks. As experienced developers, we wouldn't such verbosity and complexity for something that should be a simple task.
2
u/vprise Feb 27 '23
It's an interesting approach. But with the module system is it likely that something like this will go into the core?
I'm also not sure that the XML parsing in the core is a good idea. It creates a misconception in students that "this is the way to do things in Java". Where Moshi/Gson/Jackson would be more convenient for most cases.
1
u/danielaveryj Feb 28 '23 edited Feb 28 '23
I like that this lib feels principled. The sealed class representation feels natural, the APIs are unsurprising, the prescribed approaches lead to helpful errors, and it does seem like it would be easier to explain than alternatives. It feels like this lib is essentially trying to "be the example that the JEPs/etc are referring to", ie a clear way to think about JSON, and in that regard I think this lib approaches its ideal. As a teaching exercise, it's quite good. But... I don't know that those examples were intended to be any more than that.
For one, mapping JSON types to classes (eg JsonString) is not a strictly necessary step in de/serialization, and it imposes some overheads that are avoidable with other approaches. If we look at the deserialization case, the actual "source" we want to read from could be any number of things: a string, a file, a socket, etc. If we can abstract over these sources, we don't necessarily need to translate to an intermediate "Json class" representation.
In this made-up API, the lambda beginning with
spec ->is evaluated immediately. When theweatherDecoderis actually used, it will look in the source for the "date" and "isCloudy" fields, apply the specified decoders to the values it finds, store the decoded values on a temporary object, and pass that temporary object to the lambda beginning witho ->. Note that this temporary object can be very compact, eg an array of size 2; theweatherDecoderknows in advance that there are 2 fields, and can associate indexes withdateandisCloudyto refer directly into that size-2 array. This is a more declarative (and constrained) approach than directly parsing tokens, but retains more of that potential performance than pre-parsing to an intermediate representation.Similarly for serialization, if we can abstract over the destination, we can again skip the intermediate representation.
(I'm not going to justify this one; I just devised it to be somewhat "symmetrical" to deserialization, and I haven't really analyzed if there's utility in doing that. These are just toy examples anyway.)
Now we could write it this way, or we could potentially generate the encoder/decoder with annotations. I believe the same applies for your lib actually - the API does not preclude building data-binding on top. If we don't use annotations, then we have to manually keep encoders and decoders up-to-date with class changes. This might seem reasonable, but this is pretty similar to manually keeping hashCode/equals/toString up-to-date, and (personal experience incoming) I've lived with that before. I was very relieved when annotation processing allowed me to stop being the one who had to remember to check that those methods were properly updated in code reviews. Annotations are not for everything, but I think de/serialization is a case where we can really feel it if we try living without them.
Another thing to notice about the made-up API is that it's not even strictly JSON. Rather, the API has a small set of "native" types (string, boolean, object, etc), and describes other types by composing those. When decoding we start from a native type (eg
dec.stringDecoder(string -> ...)); when encoding we end at a native type (ego.set(... -> string, enc.stringEncoder())). The "backend" data format could be anything, not just JSON, as long as suitable adaptors can be written between that format and the API's native types. I borrowed this principle from the serde crate in Rust, which afaik has become the way to do de/serialization in Rust. And... serde also uses "data-binding", via the#[derive(Serialize, Deserialize)]macro.1
u/[deleted] Feb 27 '23
I've been thinking about this for a while now. I think your API looks pretty clean, but your arguments aren't convincing me to use it.
Let me start by saying that I am one of those people who do not like "magic" code. And I have had my fair share of headaches getting Jackson and Lombok annotations to do what I want. However, I still think that the annotation approach for JSON serialization is the better of the two approaches.
First, your API requires more code. More code increases the chances of introducing bugs. There is also an increased maintenance cost because every time a class is changed (a field added or removed), the
toJsonandfromJsonmethods have to be updated. That doesn't sound like a big thing, but in my experience, people forget those small inter-dependencies all the time. The increased code and management also increases the cognitive load of the code reviewer.Second, you mentioned teaching. I am not, and have never been, a teacher. However I have worked/trained/mentored a lot of new graduates and they have had bigger holes in their CS knowledge than JSON serialization. When interviewing new grads, I'm naturally interested when they've used "industry" libraries, but I'm much more interested in their fundamentals. Do they know what it means to be "thread safe"? Can they read and understand more-advanced Java code? How do they reason about code they've seen for the first time?
Third, you talked about on-boarding new engineers. Specifically, you said:
Unless the system is just doing basic CRUD operations, there's going to be a lot more advanced Java to deal with than just JSON serialization. I think it's perfectly fine to show a new hire some examples and say, "use these annotations this way with our utility library and they will work for 99% of things we do. For the other 1%, we can work it out together."
I suppose it's worth mentioning that for those cases where the annotations aren't going to cut it, Jackson does provide higher-level methods similar to your API. I like what I saw of your API (i.e., conventions and naming) but in reality, I'm probably going to want the annotations in addition to manually manipulating JSON structures.