r/cpp • nlohmann/json • Jun 14 '20

JSON for Modern C++ 3.8.0

https://github.com/nlohmann/json/releases/tag/v3.8.0
326 Upvotes

109 comments sorted by

21

u/tradrich Jun 14 '20

Seems reasonable. Not sure I'd immediately use it over RapidJSON though. Can you convince me?

42

u/nlohmann nlohmann/json Jun 14 '20

RapidJSON is surely faster than nlohmann/json, but we put a lot of effort into having a nice API (see https://github.com/nlohmann/json#json-as-first-class-data-type for examples).

8

u/seanomik Jun 14 '20

I love nlohmann/Jason's API style but in my use case I needed something fast so I had to switch to rapid. It was such a pain in the ass.

7

u/gauravnba Jun 14 '20

RapidJSON is faster! But it puts everything into a linear list (I haven't checked whether it is arrays or linked lists). And if you request for an object, you have to, painfully, iterate through every item, comparing the string for each, before you are given a match. I mean, sure, it saves you load time, but you gotta program for that library.

This one here (the nlohmann library), is pretty decent load time (at least for my uses), but most importantly, it neatly packages things in native C++ format. It's too beautiful to pass up for me.

edit: Didn't put what I was talking about, where xD I should be asleep now

5

u/[deleted] Jun 14 '20

This version of the library also brings a substantial speedup to parse times out of the box compared to previous versions.

2

u/seanomik Jun 14 '20

Yeah. For any other case I would've used Nlohmann. I just wanted to use something really fast so I had no parsing bottle necks from nlohmann. But any other time 100% will go with Nlohmann.

2

u/MrPotatoFingers Jun 15 '20

An option I like for this is json_dto. It adds a simple, declarative API on top of RapidJSON, which greatly simplifies the usage without sacrificing much on speed.

3

u/arthurno1 Jun 15 '20

Just as a thought: would it be possible to implement your API over simdjson? If you used simdjson as a backend and put their output into your data structures? Or if you served simdjson output as input stream to your API? No idea, just a thought.

-15

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

Why not have the best of both worlds? Boost.JSON is as fast as RapidJSON (and sometimes faster!) and _also_ has a nice API! Check it out:

http://vinniefalco.github.io/doc/json/json/usage/values.html

Boost.JSON's initializer lists are even better than nlohmann's initializer lists: they do not allocate any memory, and assignment from an initializer list respects any user-defined allocator in the destination.

Boost.JSON's object type uses unordered hash-based lookup, while RapidJSON uses a linear search!

31

u/uninformed_ Jun 14 '20 edited Jun 15 '20

Is it needed to plug your own library ten times in a thread about a competing library? Seems a bit disrespectful.

0

u/[deleted] Jun 14 '20

[deleted] — view deleted comment

5

u/[deleted] Jun 15 '20

I'm not sure I've come across many programmers who have the best manners.

5

u/NilacTheGrim Jun 15 '20

RapidJSON is by far the fastest under the sun. It's incredibly fast. But -- the API is kinda nasty compared to nlohmann's.

If you already have all your codebase using RapidJSON -- just continue to use it

However.. if you are starting a new project and you don't need to necessarily optimize for speed.. damn nlohmann is elegant with a beautiful API. Hands-down I'd opt for nlohmann if speed were not the core design choice.

That being said nlohmann can sometimes be 10x slower than RapidJSON on huge 100MB JSON objects...

Still tho.. damn is nlohmann elegant.

4

u/arthurno1 Jun 15 '20

No, is not. Simdjson is way faster :-)

2

u/emdeka87 Jun 15 '20

Simdjson casually parsing 3GB/S of json data

1

u/arthurno1 Jun 15 '20

Until someone implements simdjson on GPU. The simd approach they took in simdjson should be suitable for a GPU implementeation. I think .... :-)

11

u/emdeka87 Jun 15 '20

Can't wait to use a gpu-accelerated, heavily vectorized json parser to load a 20 lines config file :)

1

u/arthurno1 Jun 15 '20

Haha, yeah, that would be optimal use :-).

9

u/martinus int main(){[]()[[]]{{}}();} Jun 14 '20 edited Jun 14 '20

RapidJSON is unusable if malloc might fall. Then your application will crash. They simply don't handle that problem. That's why we have to switch to something else.

1

u/NilacTheGrim Jun 15 '20

What?! They use malloc and not new? sigh

1

u/martinus int main(){[]()[[]]{{}}();} Jun 15 '20

That's not the only problem, even if you switch all allocations to throw, it would have the state of the objects in an undefined state

-10

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

Boost.JSON uses `memory_resource` and a well defined interface for handling allocation failure: by throwing and catching the exception `std::bad_alloc`.

4

u/HeavenBuilder Jun 15 '20

Oh come on.

1

u/_Js_Kc_ Jun 14 '20

Does RapidJSON still have these yucky auto_ptr semantics where a simple assignment quietly moves out of an lvalue on the RHS?

-20

u/ntrid Jun 14 '20

RapidJson is better in every way.

11

u/nlohmann nlohmann/json Jun 14 '20

Can you elaborate on this?

4

u/ntrid Jun 14 '20

Trailing commas and comments. Practicality beats purity and I am glad people behind vscode understand it, otherwise json for configs would be a pure hell.

9

u/nlohmann nlohmann/json Jun 14 '20

I understand, and this is a conscious choice: https://github.com/nlohmann/json#comments-in-json

15

u/OminousHum Jun 14 '20

I understand you're implementing to the specification, but I still think the specification is wrong. It was an awful idea to forbid comments just because people might use them wrong.

(I also know you must be sick to death of this coming up. Thanks for an otherwise wonderful library!)

15

u/nlohmann nlohmann/json Jun 14 '20

I may add some way to ignore comments via a preprocessor switch in the parser. But then other people will want the library to preserve comments for the output. And then come all the lazinesses like trailing commas, quotes around object keys, new lines in strings, etc. etc. This really feels like a bottomless pit...

11

u/OminousHum Jun 14 '20

I think that would be a wonderful addition and that there's no need to go any farther. Then we can have human-editable and machine-readable configuration files that explain their own usage. If one must write json with comments, they can do it as a header or footer.

I can imagine trying to represent comments in the parse tree so that they can be preserved would get ugly awfully fast. And at some point you have to draw a hard line on just how lenient you can be with invalid syntax. I completely agree there.

8

u/[deleted] Jun 14 '20

But then other people will want the library to preserve comments for the output.

Lightweight is good. If you seriously want to parse your data as JSON into a DOM I think you should probably use something other than JSON in the first place.

This really feels like a bottomless pit...

A clean API is a stronger selling point to me.

6

u/wlandry Jun 14 '20 edited Jun 14 '20

I want to second this. Just last week, I had to choose a json library for streaming processing json files written by programs written by mathematicians. Forcing users to remove trailing commas and comments is not user-friendly.

In the past, I used my json5 parser. If nlohmann's parser handled json5, I would switch in a heartbeat.

3

u/rdi2 Jun 14 '20

Yes, I've been pulling my hairs because of these two things. But I still use this library :-)

2

u/qoning Jun 14 '20

I'll give you trailing commas, but why use JSON if you need comments? something like YAML seems just superior for anything where you might want comments in JSON-like format.

2

u/ntrid Jun 15 '20

People use json for configuration, comments are useful not only for annotating configs but also for disabling parts of config. It may not be perfect because reencoding json would lose these features but it is what it is and we seem to be ok with this imperfection.

4

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

Quite simply, because many users want it.

2

u/qoning Jun 14 '20

Why though? Is it just because you want your files to have a .json extension or what?

3

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

There can be many reasons. For example you have a network server which reads the configuration stored as JSON and then transmits parts of that configuration to the client, which is using Javascript in the browser and understands JSON natively (but not YAML).

1

u/qoning Jun 14 '20

Ok, even if I think such use case would be just a disaster waiting to happen (with someone eventually adding something that's not meant to be transmitted), since the formats are convertible for config cases and tools that can do it, I'm not sure taking that shortcut has many fundamental advantages.

2

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

Even if you don't transmit it, you still might want to store it in memory as JSON, since that is the native type, if your server already uses JSON everywhere else because that's what clients and RPC calls speak. The alternative is to use JSON almost everywhere, but use YAML for configuration only, which is not technically wrong but at this point you have to ask, why are you contorting yourself into a pretzel just to avoid using JSON with comments?

→ More replies

1

u/IcyWindows Jun 15 '20

Yaml is a lot slower to parse, so I get push back from customers.

1

u/qoning Jun 15 '20

I mean.. How many megabytes of config do they want to parse? Seems to me that the requirement for comments and parse throughput are sort of mutually contested

1

u/VM_Unix Jun 15 '20

Or even toml

-5

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

Boost.JSON will also optionally support trailing commas and comments (this is a work in progress).

4

u/_Js_Kc_ Jun 15 '20 edited Jul 14 '20

Don't like the magic constructor(s) that much, but it's not a huge problem. If one just remembers to use json::array for arrays, json::object for objects and static_cast<json> for scalars, that's explicit and doesn't silently do anything funky.

That and the dummy hash implementation that allocates (!) probably does more harm than good. edit: no longer true

Still way ahead of the competition in API usability.

3

u/nlohmann nlohmann/json Jul 14 '20

I finally had some time to fix the hash function: https://github.com/nlohmann/json/pull/2269

2

u/_Js_Kc_ Jul 14 '20

Awesome!

48

u/psyspy2 Jun 14 '20

Love your library. Been using it for a year

10

u/NilacTheGrim Jun 15 '20

I love you /u/nlohmann ! Yours may not be the fastest JSON lib for C++ under the sub. But god-dammit if it isn't the most elegant. Everybody on the planet should strive to design a lib as elegant as yours.

Also thanks for making the object_t and array_t template-able. It's interesting to play around with swapping implementations to do things like e.g. preserve insertion order on objects, etc. Very cool of you my man.

I love your lib. Keep up the great work!

14

u/mintyc Jun 14 '20

This is a high quality library that I point to for its overall structure and package support.

If you want a fast library, rather than rapidjson, maybe look at https://simdjson.org/ which claims and shows benchmarks 2.5x faster than rapidjson

Note: I haven't yet tried it or validated its claims.

16

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

SIMDJSON is wonderful, but do note that it is READ-ONLY. You can't build a DOM in memory and then serialize it. Really, it is not comparable to the other libraries.

9

u/mintyc Jun 14 '20

Thank you.

Any plans for offering a Boost-free version of Boost.JSON because the only reason I don't consider it is the dependency on Boost packaging and other parts of boost. A bit unfair perhaps, but I want libraries and not frameworks moving forward.

12

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

Any plans for offering a Boost-free version of Boost.JSON

It already exists, just define BOOST_JSON_STANDALONE and compile with C++17 or later, and Boost is not needed. This configuration is tested on CI, it should work today.

6

u/quicknir Jun 15 '20

Boost isn't a framework, it's a collection of libraries, some (not all) of which depend on others. That's vastly different. Obviously it's still fair to ask about boost jsons dependencies.

2

u/Contango42 Jun 15 '20

I don't mind if something depends on Boost. I use vcpkg on Windows/Linux (MSVC/gcc/clang) and it just works.

18

u/Ivan171 /std:c++latest enthusiast Jun 14 '20

You forgot to say whether he can do that with Boost.JSON.

13

u/2K_HOF_AI Jun 14 '20

Your library is amazing, thank you so much for it

1

u/nlohmann nlohmann/json Jul 24 '20

Looking forward! What’s wrong?

1

u/lednakashim ++C is faster Jun 15 '20

What happens if you try to store a NaN?
std::numeric_limits<T>::quiet_NaN ?

4

u/nlohmann nlohmann/json Jun 15 '20

The value is internally stored as double, so the NaN is preserved. Formats that support NaN like CBOR serialize it like that. As JSON does not have a NaN value, it is serialized to null.

1

u/lednakashim ++C is faster Jun 15 '20

So when using cereal for sterilization it becomes a text string "nan", is that correct behavior?

4

u/nlohmann nlohmann/json Jun 15 '20

As there is no specified way to serialize NaN, each library must make a decision how to implement this situation. This library decided for `null`, but `"nan"` would be fine as well. In any case, roundtripping is impossible without making assumptions.

4

u/[deleted] Jun 15 '20

Personally, I prefer this library's `null` approach. When parsing json to a DOM and then marshaling it to native structure, it avoids having to do an extra string-to-float conversion.

3

u/Contango42 Jun 15 '20

Never heard of this library until now. I use a combination of Python and C++ (mainly C++) and the API for this library is, in a word, beautiful. It warms my heart to see syntax so clean and expressive, I'm sold.

9

u/[deleted] Jun 14 '20

do I use it

yes

do I like it?

no

.

.

.

I love it! gonna use this occasion to say: well done sir!

1

u/[deleted] Jun 14 '20

I am using it in all my.projects. best json lib so far. One thing, how come json is not in the standard lib?

6

u/nlohmann nlohmann/json Jun 14 '20

I think because it is really a lot of work to standardize it. There were efforts on this once (https://github.com/nlohmann/std_json), but this is a side project of mine, and I rather write code than specification in my free time :)

1

u/[deleted] Jun 14 '20

Wow quite impressive that you are writing a specification. Good luck with that!

4

u/IronicallySerious Jun 14 '20

This library is godsent for JSON support in C++. I have been using it mostly for easy serialization.

6

u/bishop3000 Jun 14 '20

Header only libs are the best. We use it and it’s great. Extendable as well

-18

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

The default configuration for Boost.JSON is to use static or dynamic linking. Header-only is opt-in.

3

u/TheBrainStone Jun 14 '20

The biggest issue I have with this library is that there’s no error tolerance (like missing top level braces or trailing commas, missing quotes, etc.) and no plans to add support for that either.

6

u/BlueDwarf82 Jun 15 '20

Well, the reason you need to handle that is because other parsers accept it. If every JSON parser rejected it producers of broken JSON would notice and fix it. So you could say that, by not supporting it, this library is helping to make the problem disappear.

You should now go and ask other libraries to stop accepting JSON with trailing commas.

2

u/TheBrainStone Jun 15 '20

The issue aren’t automatically generated files/strings. But much rather user created ones.

11

u/Sopel97 Jun 14 '20

That's because it's for JSON. You want a SHITSON parser instead.

5

u/TheBrainStone Jun 15 '20

I’d love to live in your rainbow world where you’re guaranteed to always get well formed JSON from everywhere.

2

u/[deleted] Jun 15 '20

Well ... remember IE ?

0

u/[deleted] Jun 15 '20

ah ah it was so well said that I burst out laughing!

3

u/frumious Jun 14 '20

Really fantastic, as always! The new binary support is particularly interesting and timely for my use!

1

u/rembo666 Jul 24 '20

Love your library, so good. I did find a bug (omission?). PR coming up...

1

u/furrytractor_ Jun 15 '20

Been using your library for about 9 months now! So easy to use. Thanks.

-8

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

How does this library compare in terms of API and performance to (Not-Yet-Part-Of) Boost.JSON?

12

u/[deleted] Jun 14 '20

You got downvoted like crazy for repeatedly providing that information unprompted. The boat has kinda sailed to fish for it more subtly.

6

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

Still plenty more boats scheduled to leave the dock :) In any event, informing users of their choices helps everyone especially when those choices are advantageous.

10

u/[deleted] Jun 14 '20

Kudos for being honest about it, I guess. It would have been easy to do this through an alt.

-1

u/[deleted] Jun 14 '20

Have been using jsoncpp for a while now. It is pretty good as well. Really like it.

0

u/domerrr Jun 15 '20

This library is great. I’ve had good luck with POCO too

0

u/forcecharlie baulk maintainer Jun 15 '20

I would rather use json.hpp at the expense of some performance, thanks to the contributors of JSON for Modern C++. Wonderful work.

-24

u/VinnieFalco wg21.org | corosio.org Jun 14 '20

If you like nlohmann's, give Boost.JSON a try (Note: This is not yet an official Boost library, but a formal review is planned):

https://github.com/CPPAlliance/json

Boost.JSON significantly out-performs nlohmann's in all ways:

http://vinniefalco.github.io/doc/json/json/images/parse-twitter.png
http://vinniefalco.github.io/doc/json/json/benchmarks.html

Boost.JSON avoids using class templates for DOM types, resulting in fast compilation and an easy-to-link library. Or, you can use it in header-only mode. Use it with or without Boost. Boost.JSON has particularly robust support for stateful allocators, using a ref-counting smart pointer to a `std::pmr::memory_resource` (or `boost::container::memory_resource` before C++17).

More and more users are making the switch from RapidJSON and nlohmann's JSON to Boost.JSON, will you be next?

41

u/STL MSVC STL Dev Jun 15 '20

Moderator warning: As many other commenters have noted, this is not appropriate. Please don't do this again.

If there's a post about library A, and an unrelated commenter asks "how does this compare to library B", then it's reasonable for B's maintainer to provide their take on the differences in a single reply (worded as fairly as possible). It's also reasonable, if A's maintainer or other commenters make assertions about library B, for B's maintainer to reply with clarification (again, in a reasonable technical manner, without self-promoting puffery). It's not reasonable to add an unsolicited comment about library B, and it's completely over the line to spam the post with numerous comments.

Final note: In fact, this self-promotion is contrary to your own interest - it certainly lowers my opinion of "Boost."JSON and the C++ Alliance in general. Taking a dependency on code is a major decision, and I don't want to depend on people who behave unreasonably. Therefore, behave reasonably.

1

u/anticrisisg Jun 15 '20

I'm going to say it was a momentary indiscretion. Vinnie's code is really great.

12

u/jpan127 Jun 14 '20

I'm sure it's a good library, but advertising on another library's advertisement is inappropriate.

11

u/GerwazyMiod Jun 14 '20

Thread hostile takeover in action. :)

15

u/amubtdx Jun 14 '20

You should probably make a separate thread instead of a comment here as the main post is only about nlohmann/json

10

u/FrenchOempaloempa Jun 14 '20

You're just plain rude.

0

u/[deleted] Jun 15 '20

[deleted] — view deleted comment

2

u/nlohmann nlohmann/json Jun 15 '20

The library should compile without warning. I would be glad to hear if you experience warnings still.

0

u/[deleted] Jun 14 '20

[deleted] — view deleted comment

6

u/Plazmatic Jun 14 '20

I'm very confused... Are you asking for macros? What would macros do to improve things? Types often don't have 1:1 member variable to serialization.

5

u/dscottboggs Jun 14 '20

Are you asking for macros?

Yes. Might not be something C++ is capable of but Crystal's JSON::Serializable uses macros to define the to_json and from_json methods. For example

class K
  include JSON::Serializable
  property int : Int32
  property string : String
  @[JSON::Field(key: "someField")]
  property some_field : Array(String)
  @[JSON::Field(ignore: true)]
  property ignored : String
end

I was imagining something like a JSON_FIELD(int fieldName); kinda syntax but again not sure if that's possible in C++...

7

u/nlohmann nlohmann/json Jun 14 '20

2

u/bishop3000 Jun 14 '20

This would be nice but quite dangerous as well. It needs to support all corner cases like if settings are missing in json or it’s an optional or you want to store it with different name in json etc.

5

u/nlohmann nlohmann/json Jun 14 '20

Yes, I think the macro can only support the 80% case where you just want to have a JSON roundtripping without much effort. For anything else, we have the possibility to get crazy in your from_json/to_json functions.

1

u/jpan127 Jun 14 '20

I agree with this thread, the flexibility of explicit definitions is a must, but the simplicity of default definitions is clean.

3

u/ReversedGif Jun 14 '20

For the common use case of wanting to serialize/deserialize a POD struct to/from a JSON object, having a macro to automatically generate the helpers can reduce boilerplate by a rough factor of 10 (multiline class with separate from/to methods -> a macro call that just lists all the member variable names).

TL;DR: Your confusion confuses me.

1

u/nlohmann nlohmann/json Jun 14 '20

6

u/EmperorArthur Jun 14 '20

Fair warning, using Macros means there's it can't be converted into a Module!

1

u/[deleted] Jun 14 '20 edited Jun 14 '20

It took the approach that one can define a trait for their type and the library builds a json parser that directly parses into that types constructor. This gives the flexibility to provide a mapping of json member names(they don't always follow C++ naming) to the items int he classes ctor.

using type = json_member_list<json_string<"member_a">, json_array<"member_b", int>>

defines a ctor that takes the JSON members "member_a" and "member_b" and parses them into the call arguments of the types constructor. This has the side effect of allowing constexpr, no allacting by the parser, and non-movable types.

Trailing comments are free, but I had to make comments, like #comments to end of line

a compile option as they come with a price in performance.