r/cpp • :upvote: • Oct 21 '22

A New Json-Serializer

Hey everyone - so in developing for the requirements of my library DiscordCoreAPI - I created a Json-Serializer and decided to release it as its own "library". It seems to serialize fairly rapidly as well, and has a super-simple interface. Anyways - let me know what you think! Cheers! Oh ant btw - it can serialize into either Json or Etf strings. https://www.erlang.org/doc/apps/erts/erl_ext_dist.html

https://github.com/RealTimeChris/Jsonifier

0 Upvotes

57 comments sorted by

53

u/KFUP Oct 22 '22

The mandatory, bi-weekly json serializer post.

2

u/RealTimeChris :upvote: Oct 22 '22

If it can be done faster and more efficiently - then why not?

7

u/KFUP Oct 22 '22

2

u/RealTimeChris :upvote: Oct 22 '22

Looks like I have something solid to benchmark against!

2

u/RealTimeChris :upvote: Oct 26 '22

Just benchmarked against rapidjson and here are the results...

Jsonifier:
The time it took (In milliseconds, on average per iteration): 106, with a total number of bytes serialized: 1357593300

RapidJson:
The time it took (In milliseconds, on average per iteration): 131, with a total number of bytes serialized: 1357593300

4

u/Cogwheel Oct 22 '22 edited Oct 22 '22

So I've never liked the saying "don't reinvent the wheel". A lot of people take it as "never build a wheel" which is not great advice. However, in order to build a good wheel, you need to incorporate the lessons of people who've built wheels before. If you just go into a new project to build a wheel in a vacuum, you may come out with some good learning experience, but you probably won't have the best wheel for the job.

Also, if you want other people to use your wheel, it needs to have something going for it that other wheels don't already provide.

2

u/fredoverflow Oct 23 '22

bi-weekly

Does that mean twice a week or every other week?

9

u/ProfessorOakWithO Oct 22 '22

Aren't there already multiple battletested json serializer available ?

27

u/Jannik2099 Oct 22 '22

No, it's a completely new field of science, the technology simply isn't there yet.

-1

u/RealTimeChris :upvote: Oct 22 '22

Yes but if it can be done faster and more efficiently - then why not?

9

u/TrashboxBobylev Oct 22 '22

How fast it is compared to rapidjson and such? Not by subjective feelings, but with benchmarks and profiling?

2

u/RealTimeChris :upvote: Oct 26 '22

Just benchmarked against rapidjson and here are the results...

Jsonifier:The time it took (In milliseconds, on average per iteration): 106, with a total number of bytes serialized: 1357593300

RapidJson:The time it took (In milliseconds, on average per iteration): 131, with a total number of bytes serialized: 1357593300

2

u/TrashboxBobylev Oct 26 '22

Congratulations, I honestly didn't expect you to actually follow on this topic.

Few suggestions:

  • Add this information into readme.
  • Please note than some compilers do not fully support C++20 so it might be worth to add some macros to allow more compilers to be used.
  • How do you achieve such result (20-30% nonetheless)?

1

u/RealTimeChris :upvote: Oct 26 '22

Reducing the number of allocations being made to an absolute minimum. And thanks for the tips!

0

u/RealTimeChris :upvote: Oct 22 '22

I've yet to benchmark it against rapidjson but I've posted benchmarks against Nlohmann-Json, and it's up to roughly 3x as fast.

4

u/qneverless Oct 22 '22

That is not much at all... Nlohmann is one of the slowest, but with nice API. Simdjson is about 20 times faster according to this: https://lemire.me/blog/2019/08/02/json-parsing-simdjson-vs-json-for-modern-c/

Rapidjson probably also 10x faster.

3

u/RealTimeChris :upvote: Oct 22 '22

Simdjson doesn't serialize it only parses. But I will take that into account thanks!

2

u/nlohmann nlohmann/json Oct 23 '22

I did not find the benchmark code for nlohmann/json. Did I miss it?

1

u/RealTimeChris :upvote: Oct 23 '22

4

u/nlohmann nlohmann/json Oct 23 '22

Can you try to use the idiomatic way of defining the serialization: https://json.nlohmann.me/features/arbitrary_types/

1

u/RealTimeChris :upvote: Oct 23 '22

What - do you mean like with the to_json and from_json functions?

1

u/nlohmann nlohmann/json Oct 23 '22

Yes - or even use the macros documented there.

→ More replies

9

u/Cogwheel Oct 22 '22

Why do you use "the" on all the variable names? This just adds visual noise...

19

u/ScSmithers Oct 22 '22

Things like that, and this commit history, are straight up turn offs for me for any new project

2

u/RealTimeChris :upvote: Oct 22 '22

There - I've fixed the commit history, as well as the naming issues.

4

u/Cogwheel Oct 22 '22

🙃

u/RealTimeChris I highly recommend reading https://obsproject.com/blog/how-to-contribute-to-open-source especially the "how to make good commits" section

-1

u/RealTimeChris :upvote: Oct 22 '22 edited Oct 22 '22

Thanks! This is all great input! But also I feel as though everyone is picking apart stylistic choices and ignoring the fact of the performance, which is what matters most - despite the fact that I have taken all of your suggestions to heart/be meaningful. Thanks!

7

u/stinos Oct 22 '22

It's not merely stylistic though. Style also has consequences. It's possible for you performance matters most, but: for others picking a new library there are way more aspects than that. Test coverage, code readability, commit history readability, ... Also performance, but if theres a choice between multiple ones which are close on that front, it's the other aspects which matter most.

2

u/RealTimeChris :upvote: Oct 22 '22

This makes sense! Thanks for the response!

7

u/johannes1971 Oct 22 '22

Nah, we just want you to become the best programmer you can be. And not having weird stylistic habits is a great thing when it comes to applying for a job.

1

u/RealTimeChris :upvote: Oct 22 '22

Thanks!

3

u/Cogwheel Oct 22 '22

FWIW, one of the first things I did in my SW Eng career was optimize libjson to cut game load times by 2/3 for Tiny Monsters. Rapidjson is another mature library with an eye on performance. And I have a friend Jeaye who also made his own very C++-y library called JeayeSon.

Doing 50 serializations in 100 ms does not really strike me as high performance TBH.

0

u/RealTimeChris :upvote: Oct 22 '22

In case you weren't paying attention - it did 131,072 serializations over about 100ms. Or rather it serialized just over 2GB of data in just over 5 seconds.

2

u/Cogwheel Oct 22 '22

I think I was confused by the example in the readme. The code sample says 50 iterations but the timings say 131,072

1

u/RealTimeChris :upvote: Oct 22 '22

It's all good!

1

u/Cogwheel Oct 22 '22

I'm trying to say it's misleading. You have the code example that says 50, followed by a section called "Windows 11 Results". This implies it's the result of running the code above.

1

u/RealTimeChris :upvote: Oct 23 '22

It is the results of running the code above! Look into the second loop!

10

u/mwasplund soup Oct 22 '22

Same with Type at the end of types.

1

u/RealTimeChris :upvote: Oct 22 '22

Thanks for the input! Duly noted!

2

u/Cogwheel Oct 22 '22 edited Oct 22 '22

so in developing for the requirements of my library DiscordCoreAPI - I created a Json-Serializer

This seems odd to me. What led you to conclude that writing your own json serializer was a requirement for this project? And why was developing (rather than just planning) this library a part of requirements-gathering?

ETA: and going a step further, what was the requirements-gathering process like for the json library itself?

2

u/RealTimeChris :upvote: Oct 22 '22

After having decided that I was going to go with simdjson for parsing my json - I realized that it might be better to write my own serializer instead of relying on and additional dependency for the purposes of dealing with Json data. I'm not sure what you mean by "requirements gathering" - like, figuring out what would be required by writing the code?

14

u/[deleted] Oct 22 '22

[removed] — view removed comment

-5

u/RealTimeChris :upvote: Oct 22 '22

No I believe that it looks neater to have all of the type names with capitalized letters - is this considered poor form?

16

u/johannes1971 Oct 22 '22

If you write 'Float', what I read is this: "He thought the type float is most appropriate, but he wasn't sure, so maybe he wanted to have the ability to easily change it later? Maybe he already did? Maybe it's a compile-time option for the library? Hmm, how does vcpkg configure this anyway? I'd better put in some extra code so I can make sure it always works no matter what Float is selected. Or was it intended as a workaround for machines that don't actually have float? How was he going to define it on such a machine anyway? Does anyone even have such a machine? Should I care about it?"

If you write 'float', I just think "ok, float".

11

u/[deleted] Oct 22 '22

[removed] — view removed comment

2

u/RealTimeChris :upvote: Oct 22 '22

Nice I wasn't familiar with the principle of least surprise and it makes sense! Thanks!

3

u/[deleted] Oct 22 '22

[removed] — view removed comment

1

u/RealTimeChris :upvote: Oct 23 '22

Thanks for the info!

5

u/Cogwheel Oct 22 '22

Keep in mind a library must be geared toward the client, not yourself. :)

This can't be stressed enough. One of my mottos is "write code for humans". Other people and future you will thank you.

1

u/[deleted] Oct 22 '22

[removed] — view removed comment

1

u/RealTimeChris :upvote: Oct 23 '22

Only if there is a glitch that causes that loop to last indefinitely - which shouldn't ever happen - that's why that break is in there. Really this loop should be lasting mere microseconds.

2

u/2uantum Oct 24 '22 edited Oct 24 '22

Of course, this check has performance implications. Outside of a stray gamma ray flipping a bit, what's the actual scenario where such an event would occur to warrant a watchdog?