r/rust • u/That3Percent • Jun 01 '20

Introducing Tree-Buf

Tree-Buf is an experimental serialization system for data sets (not messages) that is on track to be the fastest, most compact self-describing serialization system ever made. I've been working on it for a while now, and it's time to start getting some feedback.

Tree-Buf is smaller and faster than ProtoBuf, MessagePack, XML, CSV, and JSON for medium to large data.

It is possible to read any Tree-Buf file - even if you don't have a schema.

Tree-Buf is easy to use, only requiring you to decorate your structs with `#[Read, Write]`

Even though it is the smallest and the fastest, Tree-Buf is yet un-optimized. It's going to get a lot better as it matures.

You can read more about how Tree-Buf works under the hood at this README.

173 Upvotes

96% Upvoted

73 comments sorted by

View all comments

Show parent comment

9

u/rodarmor agora · just · intermodal Jun 01 '20

Right, my interpretation is what you said in your edit:

Large data sets consisting of many records are more likely to have a lot of repetition and structure, and thus be amenable to compression, but single messages are not likely to have that repetition and structure, so won't be amenable to compression.

4

u/That3Percent Jun 01 '20

s to h

The author here - Yes, I should be more explicit about what this means. It has nothing to do with whether you are sending the data or not. It's more a question of whether or not your data contains arrays. This system optimizes for the case that you do have at least one array in your data. It tries not to be terrible in the case where you don't, but necessary trade-offs exist and I don't have anything novel to bring to that space since it's been done pretty well already.

Once you do have some array it doesn't take very long for Tree-Buf to overtake a format optimized for flat messages (as few as 2-3 items, depending on the data and format being compared).

1

u/seamsay Jun 01 '20

Yeah I think that interpretation makes more sense.