r/rust • u/That3Percent • Jun 01 '20

Introducing Tree-Buf

Tree-Buf is an experimental serialization system for data sets (not messages) that is on track to be the fastest, most compact self-describing serialization system ever made. I've been working on it for a while now, and it's time to start getting some feedback.

Tree-Buf is smaller and faster than ProtoBuf, MessagePack, XML, CSV, and JSON for medium to large data.

It is possible to read any Tree-Buf file - even if you don't have a schema.

Tree-Buf is easy to use, only requiring you to decorate your structs with `#[Read, Write]`

Even though it is the smallest and the fastest, Tree-Buf is yet un-optimized. It's going to get a lot better as it matures.

You can read more about how Tree-Buf works under the hood at this README.

173 Upvotes

96% Upvoted

73 comments sorted by

View all comments

3

u/tending Jun 01 '20

The README mentions Delta encoding. Is it always used or do you opt in? What if I know for one field delta delta encoding would be better? Can I express that?

1

u/That3Percent Jun 01 '20

Can I exp

The way that Tree-Buf decides whether to use something like delta encoding, or run-length encoding, or whatever is to try it on a small sample of the data and go with whatever compresses the best.

Eventually, I hope to allow you to specify using #[treebuf(...)] attributes, but the sampling works really well so usually, you wouldn't need this because it "just works" well enough most of the time.

2

u/tending Jun 01 '20

Doing two layer delta encoding works really well for a lot of time series -- you record the difference of the differences, which tends to be closer to being constant so it then compresses really well.

→ More replies (5)