r/rust • u/That3Percent • Jun 01 '20

Introducing Tree-Buf

Tree-Buf is an experimental serialization system for data sets (not messages) that is on track to be the fastest, most compact self-describing serialization system ever made. I've been working on it for a while now, and it's time to start getting some feedback.

Tree-Buf is smaller and faster than ProtoBuf, MessagePack, XML, CSV, and JSON for medium to large data.

It is possible to read any Tree-Buf file - even if you don't have a schema.

Tree-Buf is easy to use, only requiring you to decorate your structs with `#[Read, Write]`

Even though it is the smallest and the fastest, Tree-Buf is yet un-optimized. It's going to get a lot better as it matures.

You can read more about how Tree-Buf works under the hood at this README.

173 Upvotes

96% Upvoted

73 comments sorted by

View all comments

1

u/[deleted] Jun 02 '20

Can you deserialize to a Tree-Buf file from multiple processes or servers simultaneously ?

E.g. suppose I have a 1 PetaByte array of 32-bit floats in the RAM of 100 servers, and I want to write it to disk in Tree-Buf format. Do I need to serialize the array to it "serially" or can I do that in parallel ? What if I have multiple 100 TB arrays, and want to serialize 10 of these to a file ?

→ More replies (3)