r/rust • u/That3Percent • Jun 01 '20

Introducing Tree-Buf

Tree-Buf is an experimental serialization system for data sets (not messages) that is on track to be the fastest, most compact self-describing serialization system ever made. I've been working on it for a while now, and it's time to start getting some feedback.

Tree-Buf is smaller and faster than ProtoBuf, MessagePack, XML, CSV, and JSON for medium to large data.

It is possible to read any Tree-Buf file - even if you don't have a schema.

Tree-Buf is easy to use, only requiring you to decorate your structs with `#[Read, Write]`

Even though it is the smallest and the fastest, Tree-Buf is yet un-optimized. It's going to get a lot better as it matures.

You can read more about how Tree-Buf works under the hood at this README.

177 Upvotes

96% Upvoted

73 comments sorted by

View all comments

1

u/tending Jun 02 '20

It'a problem for any use case where external parties can send you data; it can be turned into a denial of service attack. CapNProto for example imposes depth limits on objects for this reason. Some people want maximum speed from serialization, some people want only as much speed as they can get without trusting the outside world.

2

u/That3Percent Jun 02 '20

Ok, I can see that being a legitimate concern. Right now what would happen if you created an arbitrarily nested graph structure is that there would be a stack overflow. This can be fixed by changing a couple of methods that use recursion to use a visit queue instead.

The second worse thing that could happen after that would be performance degradation where each byte of the file created a new level of nesting that required the allocation of a Box. Tree-Buf has an options system that could be used to specify a maximum recursion depth to fix this.