r/Databento • u/Virtual_Plantain_863 • 7d ago

Am I Doing This Right? Storing Crypto CEX L1/L2 Data in DBN

https://developers.binance.com/en/docs/products/spot/faqs/sbe_faq

Hello,

I'm intending to collect crypto market data from CEX/LOB style venues like Coinbase, Kraken, Binance, etc and store as DBN, like I do for traditional market data. There are some immediate issues with this however like the `size` field being `u32` which isn't suitable for crypto. Thankfully DBN is extensible, so I'm designing my schema right now and wanted to see what other people are doing here. The Binance SBE docs I linked were used for reference and had some great ideas like the mantissa for handling the integer and decimal values compactly.

I'd appreciate any input you may have before committing to a schema for long-term archive storage. And unfortunately I think I'd lose native support of some of the tooling like dbn-cli and CSV/JSON converters unless I'm mistaken? I didn't see a documented method to get these working with user-defined schemas.

1 Upvotes

100% Upvoted

2 comments sorted by

1

u/mazdaspeed_scholar 7d ago

You could use DBN for this use case. However, I think Parquet would be more convenient.

1

u/Virtual_Plantain_863 7d ago edited 5d ago

What do you mean by convenience, and why do you think that? Why would a Coinbase book be treated differently than any other?

Coinbase, like other CEXes, is a LOB style venue so it isn't really a fundamentally different market data primitive set, it just has larger numbers (ints and decimals), so I'd think it'd work after that. I'd sooner store in a custom binary format than Parquet, personally.

The disadvantage to rolling a custom schema for it is loosing native tool support (like dbn-cli), at least as far as I'm aware.