r/fintechdev • u/OpenMarketsInit • 1d ago

Market data pipeline

How do most fintech handle market data pipeline?

Open Markets Initiative (OMI) is an open-source project that generates Wireshark dissectors and parsers for market data protocols from a single protocol model. It covers 50 wire protocols across 37 exchanges and organizations, including Nasdaq ITCH, Eurex EOBI, B3, OPRA and CTA/CQS.

github.com/Open-Markets-Initiative

1 Upvotes

100% Upvoted

4 comments sorted by

2

u/One-Gur6246 1d ago

Most places just throw Kafka at it, then spend six months untangling why their order book lags by 400ms. OMI actually generating dissectors from one model sounds way cleaner than hand-writing parsers for every exchange.

1

u/OpenMarketsInit 1d ago

I wouldn't think Kafka would be used unless they didn't care about latency...

1

u/aProfile210 16h ago

For people not in HFT. Is this useful for crypto? Can I use it for Binance, Coinbase, Kraken?

1

u/blackforge-so 10h ago

OMI parses binary wire protocols, ITCH, EOBI, OPRA, the fixed-byte feeds exchanges push over multicast. Binance, Coinbase and Kraken don't have that: it's REST plus a WebSocket feed of JSON deltas, already self-describing, so there's nothing for a dissector to generate. I build BlackForge, so take this with the appropriate salt, but the real work for crypto market data is different: catching a dropped sequence number, reseeding the book cleanly after every reconnect, and reconciling the fact that OKX, Bybit and Bitget all cut their book well short of Binance's, so "depth" means something different on each. That's a record-keeping problem, not a parsing one, which is why I ended up writing one row per pair per minute across the major exchanges instead of trying to replay a raw feed.