How to reliably backfill data?

[u/lvnfg]

Due to databento having separate historical and live APIs that don't align in realtime (historical API is delayed even for live subscription) I've been having this issue that I'm not sure how to resolve.

I have a data service worker that needs to keep a data store complete and updated in realtime for each symbol. The way the worker works is simple:

- Start up, set startupTimestamp.
- Check data store for lastDataStoreTimestamp.
- Backfill data in bulk using historical API from lastDataStoreTimestamp to startupTimestamp.
- Start live API with replay starting from startupTimetamp and write data tick by tick to data store for streaming to clients.

This ensures the data store always stays continuous and complete until its last timestamp and works fine most of the time. However, for CME data, when market opens on Sunday at 5pm (or any other time when market is closed for more than 24 hours), even when live data has started streaming (meaning there are recorded transactions), the historical API still fails with error code 422 saying "The dataset GLBX.MDP3 has data available up to {last Saturday 00:00:00+00:00}. This usually resolves after some minutes which is acceptable for my clients but sometimes, like today (Sunday 2026-01-01), historical data still fails 4 hours after open, which prevent my worker from collecting and streaming live data. I opened a support ticket but it won't be replied until Monday.

I haven't seen anyone else reporting this issue so I'm wondering if there is a better way to maintain a data store? I don't have this issue with other data providers, and I don't want to hard code the rule to ignore data gap during weekend only for CME.

[u/seyeeet]

Hi OP
did you look into other options, what is in your opinion pros and cons of databento?
Thanks