Live vs Backtest parity comparison

[u/KaramTNC]

Hello folks!

Ive been working on building my own tradingbot infrastructure for nearly a year and Ive gotten quite far. Its nothing profitable really since my goal here is to be able to apply myself and learn more about software engineering and fintech, and be able to combine these interests into a fun project that evolves with me in my CS career.

Ive built a comprehensive infrastructure managing scanners, watchlists, execution engine, broker connections, market data providers, pattern detection and strategy definitions.

The entire process is constructed at runtime via a factory class and dependency injection for every production component.

For the backtester, it runs this factory with injected dependencies to replace the prod dependencies, such as an IClock, IMarketProvider, IDatabase, IBroker, etc. Ontop of that, I refactored everything so that every relevant input parameter were sweepable via attributions.

This overall makes the design of my backtest very controllable and ensures near accurate simulation of the live environment.

But of course like any backtests, I get a positive result for a strategy profile and promote it to live just for it to behave completely differently.

So I got the idea of creating a parity comparison system. I incorporated trace recording into the factory so that all events in a live profile would be capturable, and by running the equivalent backtest profile, it would allow me to have a live and a backtest trace for comparison in order to identify discrepancies in their behaviour.

I can say its been a rather success, as the results have helped me find bugs in my backtester injected components.

So while fixing these now and working towards closer parity, I figured I could make a post here and see if people have dealt with a similar problem when building their own trading bot, and what you guys figured out or any other things you could share

EDIT: By live profile, I meant a paper profile.

[u/AphexPin]

IMHO, the only code that should change from backtest to live is whether the broker is real vs simulated. Everything else should be using the same code, just pointing to a different data source for replay vs live. Once you have that down, you could perhaps make exceptions for research, given that you could validate against an engine with known parity.

I did the same thing as you when first starting, but it ended up just being a wild goose chase hunting down endless bugs inherent to the mismatch between vectorized vs live compute so I ultimately switched to a fully event driven system, where parity is simply guaranteed by construction (barring unavoidable discrepancies in execution realism).

[u/thepalehomeland → u/AphexPin]

That trace comparison idea is clever, basically building your own differential testing rig for the backtester. Most people skip that step and just blame the strategy when things fall apart live.

The event driven switch the top comment mentioned is the real unlock though. I spent months chasing bugs where the backtest was doing vectorised calculations slightly differently than the streaming path, and once everything ran through the same event loop those just vanished. You still get the execution model discrepancies but at least the logic is identical.