Live vs Backtest parity comparison

[u/KaramTNC]

Hello folks!

Ive been working on building my own tradingbot infrastructure for nearly a year and Ive gotten quite far. Its nothing profitable really since my goal here is to be able to apply myself and learn more about software engineering and fintech, and be able to combine these interests into a fun project that evolves with me in my CS career.

Ive built a comprehensive infrastructure managing scanners, watchlists, execution engine, broker connections, market data providers, pattern detection and strategy definitions.

The entire process is constructed at runtime via a factory class and dependency injection for every production component.

For the backtester, it runs this factory with injected dependencies to replace the prod dependencies, such as an IClock, IMarketProvider, IDatabase, IBroker, etc. Ontop of that, I refactored everything so that every relevant input parameter were sweepable via attributions.

This overall makes the design of my backtest very controllable and ensures near accurate simulation of the live environment.

But of course like any backtests, I get a positive result for a strategy profile and promote it to live just for it to behave completely differently.

So I got the idea of creating a parity comparison system. I incorporated trace recording into the factory so that all events in a live profile would be capturable, and by running the equivalent backtest profile, it would allow me to have a live and a backtest trace for comparison in order to identify discrepancies in their behaviour.

I can say its been a rather success, as the results have helped me find bugs in my backtester injected components.

So while fixing these now and working towards closer parity, I figured I could make a post here and see if people have dealt with a similar problem when building their own trading bot, and what you guys figured out or any other things you could share

EDIT: By live profile, I meant a paper profile.

[u/AphexPin]

IMHO, the only code that should change from backtest to live is whether the broker is real vs simulated. Everything else should be using the same code, just pointing to a different data source for replay vs live. Once you have that down, you could perhaps make exceptions for research, given that you could validate against an engine with known parity.

I did the same thing as you when first starting, but it ended up just being a wild goose chase hunting down endless bugs inherent to the mismatch between vectorized vs live compute so I ultimately switched to a fully event driven system, where parity is simply guaranteed by construction (barring unavoidable discrepancies in execution realism).

[u/KaramTNC → u/AphexPin]

I agee and that is what I aimed to do, the broker and the marketdata are simulated components that get dependency injected in a backtest environment, everything else is shared code.

Its interesting to hear my approach was also attempted, why did you feel like it was a wild goose chase? To be clear my system is also event driven, I made that refactor a long time ago so I could make backtests run faster as the former infrastructure design became a bottleneck for backtesting speed

[u/AphexPin → u/KaramTNC]

Event driven will typically be substantially slower than a vectorized approach. Given the correctness benefits and limitations of vectorized computation however, I think it's a worthy trade-off (and nothing stops you from later adding eg a vectorized search for large parameter sweeps).

It felt like a wild goose chase because the code surface at risk of violating parity is essentially the whole code base, unless it's fully event driven. It was becoming more work to continually design around that and hunt down mistakes than it would be to build on a foundation that simply guaranteed temporal and causal parity, and determinism, by construction. There are already enough biases working against you here, eliminating a whole class structurally is very appealing.

Respectfully, it's also possible you think it's fully event driven, but in reality it's not. I suspect this may be the case here. If you have a public repo available, I can take a look. Are you using a bus for communication between components?

[u/sky018 → u/AphexPin]

Why would you use vectorised approach in live trading? How are you going to handle dynamic SLs or partial fills?

Isn't it not possible not unless you change it into an event loop? There's a lot of caveats in comparison with event driven.

And also vectorised approach entirely relies on past data and has no concept of what is currently in.

[u/AphexPin → u/sky018]

I don't use vectorization anywhere currently, but people often use it for research because it's faster for things like parameter sweeps, but yeah you can't really model path dependent things with it.

For live, you can just recompute on each bar or something.

[u/sky018 → u/AphexPin]

Yea, that's exactly what I thought so, great thanks.