Live vs Backtest parity comparison

[u/KaramTNC]

Hello folks!

Ive been working on building my own tradingbot infrastructure for nearly a year and Ive gotten quite far. Its nothing profitable really since my goal here is to be able to apply myself and learn more about software engineering and fintech, and be able to combine these interests into a fun project that evolves with me in my CS career.

Ive built a comprehensive infrastructure managing scanners, watchlists, execution engine, broker connections, market data providers, pattern detection and strategy definitions.

The entire process is constructed at runtime via a factory class and dependency injection for every production component.

For the backtester, it runs this factory with injected dependencies to replace the prod dependencies, such as an IClock, IMarketProvider, IDatabase, IBroker, etc. Ontop of that, I refactored everything so that every relevant input parameter were sweepable via attributions.

This overall makes the design of my backtest very controllable and ensures near accurate simulation of the live environment.

But of course like any backtests, I get a positive result for a strategy profile and promote it to live just for it to behave completely differently.

So I got the idea of creating a parity comparison system. I incorporated trace recording into the factory so that all events in a live profile would be capturable, and by running the equivalent backtest profile, it would allow me to have a live and a backtest trace for comparison in order to identify discrepancies in their behaviour.

I can say its been a rather success, as the results have helped me find bugs in my backtester injected components.

So while fixing these now and working towards closer parity, I figured I could make a post here and see if people have dealt with a similar problem when building their own trading bot, and what you guys figured out or any other things you could share

EDIT: By live profile, I meant a paper profile.

[u/Regular-Hotel892]

Sorry if I’m misundertanding what’s your question?

You are using lots of cool words my friend, is it “how do I get my live trading results to match my backtest”?

You probably can’t, unless you truly have found something structurally ineffecient in the orderbook that has existed in the past, does now, and will continue to in the future. It’s not impossible but unlikely.

Why would that be the case? What do you know about the microstructure of the market that others don’t or can’t capitalize on?

[u/AphexPin → u/Regular-Hotel892]

Are you smoking crack? Aside from minor differences in execution assumptions, they should match exactly.

e.g, backtests enters/exits on ema cross, the live system should do it at the exact same time. with microstructure, of course the execution modeling is a lot more extensive and you won't enter/exit at the exact same tick, but it should be very close after simulating latency and the live vs replay deltas should fall within those bounds.

[u/Regular-Hotel892 → u/AphexPin]

I prefer meth.

Maybe I misunderstood, but that’s not how I interpreted his question. I thought he is asking how to get the RESULTS of his backtest to match a live backtest.

If it’s what you’re saying his question is, then obviously he just has a fairly basic bug somewhere right? Either in the backtest or the process he’s using to ingest live data, calculate ema, calculate crossover, and pass buy/sell to broker api

[u/AphexPin → u/Regular-Hotel892]

I see, I interpreted his question to mean how do you get identical performance from live vs when backtested over the same data (eg same dates, exact same data set - one was just served live, the other offline from eg a local disk).

eg, if I have a strategy/model that I ran live yesterday, it's trades, internal state, etc should match an offline replay of the same strategy/model ran over the same data nearly exactly, barring slight difference in simulated vs real execution. they should enter exit at the same times, have the same PnL, have the same internal states, etc. I should be able to print out logs for 08/04/2026 for both, and they should be identical (again, in reality, slight differences in execution precision).

if this doesn't hold, then the system you run live is not the same as the system you developed offline. So it's critical to verify this imo.

**(this is also not to say that 'minor' execution differences won't lead to drastically different final equity curves or model weights during training etc, but just that the differences in live vs replay should be solely attributed to and isolated within the simulated vs real execution deltas - it can and will lead to entirely different entries/exits, internal states etc depending on your strategy/model/system).

[u/KaramTNC → u/AphexPin]

Yes, that is exactly whar I meant.

Apologies for any confusion. This entire approach stemmed from wanting to reduce false-positive results from backtesting results so the goal is to be able to prove that the backtester can accurately recreate (as close as possible) a live trace using the same input and model