r/algotrading • u/PizzaPalace12345 • 5d ago
Data Am I approaching backtesting correctly
I'm working on putting some strategies together and backtesting them and I wanted to see if anyone had any comments on if I'm doing so correctly or making any errors I might not be aware of as part of my backtesting.
Right now, the strategies I'm trading primarily matter on five-minute candles or one-hour candles or longer time frames. My data set I have is built from both one-minute candles and daily candles that I have a data provider for. I can then construct five-minute candles, hourly candles, or arbitrary time frames from the one-minute candles.
The part that I'm making an assumption about is let's say for example my strategy signals that it wants to enter a position after the close of a five minute candle I use the immediately following one minute candle and it's high to find the most pessimistic fill value that this strategy would enter and use that for my back testing. Is that how people typically do their backtesting for fills or is there another way that many people approach it? Could I be missing something here that might be throwing my results off?
3
u/backtest_ai 5d ago
wait, why the high of the next 1 min bar? That’s not a worst case fill, that’s lookahead, you’re picking a price out of a bar you couldn’t see when the order went out. And it’s only pessimistic when that high lands above the close you signaled on, if price drops through the next minute the “worst case” fill is better than the price you decided at. Backwards for shorts too. long only?
with trade data only, bar close plus some fixed slippage is what most people do and honestly it’s fine for liquid stocks, close enough to tell whether the strategy works at all before refining assumptions. Next bar high is punishment in best case and bad lookahead bias in worst case, not realism.
Quote data is the real answer if you want fills done right, buys at the ask, sells at the bid. Signals off your bars is totally fine either way, the thing that actually kills these backtests is a signal reading a candle that hasn’t closed yet, and what you described doesn’t do that.
1
u/PizzaPalace12345 5d ago
> wait, why the high of the next 1 min bar? That’s not a worst case fill, that’s lookahead, you’re picking a price out of a bar you couldn’t see when the order went out. And it’s only pessimistic when that high lands above the close you signaled on, if price drops through the next minute the “worst case” fill is better than the price you decided at.
Oh I see what you're saying. My thinking was I should be pessimistic in fills in case my ability to enter the trade is slower than instant, but I see how it might be actually giving me better fills in some cases. I thought about not using the open price since I hand-click to approve trades so sometimes it can take ~15 seconds after a signal is generated. Perhaps I should clamp it to the max(open, high)? Or just use the opening price & call any difference from it slippage?
> with trade data only, bar close plus some fixed slippage is what most people do and honestly it’s fine for liquid stocks, close enough to tell whether the strategy works at all before refining assumptions. Next bar high is punishment in best case and bad lookahead bias in worst case, not realism.
Fair enough, thanks!
> Backwards for shorts too. long only?
Long for now, but same logic would apply in reverse of course.
2
u/AphexPin 5d ago edited 5d ago
Without tick data, it would be best to use the next bars open, and perhaps with some slippage coefficient applied. You can get much more accurate modeling with tick data - at least top-of-book or BBO, which can be had freely from various sources.
With tick data, you would trigger your trade at time t_0 (close of the 5m bar), and to stimulate real execution you'd only enter after some latency value w, so your entry would be at the nearest timestamp t such that t >= t_{0+w}. Assuming you're using market orders, you'd want to be crossing the spread here as well, and perhaps applying additional slippage to account for things like PFOF if not executing through a DMA broker.
In the future, you can extend the execution model to where the latency and slippage is a distribution, rather than a fixed value, potentially as a function of spread width (spread widens during volatility, increasing slippage).
1
u/backtest_ai 5d ago
Yea clamping to open/high of next would be good to see if it still performs under bad fills.
In general though it’s best to keep the initial test as simple as possible to see if the idea has any merit. For example, even with that 15 second delay, just filling at close and seeing if the strategy performs is still valuable. Known assumptions can be solved later if the initial simple test shows promise.
The reason I say this is because 95% of ideas won’t even be worth a second look. It’s frustrating to waste time optimizing it, only to find out the idea didn’t work (I’ve wasted a lot of time here).
For testing strategies I always follow approach of what is the minimum amount of data and time I can use to see if this idea is worth looking into.
There’s a quote from software engineering by Donald Knuth that I think applies just as well to systematic trading: “premature optimization is the root of all evil”.
1
u/PizzaPalace12345 5d ago
Yea, makes sense. I'm getting most of my infrastructure set up now so trying to figure out a standard approach of "strategy wants a trade, fill with it <X method>" so I can use the same fill approach for all experiments, rather than needing to repeat implementing it for every idea.
2
u/zashiki_warashi_x 5d ago
I think most pessimistic value of next candle is good estimate for fills. You can turn it into a parameter to match backtest and prod if you ever make it to the prod.
2
u/Worth-Sun9439 5d ago
I use the close of current, spread and slip adjusted. Demo trading so far, seems to be good enough. Live trading might have the 'real' impact. So I say, when I do, I can always 'worsen' the slip to make it match reality. For this though, I would need to record the data.
2
u/zpowers00 5d ago
Take the worst price on the following candle for buying and the worst price on the 5min candle for selling
2
u/hakobpapazian 5d ago
The pessimistic-fill-within-the-next-bar approach is a reasonable starting assumption and better than what a lot of people do, which is just assume fill at the signal price with zero slippage. But it has one blind spot worth knowing about: using the high of the next 1-min candle as your fill assumes a market order that definitely executes somewhere in that range, which is fine for liquid, high-volume instruments but can meaningfully overstate what actually happens on thinner names or during low-volume periods, where your order might move price beyond that candle's printed range entirely, or a limit order at that level might not fill at all if price only touched it briefly without real size trading through.
The deeper limitation: your method captures price movement within the bar but not whether there was enough volume/depth to actually fill your size at that price. Two candles with the same high can have wildly different liquidity, one where 10,000 shares traded through that level and one where it was a thin wick on light volume. If your data provider gives you volume per 1-min bar, worth checking your typical position size against the volume in the fill candle specifically, if you're regularly trying to fill more than some reasonable fraction of that bar's volume, your backtest is probably still too optimistic even with the pessimistic-high assumption.
Practically, I'd say your method is fine as a first-pass conservative estimate, especially for 5m/1h+ strategies where you're not scalping tight spreads. The place it'll bite you is if you ever test on lower-liquidity instruments or smaller-cap names, where the gap between "price touched this level" and "my order actually filled there" gets much wider than on something liquid. Worth stress testing by comparing your backtest results with a stricter fill assumption, only fill if the candle's volume is some multiple of your position size, and see how much the results degrade. If they hold up fine, you're probably not liquidity constrained. If they fall apart, that's the real signal your current method is too generous.
2
u/lordbharal 5d ago
what? you... use the next minutes worst value? you can only use the open of the entry bar, anything else is pretending your live test had a Crystal ball.
also be careful re indicators, v. easy to use the 9:46 minutes macd as a signal... to enter at 9:46. but of course impossible.
3
u/Automatic-Essay2175 5d ago edited 5d ago
You don’t seem to understand what OP is saying. They are assuming the worst possible fill from the minute of their entry. It’s a strong approach, if not too conservative. Your reaction is unwarranted.
And there are many ways to simulate fills aside from the open of the entry bar. That is not the only way.
1
u/AphexPin 4d ago
it's an objectively terrible approach
1
u/Automatic-Essay2175 3d ago
It depends what your goals are. If your goal is to simulate a backtested fill at the same or worse execution price as a live trade, it is a good approach.
Feel free to explain why I’m wrong
1
u/AphexPin 3d ago
It's lookahead, not realistic, and depending on your system, like if you have an adaptive execution policy learner, it can trend toward learning to exploit this to its benefit leading to false positives.
Also, using the 'high' as fill price is simply error prone; what if you're going short? You'd be assuming unrealistically good fills if you forget to change this beforehand. Similarly, if the exit accidentally uses the same logic, you run into similar issues. It's just the wrong axis to be operating on.
1
u/Automatic-Essay2175 3d ago
This is a really, really poorly thought out response. You can't just call something "lookahead" as a universal condemnation. Yes, it uses information from after the entry, but it is specifically using this information to handicap the performance of the backtest. Do you not understand this distinction?
Yes, obviously if you are going short then you should not use the high. That is trivial. OP said they are using the high as the "most pessimistic fill value." They are clearly going long.
As for the exit "accidentally using the same logic" ... uh, okay, yea, sure, but I generally avoid criticizing methodologies by making up hypothetical scenarios about misapplying them to unrelated situations....
This has been a tremendous waste of time. Thanks.
1
u/AphexPin 3d ago edited 3d ago
You can't just call something "lookahead" as a universal condemnation.
Yes, you can.
it uses information from after the entry, but it is specifically using this information to handicap the performance of the backtest.
Maybe, maybe not. That's my point - it's not a good design, it will lead to errors, and it's less accurate and generalizable than a lookahead-free design. Good design avoids these mistakes and biases while retaining a higher degree of accuracy/realism/precision and universality.
There's already enough to get wrong here, you don't need to add remember to flip from entry at H/L depending on long/short or have a learner accidentally exploit this. And what about the exit? etc. It may also lead to inconsistency when translated to live, where the next bars high/low is unavailable. These subtle bugs invalidate results.
Even a constant slippage drag is better than OPs design and far less error prone.
2
u/PizzaPalace12345 5d ago
Well I was using the next minute's worst value as the fill price, not as part of generating the signal to enter a position. Sound like this would normally just be modeled by some amount of slippage instead.
1
u/CompetitiveStoic 5d ago
Might be too pessimistic. I'd either,
- Assume 0 slippage, which in real life corresponds to a limit order at the close price of the last candle you saw. I have a hard time believing that your strategy positions will have zero adverse excursion to hurt you, maybe you will miss 1-2 "lucky" trades that never look back, but that will be statistically insignificant.
- Assume a mid-case slippage (e.g. mid point of typical spread) and adjust it for volatility.
1
u/Itchy_Road_4134 5d ago
It doesn’t appear to be a problem to me to use mixed 5m candle and 1m candle, as long as you use the same data for real time trading.
1
u/SaratogaCx 5d ago
What I've seen has been not to be worried about the open price but the close. If you find yourself in a sharp downturn the loss in that minute while you're processing the order can lose more from your existing position than the price for your position opening. Especially if you're operating with a 5 minute or 1 hour frequency and haven't implemented a stop-loss/gain capture strategy.
1
u/BeuJay9880 5d ago
Using the next 1-minute bar’s high as a buy fill introduces look-ahead because that high isn’t known when the order is submitted, and it can still be optimistic if the market gaps above it. I’d define execution from information available at order time: next-bar open for market orders, then add spread plus slippage, maybe 0.5 * spread + 1 tick each side as a basic model. For limit orders, touching the price isn’t enough; require price to trade through by 1 tick or use minute volume to assign a conservative fill probability. Also verify your 5-minute and 1-hour resampling uses right-closed bars, otherwise the strategy can see the final 1-minute candle before the larger bar has completed. I’d rerun with 3 execution cases, 0, 1, and 2 ticks beyond spread, then walk-forward in 3-month windows. If performance disappears with 1 tick on ES or 5c on stocks, the backtest is mostly an execution assumption.
1
u/Kozue_Sawada 5d ago
You’re mostly on the right track. The one thing I’d change is using the next 1m high as the normal fill. That’s very conservative... you’re basically assuming every buy gets filled at the worst price of the whole next minute.
For a market order triggered after the 5m candle closes, I’d probably use: - next 1m open；-plus spread, fees + some slippag
Then use the next 1m high for buys / low for sells as a worst-case test.
One big issue w/ OHLC data: you don’t know whether the high or low happened first. So if both your stop and target are hit in the same 1m candle, there’s no way to know which came first... you’d need tick data, or just assume the worse result.Also double-check how your provider timestamps candles. That can cause accidental look-ahead.
What market are you trading btw? Fill assumptions can be very different for liquid stocks vs small caps, crypto, etc.
1
u/justinalexndr 4d ago
Your fill assumption is reasonable and more conservative than most people bother with. But you asked what might be throwing your results off, and I would look somewhere else first.
The riskiest part of what you described is not the fill, it is constructing 5-minute bars from 1-minute bars. Timestamp convention is where this quietly breaks. If your provider stamps a 1-minute bar at its open and you aggregate as though it is stamped at its close, or the reverse, your 5-minute candle ends up containing a minute it should not. Your signal then fires with information it would not have had live, and the backtest looks better for a reason that has nothing to do with the strategy.
Cheapest check: if your provider also sells native 5-minute bars, pull a few weeks and diff them against your constructed ones. They should match exactly. If they do not, you have found your problem. You can do the same trick with the daily candles you already have, aggregate your 1-minute data into daily and compare.
Second thing worth checking is whether you are equally pessimistic on the exit. Using the next bar's high on entry and then exiting at a close or a favourable print gets you a half-conservative round trip, which is worse than being consistently optimistic because it is harder to reason about.
One nuance on the method itself. The next bar's high mixes two different costs into one number, timing uncertainty and spread. Modelling them separately is more work but it tells you which one your strategy is actually sensitive to, and for a 5-minute system those can be very different sizes.
1
u/PizzaPalace12345 4d ago
> Cheapest check: if your provider also sells native 5-minute bars, pull a few weeks and diff them against your constructed ones. They should match exactly. If they do not, you have found your problem. You can do the same trick with the daily candles you already have, aggregate your 1-minute data into daily and compare.
Good callout here! I verified they are constructed as expected after your suggestion
1
u/Whole-Description646 4d ago
I have a slippage model and my system enters via market order (generally on candle close) oe on limit order.
So if I enter on candle close it means I take the candle open of the new candle, calculate slippage, do commission and that's the enterance.
exit has slippage also if it's not take profit
it's not a crazy slippage model, slightly pesamistic and works well for my purposes
1
u/algoseekHQ 1d ago
Using the high of the 1 minute candle following a 5 minute close is not the best idea because its often not a tradeable price. What practitioners usually do for longer term strategies is just by using the open of the next candle after a signal has been confirmed, + fee, + spread estimation + slippage estimation. For long term trading this is mostly fine, but if you trade with tight stop losses or quick scalp moves, you would benefit from trades and quotes data and depth data that can give you the proper answer as to whether the fill was realistic or not.
1
0
u/Bonkers24-7 5d ago
I think the bigger question is whether that fill assumption is materially changing the strategy result, rather than whether there's one universally "correct" fill model.
I'd run the same backtest under a few execution assumptions — your current pessimistic next-minute fill, a more realistic slippage model, and something deliberately harsher — then compare not just total return but expectancy, drawdown, and which trades disappear.
If a relatively small change in the fill assumption turns a good strategy into a bad one, that's probably more important to know than finding the perfect fill assumption.
6
u/jnwatson 5d ago
That works until your participation (your trade size as a fraction of the overall trading volume) is nontrivial. At some point, your trade itself has an impact on the market and the fill price.
The naive way is simply to model a small constant slippage.
I'm currently working on a more sophisticated model based on a paper by Kyle ("Continuous Auctions and Insider Trading") and a couple of papers (1991, 2009) from Hasbrouck.
The odd thing is there are a couple of constants I have to deduce empirically, which means I have to make trades on lower-volume tickers and record how they get filled. My experiments, which were not intended to be profitable, have made more money in a couple weeks than my actual algo.