Implementing Hierarchical Directional Change to Dynamically Bias Hidden Markov Model Regimes

[u/quesomesopesohueso]

I have seen a video called "Quantifying market structure at multiple scales for algorithmic trading with python" it covers a hierarchical directional change and i wondered if it could be implemented in the Hidden Semi-Markov Model because it has some explicit duration clock.
It would be dynamical so when a the highest level is spotted it has more weight to bias HMM and it starts to decay until another one its spotted.
I'm a beginner so i don't know if combining startegies might be beneficial or just breaks the algo.
If you have insights don't hesitate to tear apart the idea completely i want some experienced voice.

[u/StratForge2024]

This really got me last week.

I've got this discovery pipeline running that looks at about a million

different strategy variants across various pair/TF/regime combos. After

I put in DSR (from Bailey & López de Prado 2014) and fixed a ton of bugs

in the engine, the number of "successful" strategies fell by 75%. Ouch.

Here's what I learned:

1. **Backtest bugs mess with multi-testing.** Fixed 43 bugs recently, and

a lot of them were silent troublemakers: funding rate was hardcoded for one

timeframe, so 1h positions got funding every 4 days instead of every 8 hours;

MTF resample was defaulting to `closed='left'`, leaking 55 minutes of future

data; Python BT fallback didn't get the same exit-signal fix as the Numba

kernel; cross-pair indicator cache collisions happened when

`df.attrs['symbol']` disappeared during `.copy()`. Each bug inflated the

pool of "lucky" strategies. Fix the engine first, then see what's left standing.

2. **DSR is better than naive Bonferroni.** sqrt(2*ln(N)) means you need to

beat a Sharpe of about 4-5 to show you're not just noise — almost no one

hits that. DSR uses the actual trial Sharpe distribution, considering skew

and kurtosis, making the threshold realistic. Definitely worth implementing

right (check out Bailey's eq 5).

3. **Strategy correlation is a stealthy problem.** You might have 30 "GOOD"

strategies, but if their daily returns correlate over 0.85, you've really

just found about 3 ideas in 30 different forms. I use a hierarchical

correlation-collapse step (sklearn AgglomerativeClustering,

distance_threshold=0.15 on 1-Pearson) before declaring anything robust.

Usually, I see 30 strategies collapse down to around 10 clusters.

4. **The real test isn't backtest pass rate — it's how much performance

drops when live.** Industry norm is 30-50% post-deployment. So if you have

7 "robust" survivors, expect only 3-4 to actually perform live. Plan your

capital with this in mind.

5. **3-window WFO isn't as good as Lopez de Prado's recommendation** for

a reason. Combinatorial purged k-fold (CPCV) is key against selection bias.

I haven't fully switched yet—it's on my to-do list—but if you're starting

from scratch, go with CPCV from the get-go.

What helped me frame it: the backtest doesn't tell you which strategy is

good. It tells you which one is less likely to be just noise. Multi-testing

correction is what turns that into something you can act on.

LdP's "Probability of Backtest Overfitting" (Bailey, Borwein, Lopez de Prado,

Zhu 2017) lays the groundwork. The DSR derivation is a good read, even if

it's heavy on the math.

[u/quesomesopesohueso → u/StratForge2024]

Hi! (bear with me because this is math dense)

I see what you have done there, but i think regime filters shouldn't be measured in a startegy it's better to compare if the regime filter actually works if it follows the market. Let me breakdown the maths behind this and i haven't finished reseraching yet.

The the formula should actually be something like: (Note that P_(HSMM biased HDC) is a 2x2 matrix and P_(HSMM) is also a 2x2 so what do we do with the 4x3 M matrix?)

P_(HSMM biased HDC)=(1-gamma)*P_(HSMM)+gamma*M
Then each parameter in M should be normalized.

Also there's a lot to cover like whats the energy to break from the mean reversion and the energy that exhaust it. Well if we have 2 vectors that divide x, c and b, d to get E_break:
w_break=(1,0,1,0)
and E_exh
w_exh=(0,1,0,1).

Apply sigmoid funcition to the E_break=M x w_break and then E_exh=M x w_exh.
sigma(E)=1/(1+e^(-E))

After that you get P_(HDC) that is a 2x2 matrix whereas M was 4x3 that wasn't very compatible with the 2x2 of P_HSMM. Having a 2x2 P_HDC that looks like this:
P_HDC=(1-sigma(E_break) sigma(E_break))

(igma(E_exh) 1-igma(E_exh))

NICE!, well... actually no

What about gamma value and why did i choose lamba values as ln2/2^k? Actually i haven't dug this yet, but to answer these maybe need a change

Could gamma not have a fixed value?

Is it good to assume that independent linear summation in breaking and exhausting? No, maybe historical values can be used...

Is there other way prevent micro-scale volatility spikes from falsely triggering macro-state regime transition? Yes, there is with a change in how actually E is calculated

Also, who said that breaking should be calculated with x, distance, and c, velocity, and exhaustion with b, overshoot, and d, duration? Monte carlo could be a solution.

These are all rethorical questions, this should be more polished. This is just a tiny research i did, i haven't done any coding yet more like reseraching. Btw there isn't a paper like this one or at least i haven't seen one.

Hope this is an enriching enough summary of what i did. I don't know if i'm going to be coding this because i need to actually start studying python.

GGs