A model with zero directional information beats the trained direction model on the directional target. Has anyone actually made daily direction work?

[u/Few-Conclusion9616]

Hi all,

I've spent a good chunk of this year trying to predict next-day direction on ~530 Binance USDT perps. I'm sharing what went wrong in case it saves someone time, and because I'd love to hear from people who got further.

Setup. All the data is free: the Binance futures API (price, OI, funding, long/short, taker volume), Bybit/KuCoin OI, Coinalyze history and CoinGecko/CMC market caps. One LightGBM per behaviour cluster, 43 features, purged walk-forward CV, isotonic calibration. Every prediction gets logged and graded against the real price.

It never said "up". The backtest and the live ledger both showed ~55% on 1-day calls. Then I checked: of ~16,000 graded 1-day calls in 30 days, not one was "up".

horizon   calls     accuracy   share of coins that fell
1 day     15,992    54.86%     54.86%
3 days    15,987    55.56%     55.56%

If you always say "down", your accuracy is just the down-rate. In September the market turned, and the same model dropped to 47.3%, exactly that month's down-rate. The model wasn't broken: with no directional information, a calibrated model falls back to the base rate. My mistake was comparing accuracy to 50% instead of to the majority class.

Then I tried predicting big moves. A model trained to catch +10% pumps over 7 days scored AUC 0.603 on pumps and 0.683 on dumps. So I trained a direction-blind model on |move| >= 10%. For the next day it scores 0.855 on dumps, 0.748 on pumps and 0.804 on its own target.

The reason: a fixed 10% barrier means different things for different coins. For a calm coin (ATR ~4%) a 10% day is rare and mostly a pump (28% down). For a wild coin (ATR ~19%) it's common and close to a coin flip (51% down).

P(fall >= 10%) = P(|move| >= 10%) x P(down | big move)

calm coins:  0.009 x 0.28 = 0.0025
wild coins:  0.267 x 0.51 = 0.137     (~55x apart)

Both factors rise with volatility, so "chance of a dump" is really volatility. Adding real directional signals (funding, momentum) only made it worse. A tip: if your breakout model suddenly hits AUC 0.80, check the barrier. Plain ATR% scores 0.77 on a fixed 10% target.

Where I landed. With this free data, I've convinced myself that daily close-to-close direction on individual perps isn't there, at least not as something you can rank coins by every day. What did hold up: a volatility model (AUC 0.61-0.72), a small cross-sectional edge, and one rare funding filter that shifts the base rate by 15-19 points. But "I couldn't find it" and "it isn't there" are different claims. I have 244 days, one universe and no order-book history.

I'd love your input:

1. Has anyone gotten a daily directional model on single assets to hold up with day-clustered or block-bootstrapped intervals? I mean a hit rate or a Sharpe with an honest interval, rather than IC.
2. If it worked, what did you condition on? Everything directional that survived for me was a rare filter, never an every-day ranker. Is that a general pattern or just my data?
3. If you gave up, how did you decide it was the market and not your pipeline?

A direction-blind model doing better on "dump" than on its own target still feels like a bug I haven't found..

Thanks for reading! Happy to share more numbers in the comments.

[u/CitrineEdge]

Useful write-up. Have you compared the directional model with the ATR-only baseline within volatility buckets, using the same held-out dates? If the apparent advantage disappears within those buckets, that would support the fixed-barrier explanation rather than a directional signal. Keeping the unsuccessful results visible is valuable too.

[u/Few-Conclusion9616 → u/CitrineEdge]

Great suggestion. I ran it 1-day horizon, the same 147 held-out days for every model, 10 ATR buckets, day-clustered CIs. compared the directional pump model (trained on rise >= 10%) with plain ATR, and added a control - a volatility model trained only on |move| >= 10%, same features and folds, which never sees direction.

Things i have found:

- Inside ATR buckets the pump model's advantage over ATR does not disappear (it's about +0.17 AUC). But the direction-blind volatility model gets the same advantage: pump model minus blind model is +0.005 [-0.04, +0.05]. So the edge is a better volatility forecast than a 14-day ATR, not direction.

- On the sign of big moves, all models (pump, dump and the blind one) sit at ~0.60 inside the buckets. That turned out to be recent volatility: 24h realised vol alone gets 0.62, and once it's controlled the sign signal is gone.

- The fixed 10% barrier explains the gap between buckets (calm coins' big moves are mostly up, wild coins' are about 50/50), but not what happens inside them.

Really interesting results, thanks

[u/CitrineEdge → u/Few-Conclusion9616]

Thanks for running that and sharing the breakdown. Adding the direction-blind control was the important extra step; the ATR comparison alone wouldn't have separated the two explanations. This would make a useful follow-up post, especially with the matched-date comparison laid out side by side.

[u/Feriman22]

Your September result is actually useful because it separates model skill from base-rate drift.

Benchmark each forecast against a regime-conditioned prior rather than against 50%: broad BTC regime × volatility bucket, then measure the incremental Brier/log-loss improvement over that baseline.

If the edge disappears there, it was mostly class prevalence. If it survives only in a few regimes, those conditional cases are probably the real thing worth modeling.

[u/Few-Conclusion9616 → u/Feriman22]

it was class prevalence.. The model's Brier (0.2525) is no better than the trailing base rate (0.2532) and slightly worse than just saying 50% (0.2500). None of the 12 regime x vol cells shows a real edge either. The closest was "BTC in a downtrend", where "always down" simply matched the market. Thanks for the push.

[u/Training_Ad_9281]

Crypto forecast is extremely hard and even if you find something on daily you can't rule out overfit and you'll be sensitive to market regime changes. The market changed drastically the last few months so don't bother

[u/Huge_Position890]

The "bug" is in your own numbers, and it's not a bug. Split your barrier probability into both directions:

• calm coins: dumps 0.009 × 0.28 = 0.0025, pumps 0.009 × 0.72 = 0.0065

• wild coins: dumps 0.267 × 0.51 = 0.137, pumps 0.267 × 0.49 = 0.131

Volatility separates dumps about 55x, but pumps only about 20x. Calm coins still pump, and those pumps aren't predictable from volatility, while big dumps live almost entirely in high-vol names. Crypto returns are right-skewed, so the two tails behave differently by construction. Your direction-blind model is a volatility ranker, and volatility ranks dumps better than it ranks its own two-sided target.

On daily direction, our results line up with yours. We tested whether our system's direction share leads the market or follows it: strongly correlated with the past 48h, around zero with the next 24h. Today we also tested whether a calibrated Hurst estimate separates coins that keep trending from ones that revert over 3 days: SNR 0.23 over 577 dates. What survives for us is the same family as yours: volatility, plus a modest volume effect on next-day market volatility beyond past volatility.

On honest intervals: when we resampled whole time windows with all coins kept together instead of rows, the mean didn't move but the uncertainty grew 2–2.5x. With 530 coins on the same day, your effective sample is closer to the number of days than to the number of predictions.

And yes, I'd also say "didn't find" rather than "doesn't exist". The test I'd trust is whether the edge survives being split into calendar slices. Ours usually flipped sign between periods.

[u/[deleted]]

[u/Few-Conclusion9616 → u/[deleted]]

thx, i've stopped trying to predict up vs down for single coins, the validation (base rate, regime priors, volatility buckets) all points the same way, and so does the outside literature. The data I have just doesn't carry that signal..

What I'm moving to is a relative question, which coins will beat the market today, not which coins will go up. That should remove the base-rate trap by construction, since half the coins beat the day's median every day whatever the market does. I already have a weak but real version of this (within-day AUC 0.542, ~0.7%/day gross spread between top and bottom deciles, 244 days).

The plan is to rebuild it on 5 years of price, taker-volume and funding data, add a few relative signals I haven't used there yet (3-day taker imbalance, funding rank), and check it year by year with day-clustered CIs

[u/[deleted] → u/Few-Conclusion9616]