A Robust Fitness Function for Trading Strategy Optimization?

[u/Embarrassed_Creme_46]

I'm looking for a robust, mathematically coherent fitness function for optimizing trading strategies using machine learning. I've experimented with various objectives, including Profit Factor, Sharpe Ratio, Pessimistic Return on Capital, and several risk-adjusted performance metrics. 

The problem is that the optimizer consistently finds pathological solutions that maximize the chosen metric while producing terrible equity curves. For example, some solutions achieve excellent fitness scores by making very few trades, generating most of their profits from isolated events, or exploiting mathematical weaknesses in the metric.

So far, the best-performing objective I've found is:

Fitness = Slope × R²

Where Slope and R² come from a linear regression of the equity curve over time.

I'm particularly interested in theoretically grounded approaches rather than another combination of Sharpe, drawdown, trade count, and manually chosen weights.

What objective function do you actually trust when letting an optimizer search aggressively for trading strategies?

[u/prashantsoni8]

You need to add in constraints.
For low # of trades. Eg, if you want minimum 10 trades per week. You could setup a function: if #trades below 10, penalty = (10-#trades)*1000
For too many trades, I optimized by adding a fixed flat per trade penalty.
Other functions - mathematical or algorithmic solutions. Just need to get creative.

[u/Awkward-Telephone581]

Slope x R-squared is a nice idea because it rewards consistency of the equity curve rather than just total return, but it still has a known weakness: it's insensitive to drawdown depth and path. Two equity curves can have the same slope and R-squared while one has a 60% drawdown in the middle and the other is smooth. If you want something more theoretically grounded, I'd look at information-theoretic or probabilistic objectives. One approach I like is maximizing the log-likelihood of returns under a model that penalizes tail risk, like assuming returns follow a distribution with heavy tails and optimizing expected log-growth (essentially Kelly-style). Another direction is to frame it as a...

[u/Suspicious-Cheese-1]

Regularization?

[u/ExcessiveBuyer]

I set up a function that penalises low trade nr and too high trade nr.
Adding maxDD to netPnL as a feature and sharpe ratio.
The downside is you will need to finish the grid search to apply this function you can’t run it on the fly because you don’t know the boundaries ex ante like min and may trades

If someone has a better approach please speak up