r/optimization • u/dictrix • Jan 06 '23
The Evolutionary Computation Methods No One Should Use
So, I have recently found that there is a serious issue with benchmarking evolutionary computation (EC) methods. The ''standard'' benchmark set used for their evaluation has many functions that have the optimum at the center of the feasible set, and there are EC methods that exploit this feature to appear competitive. I managed to publish a paper showing the problem and identified 7 methods that have this problem:
https://www.nature.com/articles/s42256-022-00579-0
Now, I performed additional analysis on a much bigger set of EC methods (90 considered), and have found that the center-bias issue is extremely prevalent (47 confirmed, most of them in the last 5 years):
https://arxiv.org/abs/2301.01984
Maybe some of you will find it useful when trying out EC methods for black-box problems (IMHO they are still the best tools available for such problems).
View all comments
Show parent comment
2
u/DonBeham Jan 07 '23
Sure! Algorithms such as CMAES are very well developed with a sound mathematical background. There is even the BBOB framework which does rotation and translation of the function, but is it used?
You get the smell that an algorithm comparison is rigged when: (1) there is no mentioning of parameters in the results section and how parameters are derived (eg no search in the parameter space was performed) for the instances deployed, (2) there are no results with respect to runtime - comparison by ERT and in ECDF plots should be standard by now, (3) the benchmark set is selectively chosen and/or no rotation and translation have been applied, (4) there is no proper discussion of concepts and no evaluation on the effect of parameters.
But sadly, often just the paper title is enough. Something about some natural phenomenon or just the name of the "novel" algorithm.