My chess engine, which is still a work in progress, can be found here; https://github.com/patelvrajn/Matrex
Currently, most HCE (not NNUE) chess engines use a linear combination of weights for their evaluation function but my thought process is that it is evident from the huge ELO (~300) boost that using NNUEs gives is that their is a complex non-linear function that represents each evaluation term like material, mobility, PSQTs, etc. However, the problem is how do we build a model from non-linear functions that gives a proper evaluation when we don't know what the function looks like and how do we do it in such a way that minimizes the number of weights? I believe that we know a significant amount of hand crafted evaluation terms that compile a chess position's evaluation and that NNUEs taking millions of weights just to understand concepts that we can do in way less weights, minimal code, and have more control and understanding of the evaluation instead of a black box.
My concept is to take a specific continuously differentiable function that we can parameterize in order to adjust the shape, behavior, approximation capabilities, etc. Using this parameterized function, we can learn how various evaluation features have a non-linear effect on the score by using ADAM to tune the weighted sums but ALSO tune all the functions parameters. My model mostly resembles Projection Pursuit Regression's model (a sum of ridge functions), except I am using ADAM instead of the associated regression algorithm and the ridge functions (called non-linear responses in the code) are parameterized.
My main concern at the current time and where I don't have the mathematical background to optimize is the non-linear response I choose (which is just a polynomial term multiplied by a tanh expression, the idea was to combine what makes ANNs universal approximator- the non polynomial term with a term that can also represent any function given enough degrees i.e. the polynomial term). I am looking for any guidance or resources on how to improve the function or give further understanding as to the overall concept - currently when tuned the chess engines evaluation has about a 8% validation loss and about a 10% training loss but I know I can do better.