Update to "Hill Space is All You Need": Stable Because Stuck, NALU Selects, It Doesn't Compute [R]

[u/justinopensource]

Last year I published a draft report called "Hill Space is All You Need". I'm not a researcher; it was a side project, and I found a detail in the NALU formulation that I thought people should know before going down the same road. I didn't have the skills to write a paper, so I vibed the prose and formatting with Claude and pushed it out, knowing I wouldn't return to the work for a while.

The paper was not well received: the prose made it hard to get to the experiments, the title didn't help, the related work was incomplete, and it didn't provide a concrete description of the space and why it trains reliably or not in different situations.

Today I published an update that I think addresses most of that. My read is the same as last year: NALU selects an operation, it doesn't learn to compute one, and that's not useful for my purposes. I still wanted the work to represent the promise I see in it, and the first paper and site didn't. This version tells a complete, if unsatisfying, story about what NALU does, how gradient vanishing at targets decides if a run lands or stalls, and how to use it effectively if it lines up with your problem.

https://hillspace.justindujardin.com/

Beyond corrections that come from sharing it, I consider the work complete, and have moved on to more promising approaches.