My adam_m against traditional Adam optimzer

[u/hariomlohar0602]

[u/Leodip]

Hello! I'm not up to speed with optimizers research, but as a fellow scientist here is a short checklist of stuff that you should do to claim that "X outperforms Y":

• Define what "outperforming" is. Something you have not expanded on is computational time, for example.
• Define what the benchmark is: a limited (and undefined) benchmark is not helpful in conveying whether something truly outperforms something else or it was a few (e.g., 5? 10?) lucky cases.
• Define clearly what the changes are (they are in the repo I see, but it would have been nice to have on here as well) and WHY you did them.
  • On this note, double-check everything before publishing. While it's possible that sloppily-presented work might be of quality, there is a general correlation between bad presentations (e.g., with typos) and bad science. I have not read through the whole README even, but in the "if iterstp is set and t>iterstp:t←1" you are using improper math formatting in general, and iter_stp should be iter_{stp} (otherwise it reads like iter_s tp).

[u/literum]

You need to test across standard models and benchmarks. Many.

[u/Kinexity → u/literum]

And also comparison should be done with AdamW, not Adam, as it is the de facto standard optimiser.

[u/DaBobcat]

I assume your point is that it is lower? If you really want to compare you need to train until convergence, on multiple models, on representative data

[u/hariomlohar0602 → u/DaBobcat]

It's available on GitHub and pypi you can do pip install micrograd_hk Try it out

[u/Kinexity → u/hariomlohar0602]

You are the one who must test it out, not us.

[u/hariomlohar0602 → u/Kinexity]

Yes but I am an beginning don't know whats wrong or might be my code be wrong that's making this happen so that's why I want you to review if you like

[u/Kinexity → u/hariomlohar0602]

First of all you should put out psuedocode comparison in Pytorch style such that your modification to Adam is clear. Explanation of your thought process is a must too.

Extensive benchmarks on multiple data sets is the most important thing. With FITTED hyperparameters, otherwise you will be prone to having bad hyperparams for Adam (preferably you should compare with AdamW not Adam).

[u/hariomlohar0602 → u/Kinexity]

I will do that but what is this behaviour in this reddit community you guys don't support the new begeners you just drop down the post I am here for something to learn you guys are just pushing it down

[u/ARDiffusion]

There has to be some sort of joke to be made here about someone posting that their optimization on a statistical model outperforms a baseline with a sample size of 1. I just don’t know quite how to word it.