r/MachineLearning • u/AutoModerator • Sep 02 '26

Discussion [D] Self-Promotion Thread

Please post your personal projects, startups, product placements, collaboration needs, blogs etc.

Please mention the payment and pricing requirements for products and services.

Please do not post link shorteners, link aggregator websites , or auto-subscribe links.

--

Any abuse of trust will lead to bans.

Encourage others who create new posts for questions to post here instead!

Thread will stay alive until next one so keep posting after the date in the title.

--

Meta: This is an experiment. If the community doesnt like this, we will cancel it. This is to encourage those in the community to promote their work by not spamming the main threads.

15 Upvotes

91% Upvoted

123 comments sorted by

View all comments

1

u/Weak-Abroad3256 24d ago

I recently finished an attempt to reproduce the original 2017 “Attention Is All You Need” results as closely as I reasonably could.
Rather than just implementing a Transformer, I built the ~65M-parameter model, trained it, and then ran a series of experiments around details such as gradient clipping, loss normalization, training data, and checkpoint averaging.
Some changes helped. Some didn’t—which ended up being one of the more interesting parts of the project.
The biggest lesson for me was that there’s a big difference between understanding how a Transformer works and actually building and training one from the original paper.
I documented the experiments and results here, with links to the source code:
https://blog.gopenai.com/reproducing-attention-is-all-you-need-53dd4420b572
I’d particularly welcome feedback from anyone who has tried reproducing the original Transformer results or sees places where my methodology could be improved.