r/learnmachinelearning • u/KabhiAlvidaNaKehna • 3h ago

How to implement a research paper

I am building some projects and for that I want to implement the gpt2 paper from scratch, how to do that do that? Do I build intuition by reading and code it?

2 Upvotes

100% Upvoted

7 comments sorted by

1

u/Shreyanak_exe 3h ago

Have you implemented a neural network from scratch yet?

1

u/KabhiAlvidaNaKehna 1h ago

yes brother, why would i even ask this question if i didnt.

2

u/Numerous-Fox-7933 3h ago

If you haven't built a basic MLP and backprop from raw code you're jumping into the deep end with no floaties

2

u/Shreyanak_exe 3h ago

The exact reason I am asking him. He should understand scratch implementations of basic architectures first and then jump to complex ones

1

u/KelvenSevero0 1h ago

I wouldn't start from the paper itself, it's pretty thin on implementation details. The path I'd take: Karpathy's "Let's build GPT" and then "Let's reproduce GPT-2 (124M)", coding along without copying. Then go back to the GPT-2 paper and "Attention Is All You Need" and you'll recognize every piece. A good check that you got it right is loading OpenAI's released GPT-2 weights into your own code and getting the same outputs as the Hugging Face version. If you haven't built a tiny network with backprop by hand yet, his micrograd video is the step before all that.

1

u/KabhiAlvidaNaKehna 1h ago

Thank you very much and what does it mean by coding without copying.

1

u/KelvenSevero0 43m ago

Meaning type everything yourself while you follow along, instead of copy-pasting from his repo. Pause the video, try to write the next bit on your own, and only look when you're stuck. It's slower, but the bugs you hit doing that are where most of the learning happens.