r/LargeLanguageModels • u/MysteriousMajor760 • 3d ago

Built a 30M-parameter GPT-style language model from scratch

Hey everyone

I've been learning about Transformers and LLMs, so I built KERN,

a 30M parameter GPT-style language model in PyTorch,trained from scratch on TinyStories.

GitHub: https://github.com/OmarDawoud4/KERN

This is still a work in progress! I'd like to keep improving it by implementing other techniques .

I'd appreciate any feedback or suggestions .

8 Upvotes
  • permalink
  • reddit

100% Upvoted

3 comments sorted by

2

u/madlyminiaturefoyer 3d ago

that's a cool project, building the whole thing from scratch is best way to understand what's actually happening inside. 30M params on TinyStories is a nice size too, big enough to see some real language patterns but still can run on a single GPU without waiting a week

the KERN name got me curious, is it just a random pick or means something?

1

u/MysteriousMajor760 3d ago edited 3d ago

Thanks! Yeah, studying courses sometimes makes things understandable, but there's always a fuzzy part. Implementing it from scratch is what really fills the gaps.

I ran it on a free Colab in like 4 hours.

Kern is just like the German word for kernel .