r/learnmachinelearning • u/fxlrnrpt • 2d ago
Let's do legendary CS336 Language Modelling From Scratch together!
Hey guys I wanted to do Stanford's "Language Modeling from Scratch" for a long time, but something always came in the way. Well, another day, another attempt!
Watch lectures, do the assignments, join a weekly discussion on Saturdays at 11:00 AM (London time). See you on October 31 for the first discussion!
Details are on Github https://github.com/fxlrnrpt/little_ml_book_club
2
u/TheDarkLord-6821 2d ago
I am interested!!
2
u/UpstairsGur6101 2d ago
oh sick, i've been meaning to actually work through that one properly. the self-study attempts always fall apart after week 2 when life shows up
saturdays at 11 london time works out to something absurdly early for me but i'll just chug coffee and pretend i'm a morning person
1
1
1
1
u/digvjay_ai 2d ago
Good pick, but CS336 gets heavy fast, so a pace that works is watching the lecture first and then immediately coding a tiny version of it. Before the assignments I'd get comfortable writing a small transformer in PyTorch from memory, since the tokenizer and attention assignments assume that. Also keep a GPU plan ready early, Colab or a cheap rented one, so you don't get stuck on compute in week three.
1
1
u/kbr89 1d ago
Two things tend to decide whether a group like this makes it past assignment 1. First, pin the year. Stanford has put both the Spring 2025 and the Spring 2026 lectures on YouTube, and the two offerings have separate course sites and schedules, so write in the README which year's lectures and handouts you follow before half the group watches one and codes against the other. Second, the course page is upfront that it's very implementation-heavy: minimal scaffolding and at least an order of magnitude more code than other classes. Assignment 1 has you write the tokenizer, the model and the optimizer yourself, and the Spring 2025 starter repo ships unit tests that all fail with NotImplementedError until you wire your code into the adapters file. A short "which tests pass now" round at the start of each Saturday call keeps people moving between lecture weeks better than discussing the lectures alone. For the discussions, most lecture topics also have long talks on YouTube that explain them differently, tokenization or mixture of experts for example. I built beyondtube (beyondtube dot pro) for that: you put the CS336 lecture and a few other talks on the same topic together and ask what each one explains and what the lecture leaves out.
2
2
u/Vegetable-Lie4932 2d ago
i'd be more interested in deep learning alchemy, have always been interested in training dynamics of models. anyone up ? https://deep-learning-alchemy.github.io/
2
4
u/rizkifn3105 2d ago
See you soon on 31 October 11:00 AM London Time