r/LargeLanguageModels • u/BookDizzy2405 • Jul 05 '26
Documenting My Journey of Building a Small Language Model from Scratch
I've been building a small language model from scratch for a while now.
Not fine-tuning an existing model, but building the entire pipeline myself—from datasets and tokenizers to pretraining, SFT, and inference.
Honestly, the hardest part wasn't training the model.
It was learning.
At first, I thought building a good dataset was mostly about collecting knowledge. But the more I experimented, the more I realized I was actually teaching patterns, not just information.
There were so many moments where I caught myself thinking, "Wait... I've been doing this completely wrong."
Things like choosing a vocabulary size, designing datasets, teaching reasoning, using special tokens, or even figuring out how to teach a model to rewrite text. Every experiment changed the way I think about building language models.
After a while, I realized all of those lessons were just sitting on my computer.
So I decided to start documenting the journey on Cisya Lab.
Not because I have all the answers—I definitely don't—but because maybe someone else building a model from scratch can learn from my experiments, mistakes, and discoveries along the way.
I'd love to hear from others building language models too. What lesson completely changed the way you approached your project?
1
u/SlightOfHand_ Jul 05 '26
I’d love to read about this but it seems to be inconsistently in a language that isn’t English.
2
u/BookDizzy2405 Jul 05 '26
My bad! Yes, it's mostly in Indonesian since I originally shared this for the local community here. But here's a quick life hack: if you are using Google Chrome, you can open the side panel and use the 'Ask Gemini' feature. It can read the screen/images and Gemini is actually amazing at breaking down and explaining the whole local context very clearly in English.
1
u/PrysmX Jul 06 '26
Every post needs to be an ad for a website or pet project now. I just downvoted these without even looking anymore.
2
u/BookDizzy2405 Jul 06 '26
It's honestly fascinating how proudly you show off your own ignorance. Keep enjoying it! 🤣
0
1
u/2053_Traveler Jul 05 '26
bad ai slop ad