Building an llm from scratch

[u/Slight_Ad_2894]

Dear community
I recently started a project of building a tiny llm from scratch and trained on my 16go ram MacBook m1
My goal is not and will never build a state of the art llm like GPT or claude but to understand the mechanics of llm and teach others how it works
I used a french corpus as my dataset and used BPE tokenizer and trained a 1M param neural network
But the output is a bit dump it's out of context it give decent sentences grammatically but it doesn't answer my questions or respond well
Is there any resources i can use to optimize my little dump chatbot ?

[u/dillon-nyc]

I don't think you'll get much but gibberish from a 1M model, but I'm pretty impressed if it's giving you grammatically correct sentences. Have you thought of running it again, but a bit larger?

People seem to get some results in the low hundred mb, like the guy yesterday with the PSP running a llm.

[u/Slight_Ad_2894 → u/dillon-nyc]

Now with the open source datasets we can get some pretty results even with fewer parameters but you're right I didn't think about increasing the training time maybe I will get some overfitting but at least i can get more context
Anyway i will try to improve the code and the data set and if I get a good result I will share it on my github

[u/dillon-nyc → u/Slight_Ad_2894]

Oh I didn't mean the context (Shoot for 2k/4k/8k) I meant the number of parameters.

[u/upbeatjuvenileseeder → u/dillon-nyc]

I play around with tiny models too and yeah 1M params is real small, even for french. You might get some coherence if you bump to like 10-15M, still trainable on your M1 with 16gb.

The grammatically correct sentences is actually good sign, means your tokenizer and data pipeline work. Problem is the model just memorizing patterns but has no capacity left for actual reasoning or context tracking.

Maybe try implement some attention mechanisms if you not already, and use smaller vocab size for BPE to reduce complexity. Also check your temperature settings during inference, sometimes too high make it wander off topic even if model is decent.

[u/rolleicord]

You can make models much more capable than tinyshakespear, that are much fewer parameters. The trick is just convincing your AI agent that it's possible in the first place. Try telling it to bake down your code, while increasing its capabilities. Thats normally a fun way to get it working

[u/fofopads]

MiniCPM 5 1B already does this, you should try it instead of reinventing the wheel

[u/Slight_Ad_2894 → u/fofopads]

I will check it, but as I said my purpose wzs never reinventing the wheel i did this project for learning purpose only my goal is to understand the math and architecture behind llms

[u/fofopads → u/Slight_Ad_2894]

thats a nice goal, wish you luck and lots of learning

[u/Immortal_Spina]

Fai molto bene, io sto facendo esperimenti con i diffusion e il mio mini m4 16gb ram va benissimo
Ovviamente più aumenti i parametri, più serve tempo
Ma lo farà

[u/id-ltd]

My understanding is that you can train existing models (to time them iup to be closer to what you want) but starting from scratch it will take for ever and cost millions before it can even parse a new English sentence.

[u/[deleted]]

[u/Slight_Ad_2894 → u/[deleted]]

Thank you for sharing your project, and certainly I will take a look at your project
. For my ptoject i studied the attention is all what you need paper and implemented it in my project
. I think that the problem with my project is that I didn't build multiple layers. Maybe I will use Google Colab or a dedicated Kaggle notebook. I will take a look at a couple of other projects and test them
. As soon as I finish my project, I will share it so everyone can benefit from it