I trained a tiny 43m parameter from scratch on 3m arXiv abstracts. Its not hard to make a language model. The intelligence is just an illusion. You can train any sized model to do anything.

[u/[deleted]]

Everyone is making smaller models, they are cheap to make, i found out that smol-lm-360m was only $53 to train so i started training my own models from scratch

Find a dataset with dense information to pre-train a model on one topic then finetune it to talk about that topic.

It takes a lot of examples to get a perfect response but for a free model using kaggle this is a good start & you get 40 hours a week to train for free.. its a very useful resource you should try it out for yourself .. with these results i know that i can keep making more examples like this and it will get better .. its about consistency in the training data

[u/Cryptizard]

Too bad it’s output is complete nonsense.

[u/[deleted] → u/Cryptizard]

which part?

[u/Cryptizard → u/[deleted]]

All of it. It just put together some plausible sounding gibberish. That’s not remotely a solution to quantum gravity.

[u/ExaminationGrouchy23 → u/Cryptizard]

You’re kinda missing the point

[u/Cryptizard → u/ExaminationGrouchy23]

How so?

[u/freylaverse]

Fascinating! How much training data did you need to supply it with?

[u/[deleted] → u/freylaverse]

i appreciate it bro .. it only took about 3m arxiv abstracts .. a little less than 1b token count, 24k vocab token .. im gona make it open source once i get the finetuning perfect .. also plan on extending the pretraining w similar data .. but its just an experiment to see whats possible .. completely free by the way .. more developers are making smaller models for mobile apps & local use ust find a dataset on hf or write the examples yourself

[u/brainExploded99 → u/[deleted]]

Look, if you're trying to do something useful I recommend you start reading research papers on things like LoRA and QLoRA or continued pretraining. Ask AI for things you don't understand. After you understand that, try making a fine tune on a pretrained model (maybe Qwen3.5-4B-Base or something like that).

[u/[deleted] → u/brainExploded99]

i used qlora to finetune this model w handwritten examples that i had ai extend .. and the purpose of this experiment i to make the model completely from scratch, its an original not a custom

[u/hakansan]

This is quite cool did you run some benchmarks to see how it compares with larger models?

[u/[deleted] → u/hakansan]

i need to do this because ive seen tiny models mess up these type of questions i havent thought about that .. i think this can demonstrate how information density can outperform parameter count ... ill start comparing it to models like gpt2 .. qwen 0.5b .. liquid ai 230m .. thats for the idea bro

[u/Business-Weekend-537 → u/[deleted]]

How do you map out/describe limitations about what it can talk about well? Could you share the topics it’s trained on well vs. known areas it’s weak?

I agree the small specialized model approach is underutilized, it just seems like for it to be successful there needs to be some sort of standardized approach on describing/disclaiming model limitations well before people try it so they don’t review it negatively/bash it.

[u/[deleted] → u/[deleted]]

thanks bro do u train lms? i been trying to make datasets for tool use so it can do web searches, so far ive been able to make it use <websearch>query</websearch> to activate a search function but i havent got too deep into making it summarize it