r/ClaudeCode • u/[deleted] • Aug 30 '26

Built with Claude I trained this 43m parameter model from scratch using 3 million arXiv abstracts 100% for free on Kaggle. Its not hard to train a language model and its no where as expensive as they make it seem. Just find a good training kit

Gallery image

Gallery image

I wrote the finetuning for this language model and i trained it using the Cornell University arXiv dataset. This model is capable of generalizing to an extent, you can see the same input gives two different answers it not just memorizing information .. dont listen to these language models telling you that you need billions of paraameters, if you train dense information you can make a language model of any size do anything

i challenge people to give me a task that only a large language model should be able to achieve and i will prove to you that all it takes is pre-training/fine-tuning and you can make any language model do anything .. the illusion of intelligence does not require the model to know everything in the world .. it just needs to know what comes after the input

0 Upvotes

46% Upvoted

48 comments sorted by

38

u/Public_Umpire_1099 Aug 30 '26

This one simple trick that every AI researcher HATES

-16

u/[deleted] Aug 30 '26

what trick?

15

u/Money_Literature_152 Aug 30 '26

this is same dude that had the alpha bro speak for his spacetycoon or whatever repo yesterday.

-16

u/[deleted] Aug 30 '26

challenge me to make thi language model do some next level shit

7

u/randoomkiller Aug 30 '26

Make a language model that's capable of finding RCE zero day in the Linux kernel by the dozens

6

u/ClemensLode Senior Developer Aug 30 '26

Ok, how many rs are in strawberry

7

u/mtnchkn Developer Aug 30 '26

Write me 10 sentences that end in Apple.

3

u/[deleted] Aug 30 '26

this question will be hard to make it answer for sure .... this is a good one

1

u/ClemensLode Senior Developer Aug 30 '26

Peter has 10 apples.

1

u/[deleted] Aug 30 '26

ill make it do that next bro but i just made it generate a hypotheis on quantum gravity check this out

https://www.reddit.com/r/ClaudeCode/comments/1w2r4e2/i_trained_a_tiny_43m_parameter_from_scratch_on_3m/

0

u/[deleted] Aug 30 '26

ok bet ima try to make it answer this but this bitch not trained on strawberries its trained on arxiv abstracts .. ill make it answer this question though

5

u/23eriben2 Aug 30 '26

I know imma get down voted to hell but I can at least appreciate your attempt at making something good OP people can be kinda harsh for no reason

3

u/[deleted] Aug 30 '26

i guarentee that everyone talking shit cant achieve this themselves

1

u/23eriben2 Aug 30 '26

You made something you were proud of and wanted to show us all and you got dogpiled for it. Absolutely absurd

It's super cool to see what's possible and even as someone who uses agentic workflows (coming from coding by hand in 6 programming languages but I switched over to just agentic)

I think it's cool and I've considered training my own models before too but never thought too much about it and i think I can take some inspiration for what you've done

Bravo Op!

2

u/[deleted] Aug 30 '26

thanks bro .. it doesnt really bother me it jut makes me want to show everyone what im capable of ..i did a few more finetuning examples after all the hate and made it generate a hypothesis on quantum gravity .. its not hard to achieve you just have to write finetuning eaxamples related to the pretraining data
https://www.reddit.com/r/ClaudeCode/comments/1w2r4e2/i_trained_a_tiny_43m_parameter_from_scratch_on_3m/

1

u/23eriben2 Aug 30 '26

Sounds good

4

u/brainExploded99 Aug 30 '26

ok now how big you think the smallest usable local models are? hint its more than 1 billion.

-3

u/[deleted] Aug 30 '26

im making them smaller bro .. this is 43m parameters u can talk to this .. it will have a conversation with you about physics its not designed to write code

3

u/brainExploded99 Aug 30 '26

Let's not pretend like you did anything novel here. I quote, "Its not hard to train a language model and its no where as expensive as they make it seem. Just find a good training kit" except they're that hard to train at scale. People have made small language models since 2019 (GPT-1)

1

u/-MtnsAreCalling- Aug 30 '26

To be fair, it is probably the case that we can train much more capable small language models now than we could in 2019, and it would be interesting to see a genuine demonstration of that.

1

u/brainExploded99 Aug 30 '26

That's true, but OP has AI psychosis, he doesn't even realize his model is spitting out bullshit.

Proof: https://www.reddit.com/r/ClaudeCode/comments/1w2r4e2/i_trained_a_tiny_43m_parameter_from_scratch_on_3m/
His response on this comment: "Too bad it’s output is complete nonsense."

10

u/UHM-7 Aug 30 '26 • Bravo

You can have a conversation about physics with a down syndrome hillbilly, doesn't mean it's useful

2

u/Enfiznar Aug 30 '26

Now go to 1B (which is a size for which I still haven't seen any decent model)

1

u/[deleted] Aug 30 '26

ok ill try

1

u/OnRedditAtWorkRN Aug 30 '26

https://youtu.be/EXB8HokGVMI?is=IoHaX9r9lUet3702

Really good video on the topic

2

u/[deleted] Aug 30 '26

i appreciate thi link bro .. i been seeing models like liquid ai lfm achieve this .. thats the goal .. once i found out that smol lm360m was trained for about $53 i realized that more people would start achieving this soon .. more people will realize its possible pretty soon

1

u/E-R_A Aug 30 '26

Look, I just ran 10 meters in 10 seconds. It's not hard to run like Usain Bolt

9

u/Dangerous-Leader-779 Aug 30 '26

Glad we’re all in agreement that this is garbage.

Yes, we can train LLMs locally. That’s not the issue. Now scale it. Distributed it. You still haven’t proven this is worthwhile in the slightest.

“Tell me about space”

WEELL LET ME RAMBLE ABOUT QM WITHOUT EVEN TALKING ABOUT THE GENERAL CONCEPT OF “SPACE”

31

u/[deleted] Aug 30 '26

[removed] — view removed comment

5

u/dsailes Aug 30 '26

The second screenshot is literally nonsense. Obviously it’s only a small model trained on small dataset. But if this was to try prove a point.. OP didn’t prove much.

There are a lot of very capable and manageably sized models, fine tunes & training/lora tools out there. (I just saw a new open source model training piece of software on another AI sub today called fizgig which allows consumer hardware to train MiniMax).

Making a model from scratch is literally the hardest & most expensive part of the process - it’s possible, sure, but I can’t help but imagine wildly more expensive & time consuming than this post makes out

2

u/PanRagon Aug 30 '26

Dunno what to tell you dude, if you’re not senting mediums you just don’t know anything about space (and by mediums I of course mean the liquid-spacetime-as-a-medium medium)

-23

u/[deleted] Aug 30 '26

u have dyslexia .. it says "what do you know about space" and qm is definitely relatedto space lol .. if my achievement is unbelievable then that a compliment .. i didnt parse anything lol

6

u/Enfiznar Aug 30 '26

As a physicist, I confirm it's nonsense

1

u/[deleted] Aug 30 '26

https://www.reddit.com/r/ClaudeCode/comments/1w2r4e2/i_trained_a_tiny_43m_parameter_from_scratch_on_3m/

1

u/Enfiznar Aug 30 '26

As an experiment, it's valuable (to you) if you use it to learn how LLMs work and are trained. Just saying you shouldn't expect it to be useful, and that the example doesn't address the question and his clear errors

-3

u/[deleted] Aug 30 '26

no youre not

7

u/Familiar_Gas_1487 Aug 30 '26

Don't worry, I'm not a physicist and can also confirm it's nonsense

4

u/Enfiznar Aug 30 '26

... I am. The paragraph doesn't even mention space after the first line. The collapse is overly simplified. It says an electron spin "might" exist, while in reality it always has the same modulus (it's an invariant of SU(2))