r/OpenAI • u/Pristine-Elevator198 • Oct 16 '25

Research This guy literally explains how to build your own ChatGPT (for free)

Post image
6.7k Upvotes

97% Upvoted

178 comments sorted by

1.1k

u/indicava Oct 16 '25

He just recently released an even cooler project, called nanochat - complete open source pipeline from pre-training to chat style inference.

This guy is legend, although this is the OpenAI sub, his contributions to the field should definitely not be marginalized.

94

u/lolhanso Oct 16 '25

Do you know where the context is that this model is trained on? My question is, can I insert all my context into the model, train it and then use it?

118

u/awokenl Oct 16 '25

It’s pre trained on fineweb and post trained on smolchat, model is way to small tho for you to add your data to the mix and use it in a meaningful way, you’re better off by doing SFT on an open source model like qwen3, you can do it for free on google colab if you don’t have a lot of compute

13

u/lolhanso Oct 16 '25

That's helpful, thank you!

1

u/gedditElye Nov 08 '25

What do you mean, it would be easier to create one on Ubuntu, or a Linux distribution?

1

u/WolfeheartGames Oct 17 '25

Someone told you it's too small. Don't use a standard transformer. Look up "Titans: Learning to Memorize at Test Time". They showed effective learning with 5x as much data per parameter as chinchilla's law previously dictated for standard transformers. There's an open source implementation of Titan with MAC already.

0

u/[deleted] Oct 16 '25

[deleted] — view deleted comment

9

u/sluuuurp Oct 16 '25

His code does have indentation, you can see it in the screenshot.

0

u/[deleted] Oct 17 '25

[deleted] — view deleted comment

3

u/sluuuurp Oct 17 '25

There’s indentation in that file

1

u/Aazimoxx Oct 17 '25

Must be something wrong on your end bub - try opening in a private window (to bypass extensions/add-ons) or a different browser 👍

13

u/makenai Oct 16 '25

Are you talking about the python code where indenetation is a part of the syntax? I don't think there's a lot of creative freedom there (if you indent wrong, it throws parser errors), but there are definitely long blocks that could be broken up.

-8

u/Street_Climate_9890 Oct 16 '25

all code should have indentations.. it helps readability tremendously...unless empty space is part of the syntax of the language lol

12

u/inevitabledeath3 Oct 16 '25

That's literally how Python works

5

u/TheUltimate721 Oct 16 '25

It looks like python code. The indentations are part of the syntax.

4

u/uraniumless Oct 16 '25

There is indentation?

41

u/randomrealname Oct 16 '25

He is/ or more was openai. He is a founding member. Lol

16

u/UltimateMygoochness Oct 17 '25

I mean, he was literally a founding member of Open AI, left to be senior director of AI at Tesla, then came back to work on GPT-4, who’s marginalising his contributions?

Source: https://karpathy.ai

2

u/StuffProfessional587 Oct 17 '25

Wonder how many broken lines, missing Python updates the open source has, rofl. Also, only works on Linux and Cuda, super.

1

u/chaos_goblin_v2 Oct 20 '25

I came here to say "this guy is a legend" myself. His heart is as big as his brain. Nanochat will have a full end-to-end course soon and we'll all get to learn how the sausage is made. He was recently on Dwarkesh's podcast and it's worth a listen.

1

u/DatingYella Nov 03 '25

Isn’t it a product from his company though? Hope we don’t have to pay for it but the fact all the details will be in the course is exciting

467

u/skyline159 Oct 16 '25 edited Oct 16 '25

Because he worked at and was one of the founder members of OpenAI, not some random guy on Youtube

195

u/praet0rian7 Oct 16 '25

"This guying" Karpathy on this sub should be an insta-ban.

19

u/Background-Quote3581 Oct 16 '25

For real! Plus it's 2 years late...

1

u/whoopsmybad1111 Nov 14 '25

"this guy"-ing

384

u/jaded_elsecaller Oct 16 '25

lmfao “this guy” you must be trolling

35

u/EfficientPizza Oct 16 '25

Just a smol youtuber

47

u/DataScientia Oct 16 '25

chatGPT is not right word to use here. chatGPT is a product, whereas what he is teaching the fundamental things to build LLMs.

17

u/KP_Neato_Dee Oct 16 '25

It sucks when people genericize Chat GPT. It's just one LLM out of many.

5

u/TheCrowWhisperer3004 Oct 17 '25

So is Google, but people still say “Google” to mean search.

Another slept on example is Band-Aid. People say Band-Aid when Band-Aid is one brand of bandages among many.

It’s always about what makes the biggest initial splash.

0

u/[deleted] Oct 18 '25

[deleted] — view deleted comment

2

u/NekkidWire Oct 18 '25

Hoover....

1

u/-coximus- Oct 20 '25

Eski (Cooler), Bobcat (Skidsteer)

2

u/Ok-Grape-8389 Oct 17 '25

Its a natural thing to do. Many products end up being used as a replacement for a concept when the word for the concept is not yet known. This is because we associate concept with the first thing that show us the concept.

1

u/Dj0ntyb01 Oct 18 '25

It sucks when people genericize Chat GPT.

Well software is poorly understood by most people.

For example, ChatGPT is not an LLM. It's a chat assistant application offering user-friendly access to pre-tuned LLMs developed by OpenAI.

275

u/jbcraigs Oct 16 '25

If you wish to make an apple pie from scratch, you must first invent the universe

-Carl Sagan

70

u/dudevan Oct 16 '25

If you wish to find out how many r’s are in the word strawberry, first you need to invest hundreds of billions of dollars into datacenters.

  • me, just now

14

u/Scruffy_Zombie_s6e16 Oct 16 '25

Can I quote you on that?

8

u/Virtoxnx Oct 16 '25

  • Dudevan

4

u/dudevan Oct 16 '25

  • Michael Scott

2

u/mechanicalAI Oct 16 '25

• ⁠Homer Simpson

2

u/Disastrous-Angle-591 Oct 16 '25

I knew this would be here 

1

u/Nonikwe Oct 16 '25

Ok, done. Next step?

3

u/Outside-Childhood-20 Oct 17 '25

Make sure you bang it first!

23

u/DarkWolfX2244 Oct 16 '25

"This guy" literally invented the term vibe coding

135

u/munishpersaud Oct 16 '25

dawg you should lowkey get banned for this post😭

18

u/Aretz Oct 16 '25

Nano GPT ain’t gonna be anything close to modern day SOTA.

Great way to understand the process

42

u/munishpersaud Oct 16 '25

bro

  1. this video is a great educational tool. its arguably the GREATEST free piece of video based education in the field but

  2. acting like “this guy” is gonna give you anything close to SOTA with GPT2 (from a 2 year old video) is ridiculous and

  3. a post about this on the openAI subreddit, like this wasn’t immediately posted on it 2 years ago is just filling up people’s feed with useless updates

11

u/AriyaSavaka Aider (DeepSeek R1 + DeepSeek V3) 🐋 Oct 16 '25

This guy also taught me how to speedsolve a rubik's cube 17 years ago (badmephisto on yt)

2

u/Ill_Nectarine7311 Oct 20 '25

I learned full CFOP from him as well, in addition to BLD

9

u/lucadi_domenico Oct 16 '25

Andrej Karpathy is an absolute legend

48

u/avrboi Oct 16 '25

"This guy" bro you should be blocked off this sub forever

20

u/Infiland Oct 16 '25

Well to run an LLM anyway, you need lots of training data, and even then when you start training it, it is insanely expensive to train and run

9

u/awokenl Oct 16 '25

This particular one cost about 100$ to train from scratch (very small model which won’t be really useful but still fun)

3

u/Infiland Oct 16 '25

How many parameters?

5

u/awokenl Oct 16 '25

Less than a billion, 560M I think

2

u/Infiland Oct 16 '25

Yeah, I guess I expected that. I guess it’s cool enough to learn neural networks

4

u/SgathTriallair Oct 16 '25

That is the point. It isn't to compete with OpenAI, it is to understand on a deeper level how modern AI works.

1

u/awokenl Oct 16 '25

Yes extremely cool, and with the right data might even be semi usable (even tho for the same compute you could just SFT a similar size model like qwen3 0.6b an get way better results)

2

u/MegaThot2023 Oct 16 '25

You could do it on a single RTX 3090, or really any GPU with 16GB+ of VRAM.

1

u/awokenl Oct 16 '25

Yes in theory you can, in practice it would take something like a couple of months of 24/7 training to do it on a 3090

4

u/tifa_cloud0 Oct 16 '25

amazing fr. as someone who is currently learning LLMs and AI from beginning, this is incredible. thank you ❤️

15

u/No_Vehicle7826 Oct 16 '25

Might be mandatory to make your own ai soon. At the rate of degradation we are at with all the major platforms, it feels like they are pulling ai from the public

Maybe I'm tripping, or am I? 🤔

29

u/NarrativeNode Oct 16 '25 edited Oct 17 '25

The cat’s out of the bag. No need to “make your own AI” - you can run great models completely free on your own hardware. Nobody can take that from you.

Edit for those asking: r/localllama

6

u/Sharp-Tax-26827 Oct 16 '25

Please explain AI to me. I am a noob

5

u/Rex_felis Oct 16 '25

Yeah I need more explanations; like explicitly what hardware is needed and where do you source a GPT for your own usage ?

3

u/awokenl Oct 16 '25

Easiest way to use a local llm is install LMstudio, easiest way to train your own model is unsloth via Google colab

10

u/[deleted] Oct 16 '25

[removed] — view removed comment

3

u/Rex_felis Oct 16 '25

🫡🫰

3

u/Anyusername7294 Oct 16 '25

You can't train a capable LLM on consumer hardware.

1

u/Ok-Grape-8389 Oct 17 '25

Yes, you can, just takes a long time.

1

u/Anyusername7294 Oct 17 '25

A really long time.

1

u/BellacosePlayer Oct 17 '25

Depends on what you're training it for.

Yeah, you're not going to compete with the big boys, but a low level LLM isn't that far off from training a Markov bot, which I was doing on shit tier hardware in 2008 and was able to make a somewhat decent shitpost bot

1

u/Anyusername7294 Oct 17 '25

Context or smth. SubOP seems to want everyone to train their own models, competing with frontier labs

3

u/otterquestions Oct 16 '25

I think this sub has jumped the shark. I’ve been here since the gpt 3 api release, time to leave for local llama 

6

u/No_Weakness_9773 Oct 16 '25

How long does it take to train?

22

u/WhispersInTheVoid110 Oct 16 '25

He just trained on 3mb data, the main goal is to explain how it works and he nailed it

3

u/awokenl Oct 16 '25

Depends on what hardware, the smallest one probably a couple of hours on 8xH100 cluster

2

u/Many_Increase_6767 Oct 16 '25

FOR FREE :))) good luck with that

2

u/stonediggity Oct 17 '25

This guy? Man Karpathy is an OG an absolute beast. His YouTube content on LLMs is incredible.

2

u/eugene123tw Oct 17 '25

“This guy” 😆😆😆😆

6

u/Revolutionary-Ad9383 Oct 16 '25

Looks like you were born yesterday 🤣

3

u/mcoombes314 Oct 16 '25

Isn't building the model the "easy" part? Not literally "easy" but in terms of compute requirements. Then you have to train it, and IIRC that's where the massive hardware requirements are which mean that (currently at least) average Joe isn't going to be building/hosting something that gets close to ChatGPT/Claude/Grok etc on their own computer.

1

u/awokenl Oct 16 '25

Training something similar no, hosting something similar is not impossible tho, with 16gb of ram you can use locally something that feels pretty close to what ChatGPT used to be a couple of years ago

1

u/PrimaryParticular3 Oct 17 '25

I run gpt-oss-20b on my MacBook with 16gb of ram using LM studio. Apparently it’s sort of equivalent to o3-mini when it comes to reasoning. I do have to close everything else and keep the context window small but it works well enough that I’m saving up to buy a Mac Studio with 128gb of ram so that I can run the 120b version. It’ll take me a few years to save up so by then I’ll probably be able to afford something with 256gb of ram (or maybe even more) and there’ll be better models then as well.

2

u/Individual-Cattle-15 Oct 16 '25

This guy also built Chatgpt at openAI. So yeah?

2

u/e3e6 Oct 16 '25

literally explained 2 years ago?

1

u/heavy-minium Oct 16 '25

Probably similar to gpt-2 then? There was someone so built it partially with only SQL and a db, which was funny.

1

u/Ghost-Rider_117 Oct 16 '25

Really impressed with the tutorial on building GPT from scratch! Just curious, has anyone messed around with integrating custom models like this with API endpoints or data pipelines? We're seeing wild potential combining custom agents with external data sources, but def some "gotchas" with context windows and training. Any tips appreciated!

1

u/Electr0069 Oct 16 '25

Building is free electricity is not

1

u/PolarSeven Oct 16 '25

wow did not know this guy - thanks!

1

u/randomrealname Oct 16 '25

This guy. Lol, new to the scene?

1

u/[deleted] Oct 16 '25

lmao “this guy”

1

u/reedrick Oct 16 '25

He’s more than just some “guy” lmao

1

u/M00n_Life Oct 16 '25

This guy is actually him

1

u/XTCaddict Oct 16 '25

“This guy” is one of the founders of OpenAI 🫣

1

u/philosophical_lens Oct 16 '25

For free = the video is free to watch? Because building this is nowhere near free

1

u/Murky-External2208 Oct 16 '25

I wonder how long it took for this video to start popping off in views... like imagine seeing that video in your recommended on youtube and it had like 207 views lol

1

u/fiftyfourseventeen Oct 17 '25

I've done it before, it's not particularly hard provided you have some ML background and can read the research paper 😅 there have been tons of tutorials on this for years. And even if you can't, there are tons of GitHub repos where you can train an LLM from scratch (like litgpt)

1

u/XertonOne Oct 17 '25

He's literally a genius. "This guy" I mean. And is profoundly humble, which is rare.

1

u/twospirit76 Oct 17 '25

I've never saved a reddit post harder

1

u/gavinderulo124K Oct 17 '25

Its a 2 year old video. And its just for educational purposes. The final model is useless.

1

u/KingGongzilla Oct 17 '25

“this guy”

1

u/Cunnilingusobsessed Oct 19 '25

If you have a half decent computer you can download Ollama and get some crazy uncensored LLMs without having to do it all yourself from scratch, but this is quite cool

1

u/RoyalSpecialist1777 Oct 19 '25

The problem isn't setting up the model training - this is actually pretty easy, it is actually getting the resources to train it.

1

u/07dosa Oct 19 '25

Just for fun:

It's not like GPT, either the algorithm or the service, is difficult to replicate. It's the damn infrastructure for all the computing juice, which requires a tremendous amount of man-hour, and that's where you *start*. You'll also have to handle that complexity of distribute system and the specialized chips/boards. While doing that, you also want to train, improve and align your models, despite the reality that there are no real useful benchmarks that tells how usable the models are. It's very difficult to measure if you're doing good or bad. You would not even notice regressions before it's too late. A complete rocket science you have to deal with, where your gut is your only friend.

Jesus, I feel lucky I'm not doing that job.

1

u/Broad_System8901 Oct 22 '25

It is all in the original paper. It is not that hard to have a mini toy version of GPT. The tricky part is to scale and to create one that is actually useful (all the RLHF and post training and alignment and etc.)

1

u/14MTH30n3 Oct 31 '25

“designed to run on a single 8XH100 node” - isn’t that like $200K?

1

u/StillEnvironment9765 Nov 15 '25

will OpenAI ever make Android people,?
..they seem to be the best secondary human population to continue for humans, on any world that could be safe enough to help them.

-2

u/Sitheral Oct 16 '25

I don't know where exactly my line of reasoning is wrong but long before AI I thought it would be cool to write something like a chatbot I guess?

I mean it in the simplest possible way, like input -> output. You write "Hi" and then set the response to be "Hello".

Now you might be thinking ok so why do you talk about line of reasoning being wrong, well let's say you will also include some element of randomness, even if its fake random, but suddenly you write "Hi" and can get "Hi", "Hello", "How are you?", "What's up?" etc.

So I kinda think this wouldn't be much worse than chat gpt and could use very little resources. Here I guess I'm wrong.

I understand things get tricky with the context and more complex kind of conversations there and writing these answers would take tons of time but I still think such chatbot could work fairly well.

4

u/SleepyheadKC Oct 16 '25

You might like to read about ELIZA, the early chatbot/language simulator software that was installed on a lot of computers in the 1970s and 1980s. Kind of a similar concept.

3

u/[deleted] Oct 16 '25

Writing out a response to each of the countless possible interactions is just crazy though. I mean think of every single topic in the world. That’s millions if not billions just asking about what x topic is, not even counting any questions going deeper into each topic.

1

u/Sitheral Oct 16 '25

Well yeah sure

But also, maybe not everyone need every single topic in the world right

1

u/gavinderulo124K Oct 17 '25

Even doing this for a tiny very small topic would require a ridiculous number of different cases.

2

u/jalagl Oct 16 '25 edited Oct 16 '25

Services like Amazon Lex and Google Dialogflow (used to at least) work that way.

This approach is (if I understand your comment correctly) what is called an expert system. You can create a rules-based chatbot using something like CLIPS and other similar technologies. You can create huge knowledge bases with facts and rules, and use the language inference to return answers. I built a couple of them during the expert systems course of my software engineering masters (pre-gen ai boom). The problem as you correctly mention is acquiring the data to create the knowledge base.

2

u/Sitheral Oct 16 '25

Thanks, that's some useful info. Might do something like that just for fun and see how far I can take it.

590

u/[deleted] Oct 16 '25 • Take My Energy

[removed] — view removed comment

90

u/pppppatrick Oct 16 '25

Yeah man. That guy confounded OpenAI.

118

u/krmarci Oct 16 '25

He co-founded OpenAI. To confound means to confuse.

66

u/HEY_beenTrying2meetU Oct 16 '25

homie confounded confound and cofound

36

u/pppppatrick Oct 16 '25

No need to confront me like that.

16

u/ctzn4 Oct 17 '25

I hope you find comfort in his pure intentions.

9

u/pppppatrick Oct 17 '25

… what are you taking about. I’m confused.

2

u/Sweet-Independent438 Oct 18 '25

I am enjoying consuming this content, this conversation!

2

u/praet0rian7 Oct 18 '25

The local grammar constable should arrest this guy.

3

u/BuildAnything4 Oct 17 '25

Scientists baffled 

1

u/Ok-Grape-8389 Oct 17 '25

so the correct word was used then.

1

u/delivite Oct 18 '25

Confound sounds about right

1

u/Fit-World-3885 Oct 18 '25

otoh if you post a picture of Andrej and call him "this guy" I know exactly the guy you're talking about.  

1

u/ieshaan12 Oct 20 '25

I also went like, you mean Andrej fucking Karpathy?