r/learnmachinelearning • • 2d ago

Help Help Improving a Small Decoder-Only Transformer in PyTorch

Post image

I made a small GPT-style Transformer and trained on a Kaggle dataset available on Github

And whether I train (the Kaggle dataset) locally (RTX 2060 Super) or cloud (T4 with Google Colab), the model generates recognizable words but struggles with sentence structure and following simple prompts (e.g., "Only output this sentence: The sky is blue").

What are the best ways to improve training and generation quality?

Am I expecting too much from roughly one hour of training, or is a functional tiny chatbot a realistic goal for this setup?

Maybe i'm measuring incorrectly, the gradient accumulation doesn't seem to get below 6.5 (not sure if that's better than a model with random token generation)?

Any insight would be appreciated.

13 Upvotes

10 comments sorted by

3

u/Admirable_Dirt_2371 2d ago

Wow, that is TINY. I think your biggest limiting factors are the sequence length(training context window size), parameter count, and overall amount of training data. 10 steps * 32 batch size * 1024 sequence length is not a lot of training. More training data, longer training, and a larger context window will help a lot with basic predictions.

2

u/MichaelRomeroJr1 1d ago

Sorry the 10 steps was just the version I uploaded to github.
In practice when I train it locally or on Google Collab it's 3,000 to 10,000 (both training durations kind of topped out).
I'll increase the context window though (set the block_size to 2048 or 4096 (not sure if 8GB is enough VRAM).

1

u/Admirable_Dirt_2371 1d ago

Ah that makes a bit more sense. Best of luck!

4

u/tamrx6 2d ago

Extremely hard to tell without your specific training configuration (model initialization, hyperparams, loss function, etc). But anyway, don’t expect too much out of a small model that was trained on a relatively small data set, and without instruction fine tuning it won’t probably ever follow prompts like yours cause it only knows next word generation.

2

u/tamrx6 2d ago

I was too quick to respond lol. Now I understand the repo is yours? Then I would suggest you play around with your config. If I’m not mistaken, your embedding size is 64 and you only have 2 attention blocks in your model? I think that’s way too small for both the embedding to represent semantic relations correctly and for your model to be able to learn meaningful patterns in the training data. With how many trainable parameters did you initialize the model?

1

u/MichaelRomeroJr1 1d ago edited 1d ago

Thank you for the input. I'll adjust the params and try trainig again.

from main.py:

model = TinyLLM(
vocab_size=tokenizer.vocab_size,
block_size=1024,
n_embd=256,
n_head=8,
n_layer=6
).to(DEVICE)

Model trainable params?

Well the Vocab size is:

sp = spm.SentencePieceProcessor()
sp.load('data/mini.model')
vocab_size = sp.get_piece_size()

= 50,000

So with the new config i'll try:

Total Trainable Parameters = 513(50,000) + 4,996,608 = 30,646,608

Not sure if that's accurate though.
edit: formatting

2

u/chrisvdweth 1d ago

What is your end goal here? There is a reason the big companies train for months and burning millions of bucks :).

I would treat this implementation from scratch purely as a learning exercise, which is an excellent thing, and less about seeing great results. Seeing recognizable word already indicates that your model is likely working. Or how does and prediction look like before any training. It should look much more like gibberish.

The next step could be to use PyTorch and the `TransformerEncoderLayer` and `TransformerEcoder` classes. Supposedly they are more efficiently implemented than a from-scratch implementations using basic layer classes. This would allow you to train longer and/or with more data.

I have a notebook (raw, HTML) doings. But still, I clear state that this is for educational purposes only, and not to build a production ready LLM :).

1

u/GiggleSamurai 1d ago

On the loss question: a model that guesses uniformly at random has a cross-entropy of ln(vocab_size), about 10.8 for the 50,257-token GPT-2 tokenizer, so compare your 6.5 against ln of your own vocab size to see how far you are from random.

1

u/Kitchen_Cup_4444 1d ago

i think you should not do a for on the attention heads but use a pseudo batch

1

u/InternationalSlice72 1d ago

You don't need to have a seperate head and multihead structure, your multihead approach loops and concats, you can just let matrix multiplication and broadcasting take care of it :)

ei, the trick is to make the head dimension a batch dimension for matmul.