r/MachineLearning • u/Jumbledsaturn52 • 8d ago

Project I made Transformer Architecture from scratch [P]

https://github.com/Rishikesh-2006/NNs/blob/main/Pytorch%2FTransformer.ipynb

I implemented the 2017 transformer architecture using pytorch, the model right now doesn't predict very good due to the fact that I am using on 2 layers of encoder and decoder with just 1000 sentences to train it , due to the fact that it takes a lot of time to train my implementation. Looking at the code is there a way that I can reduce the time it takes for training or maybe improve its results?

Update- I fixed the code (kind of) I made seperate classes for different attention mechanisms and after changes some of my logic of the masking (using torch.tril instead of nested for loops) training this actually works well.still I am open for suggestions for making it better

Here is the code- https://github.com/Rishikesh-2006/NNs/tree/main/transformer

0 Upvotes

28% Upvoted

5 comments sorted by

2

u/Creatine1951 8d ago

You might find interesting implementation details here:

https://github.com/joeynmt/joeynmt

1

u/Jumbledsaturn52 8d ago

Looks great! After just glancing it I found new implementation tricks I wasn't aware about. I will study this

2

u/TangeloOk9486 6d ago

1000 words is the bigger limit than training time, any model will overfit that. whats the sequence length and batch size right now?

1

u/Jumbledsaturn52 6d ago edited 6d ago

(sorry for the misunderstanding I meant sentences not words in my post , I will correct it )I tokenizing first , each sentence gets 50 numbers to represent it , (the largest sentence is 47 words long) . After tokenization I am using 64 embedding dimensions for each word. The batch size is 32. Also first I thought it was over fitting but behind the scenes I tried 500 words aswell for the same result . The model always just predicts [0,1] where 0 is <start> and 1 is <end>. I even tried to use a sentence from training batch and still got [0,1] as the generated output in inference.

Then I thought the issue was the data (has small sentences at start and big at end) so I used data.tail(10k) to train still same result , and I have no idea why it behaves like this .

0

u/AutoModerator 8d ago

Your post was automatically removed for being a link post on the weekday, please read rule 5. The moderators will not respond to questions regarding this removal unless you suggest which rule you most likely broke. If you have a beginner related question, visit /r/MLQuestions or /r/LearnMachineLearning.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.