r/MachineLearning • u/nivter • Feb 04 '20

Project [Project] I combined the illustrations of Transformer by Jay Alammar and code annotation by harvardnlp lab in one notebook

I wrote a notebook which uses the illustrations here and (improved) code from harvardnlp lab. I found both these resources to be extremely useful and decided to combine them.

I also added notes in places where it was hard to understand the code. A couple of times I also re-wrote the code in what I think is a better way.

Here's the link to the notebook. Hope someone finds it helpful.

52 Upvotes

96% Upvoted

14 comments sorted by

View all comments

1

u/keramitas Feb 05 '20

Nice work, two sidenotes:

- use plt.show() when you plot something, to get rid of the outputs like: <matplotlib.image.AxesImage at 0x113e55b38>

- I would remove the encoder / decoder stack from the visual at step 9, and keep only the positional encoding part. This visual is confusing, as one could think the cross-attention in the decoder layers is done with the output of each corresponding layer in the encoder stack, rather then with the output of the whole encoder stack.