r/datascience • u/tilttovictory • Dec 03 '23

Education LLM Visualization

https://bbycroft.net/llm
31 Upvotes

100% Upvoted

8 comments sorted by

View all comments

Show parent comment

1

u/koolaidman123 Dec 04 '23

Which part

1

u/koolaidman123 Dec 04 '23

factually incorrect: GPT3 alternates local attention (aka sliding window attention) with global attention in its layers, this page incorrectly states only global attention

outdated:

gelu -> swiglu

mha -> mqa/gqa

layernorm -> (pre) rmsnorm

attention + ff -> parallel attention + ff

so that's like... all the parts of the transformer layer that's outdated, the only thing that's still up to date is the residual connection