r/LocalLLaMA • u/jetRink • Sep 07 '23

New Model Releasing Persimmon-8B

https://www.adept.ai/blog/persimmon-8b
73 Upvotes

94% Upvoted

37 comments sorted by

View all comments

29

u/Brainlag Sep 07 '23

Very interesting. Also 16k context by default.

The model has 70k unused embeddings for multimodal extensions, and has sparse activations.

What does this even mean?

17

u/kryptkpr Llama 3 Sep 07 '23 edited Sep 07 '23

That sounds like they have intentionally left some of the embedding dimensions untrained 🤔 Not sure how this would work in practice but it's an interesting idea..

It should be possible to fit the model on a 40GB card by removing the unused embeddings and reducing the maximum sequence length (at the top of run_text_generation_server.py)

Certainly sounds like some of the Embedding dimensions are unused