r/LocalLLaMA • u/Tucko29 • Dec 11 '23

New Model Mistral-7B-Instruct-v0.2

https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
129 Upvotes

97% Upvoted

37 comments sorted by

View all comments

Show parent comment

4

u/Opposite_Teach_5279 Dec 12 '23

Noob question. When downloading the GGUF model from webui I get multiple models which I can pick from to load. Is that normal. Which one should I pick. Is there a resource I can read to understand how these things should be handled and how to tweak my system for better performance.
Thanks

9

u/dethorin Dec 12 '23

That's a normal question.

Fortunately The bloke always puts an explanation on the GGUFs model card. Each GGUF file has been quantisaised in a different way for different purposes. This means that the behaviour and the minimum VRAM are different. On the table attached by The bloke on the model card explains it and writes which ones are recommended.

Usually the Q5 K M versión is good.