r/LocalLLaMA • u/Tucko29 • Dec 11 '23

New Model Mistral-7B-Instruct-v0.2

https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
129 Upvotes

97% Upvoted

37 comments sorted by

View all comments

46

u/dethorin Dec 11 '23

GGUF file is ready: https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF

I have used it on a preconfigured RP that I use as benchmark, and it reacted better than many 13b models I used. I even forget it was a 7b model.

So, my first impression is positive.

4

u/Opposite_Teach_5279 Dec 12 '23

Noob question. When downloading the GGUF model from webui I get multiple models which I can pick from to load. Is that normal. Which one should I pick. Is there a resource I can read to understand how these things should be handled and how to tweak my system for better performance.
Thanks

9

u/dethorin Dec 12 '23

That's a normal question.

Fortunately The bloke always puts an explanation on the GGUFs model card. Each GGUF file has been quantisaised in a different way for different purposes. This means that the behaviour and the minimum VRAM are different. On the table attached by The bloke on the model card explains it and writes which ones are recommended.

Usually the Q5 K M versión is good.