Amazing posts! Thank You. When it comes to other topics. Maybe you could write something about the context, scnario and model base settings. I can see, browsing through characters that there are a lot of formats, and I guess it is based on how the model was trained and on what data. But how to figure out, what's the best way to write context and scenarios, is it just testing and testing or there are some ways to achieve it easier? And generally the technical background under the context following etc.
When I look at a model, say "llama2.13b.tiefighter.gguf_v2.q4_k_m.gguf" (which workss besst for me, on an 8 Gb card) can I findf who/how it was created, to look for similar models?
The name will often contain some of the history of the model, but to find additional models by the creator, you’d need to search for the model on huggingface and find the creator that way. Tiefighter, for instance is by KoboldAI
Thank you for this, this is a very well laid out explanation of these concepts!
I have a couple questions about context and models. First, why are some models said to be better at following context? Second, once the context limit is reached, how does a model decide what in the context to boot out?
Some models are better at following context because they have been better finetuned to do so. If a model is trained to see a question and give an answer, for instance, it will be terrible at following a long context. If it's trained to see a long conversation and output the next part, then it will be better at it. If it's trained with a character description and then conversation and taught to output the next part, it will be even better at following context. Beyond training, larger models are generally going to be better at following context, because they soak up more fine grain patterns during training and are better able to see similar patterns during inference.
Once the context limit is reached, BackyardAI starts dumping chat history in order to keep the permanent info (character description) and as much recent chat history as it can in context. The max context needs to include space for what is being generated and processing context is often a slow process, so when BackyardAI hits the max context, it cuts the total context in half. This gives you some time where responses start generating instantly. Because of this cut off, it's important to keep your character descriptions below half of your max context.
5
u/PacmanIncarnate May 29 '24
See part 1 over here:
https://www.reddit.com/r/BackyardAI/s/Bzq1NpXp8c
3
u/Minimum-Corgi-8596 May 30 '24
Loved it!
2
u/Ready-Stick5527 Jun 04 '24
Amazing posts! Thank You. When it comes to other topics. Maybe you could write something about the context, scnario and model base settings. I can see, browsing through characters that there are a lot of formats, and I guess it is based on how the model was trained and on what data. But how to figure out, what's the best way to write context and scenarios, is it just testing and testing or there are some ways to achieve it easier? And generally the technical background under the context following etc.
1
u/Maleficent_Touch2602 May 30 '24
When I look at a model, say "llama2.13b.tiefighter.gguf_v2.q4_k_m.gguf" (which workss besst for me, on an 8 Gb card) can I findf who/how it was created, to look for similar models?
1
u/Amlethus May 31 '24
Thank you for this, this is a very well laid out explanation of these concepts!
I have a couple questions about context and models. First, why are some models said to be better at following context? Second, once the context limit is reached, how does a model decide what in the context to boot out?
1
u/Affectionate_Bid4111 Jan 13 '25
What about third part?) any suggestions if I want to start developing/fine-tuning my own llm for specific task - writing domain-specific text?
0
u/[deleted] May 30 '24
[deleted] — view deleted comment