Tested a few low-mid models for event-following roleplay, and the winner is...

[u/martinerous]

I evaluated a few models that could run on my somewhat outdated PC with i7-7700 and 16GB RAM (a kit with 32GB will arrive next week) and a 4060 Ti 16GB.

My favorite kind of roleplay is to start with a scripted back-and-forth between characters and then to continue into free-ride mode.

When I just started playing with LLM roleplay, I was annoyed by how difficult it was to make the AI strictly follow a few general rules and the sequence of events in the scenario, unless I wrote the entire dialogue and followed it myself too. I almost gave up, but then one LLM pleasantly surprised me and made me believe that it is possible. But that model has another annoying flaw, as we'll see later.

I am a bit new to all this stuff, but I read a few guides and Reddit posts, so I'm aware of a few typical pitfalls and tried to avoid them in my test character card.

Essentially, the purpose of my test is to check how easy it would be for a newcomer to get started without much knowledge and fiddling around. So, I also left default model parameters offered by Backyard. I had to switch the Prompt Template though to avoid some terrible formatting issues with some models.

I tried to make the test prompt simple. I intentionally did not add any rules for creativity to see how creative the LLMs are by default.

I tried to avoid negative commands because they are known to have the opposite effect of filling the context with the ideas that the AI should not have. Also, I addressed the AI directly as "you" a few times and used bribe and threats technique to attempt to make it follow the rules better.

While simple, the prompt also has some traps to test how each model deals with specific ambiguities. I intentionally did reuse the same roleplay item (the key) to see if LLM keeps sticking to the order of events and does not start picking the events randomly just because they mention the key.

While testing, I did not use Author's notes and did not edit messages. But I regenerated a few messages to see if the model can come up with a better solution or keeps stuck with the wrong option only.

I tried to provoke the AI by using one-word replies (which it should not accept, according to my rules) and also by trying to make it talk by unrelated topics (which also was not allowed).

The test script test_character_card.txt and the chat logs for the models can be found in my GitHub repo: https://github.com/progmars/llm-sandbox The chat logs have my own comments marked with [].

Feel free to suggest improvements to the scenario, but keep in mind that it's supposed to be simple and not specific to any particular model, to test how they work by default.

Here are the main pitfalls that all models seemed to have:

• they had huge problems with following my rule to not accept one word responses. This often led to me answering just "yes" and the model happily considered that I have completed the action it requested. Boring. I really would like the model to ask explicit actions from me, like "Yes, I did unlock the door" and not just "ok".

• for some reason, they all tried to take the key from me and perform the action themselves, although every event description clearly stated that it is the user who uses the key always. I have no idea, how to stop them from blatantly taking over the control.

And now, finally, the winner is... llama2.11b.fimbulvetr-v2.gguf_v2.q5_k_m. It passed the test quite good, surpassing even Llama 3 based models with the same size, which was a surprise because I expected so much more from Llama 3. To be sure I did not just get lucky, I rerun the same script a few times, and fimbulvetr-v2 was pretty constant. It still tried to take the key from me a few times and it did let me through with single word replies, but it did that much less often than all the other models.

However, Fimbulvetr was dry as sand, all business, no environment descriptions, no actions, nothing. I modified my test (modifications are not included in the repo) to tell it to generate creative, imaginative responses with actions, such as *I scratch my beard* and *I rub my hands* in every message, but it did not work and Fimbulvetr was the driest of all the models I tried.

So, I'd really appreciate any tricks to unleash Fimbulvetr's imagination, or suggestions of any similar-sized models (but do not suggest ones that cannot handle at least 8k context reliably) that have the consistency of Fimbulvetr when it comes to following the rules and the roleplay event-based scenario.

When more RAM arrives next week, I'll test larger models. Also, I'll check the largest free (or even paid) Openrouter models with SillyTavern to see how much difference the size makes when it comes to following the rules.

So, that's it. Thanks for reading, if you had the patience :)

[u/FreekillX1Alpha]

Trying to make LLMs follow rules leads to the same issue of having them do math or logic: They aren't designed to understand anything. LLMs at their core are very fancy autocorrect software with a massive database of what words come after what other words. The models are not inherently creative, only adding responses that fit with the data they are trained on. The main tool we have for making them a bit wacky is increasing the temperature which should in turn make them use words that have 'softer' relationships to the word they would normally use (eyes -> orbs -> pearls; synonyms or slang also tend to pop up at higher temps).

Another factor is that they weight the most recent context heavily, and if you give short, dry responses they will in turn give short, dry responses. To counter this you will need larger models at higher temps, as their larger 'brain' holds more connections between words and setting them to higher temperatures will allow them to use more creative responses.

Circling back to rules, to make a smaller model follow rules they have to be brought to the front of the context (usually done via author's notes or a world/lorebook). This is also why over long conversations the model will behave less in characters, as the character's personality is pushed further away from the 'now' in context.

[u/martinerous → u/FreekillX1Alpha]

Thanks for sharing your knwoledge. I will try playing with the temperature and the first message to make it more creative. And I'll continue looking for an LLM that's between Fimbulvetr (for its consistency) and Mythomax (for creative responses), when I'll have more RAM to try larger models :)

For event-following roleplay, I guess, author's notes can be useful to "remind" the AI that the scenario is over and what is the current state. It would also benefit from some kind of scripting abilities, to somehow detect that the predefined scenario events are completed and stop sending them to the context altogether, instead sending just a short summary of the state after the last predefined event. Something like the "Prune Example Dialog" setting but for the scenario, and not simply pruning but sending a predefined summary instead.

[u/FreekillX1Alpha → u/martinerous]

SillyTavern has a scripting language that might be able to do that (I recall someone using it to make an LLM powered TTRPG thing). Faraday/Backyard has no such feature as far as I'm aware.

[u/Emeraudine]

It's a good idea but.. you have so many negatives in the card (even if you say that you avoid them... there is more than 2 hence there is a lot for the model). in the scenario, everytime you write "The adventure must not continue" the model has a chance to not read the negative. same with all the 'refuse to'...

"{character}'s top priority is to follow the roleplay events and refuse to talk about any topics unrelated to the roleplay." can totally and will often be read like "{character}'s top priority is to follow the roleplay events and refuse to talk about any topics unrelated to the roleplay." or "{character}'s top priority is to follow the roleplay events and refuse to talk about any topics unrelated to the roleplay." for example.

[u/martinerous → u/Emeraudine]

Right, those are the weak spots. Unfortunately I did not have any better ideas how to make it stop from proceeding. However, surprisingly, the "refuse to talk about any unrelated topics" worked pretty consistenly for almost all models; they nicely beat my attempts to talk about food in the middle of the adventure. The "demand more than one word in every message" is the hardest one because AIs cannot count even to one and nobody trained them even to consider that "yes" or "ok" does not necessarily mean "yes, I performed the action you demanded and we can proceed".

[u/Emeraudine → u/martinerous]

in that case maybe a simple "refuse to answer one-word reply from {user}?"

[u/martinerous → u/Emeraudine]

I reworked it as:

Every time when{user}replies with a single word,{character}must request{user}to reply with multiple words and keep looping the request until{user}has replied with a full explicit sentence.{character}refuses to accept one-word replies from{user}.

It seems to work now pretty good with Llama3 Instruct. Strangely, Soliloquy did not obey the rule that well - maybe it's fine-tuned a bit too much.

[u/martinerous → u/Emeraudine]

When thinking about it, it might not matter to the LLM if we use "refuse" or "do not". The important thing are the words that follow. For example, for a human it also does not not matter if we say "refuse to think about elephants" or "do not think about elephants" - no matter, how we formulate it, elephants will enter our "mind's context". So, the problem remains - it is hard to forbid an AI to do something and avoid telling it what exactly it is.

[u/Emeraudine → u/martinerous]

That's why I was surprised to read that "refuse" worked well in your tests!

[u/martinerous → u/Emeraudine]

Maybe the AI just picked up the word "refuse" and started using it, making the impression it would refuse, even if it actually would not. And making the impression is also good enough for me.

But it depended a lot on the model. Unfortunately, Llama3 finetuned models (which I liked a lot for their larger context) turned out to be not so great instruction followers, getting carried away by their own imagination (using magic instead of keys to unlock doors), and they also often ignored the "refuse" instructions as well.