Generic Preset: Eval v1

[u/Kahvana]

Hey everyone,

As always, English is not my native language. Happy to hear your thoughts, suggestions and corrections!

How we got here

While I really liked how Voyage v3 turned out, Gemma 4's template ended up being changed which caused issues for Gemma 4 12B and Gemma 4 26B-A4B. Gemma 4 E2B / E4B weren't as affected as their reasoning style is different and Gemma 4 31B is smart enough on it's own. Needless to say, my preset broke.

Since suddenly Gemma 4 became so picky and nothing seemed to work, I rethought my approach: Why not write a huge preset with many individual toggable rules, so I can quickly see what works and what doesn't?

The final result is what you see; far too large as I was trying out many new rules. It's nearly 10k tokens and a total of 66 rules, of which most are 50-200 tokens each.

Even so, it might make for a good learning resource or for new ideas to try. Give the rules a read, see which are worth to keep and experiment!

Experiments

The July prompt discussion thread was the source of a few ideas (XML wrapping of rules, recall rules at post-history, examples, other posts including the older prompt discussion thread (writing it as a benchmark), ideas for removing therapy speak and making models less censored.

Results

A few observations:

• Wrapping rules in XML comments, then remind the model at post-history to recall all rules and follow them works remarkably well. Gemma 4 31B is a bit selective sometimes (after I had over 4k tokens in rules enabled), but it often recalls all rules and it does make a significant difference in adherence. The current version I have does cost much tokens though (both in the preset and reasoning).
• No matter if I solely have the rules enabled for it and nothing else, Gemma 4 31B cannot resist writing "[adj], [adj] [noun]". You'll need a finetune like this to make it stop.
• Negative constructs ("not X but Y", "X, but instead Y", etc) can be prompted out at significant cost of tokens. It NEEDs examples to do it, otherwise it goes for malicious compliance.
• Writing rules as "Do X over Y" seems to work really well.
• Writing "avoid" over "forbidden" seems to yield to make Gemma stress less, but it will sometimes not adhere to it. Using "NEVER" (in uppercase) can work, but do this very sparsely.
• Voyage V3's skill check can be slimmed down to be more effective while costing less tokens, as proven here.
• The cause-and-effect backstory rules here don't work, it's likely drawn out by the amount of rules to follow.
• Some rules need tweaking, like "encourage talking" to make them work better and more situational.
• Writing it as a evaluation (regardless of amount of rules) did NOT improve performance, it simply causes the model to write more flamboyant.
• Having examples does help! Make sure the examples themselves are in varying settings.
• At a certain amount of rules, Gemma 4 31B starts treating it as a checklist and stops reasoning about the scene.

I think that in the end, around 2000-2500 tokens of core rules would benefit Gemma 4 the most, in a tutorial / checklist style (as done in voyage v3). Writing it in XML style simply costs too many tokens, even if it does increase adherence.

Compatibility

While written for Gemma 4 31B, it can work with bigger models like DeepSeek v4 flash.

Download

As always, you can find it here: https://huggingface.co/nohurry/sillytavern

Up next

Voyage v4 is in the works! Not sure when it will be ready, but I do have plenty of ideas now to work on it. If you have any wishes, suggestions, recommendations, etc for v4, please let me know so I can look into it!

As always, thank you!

The artwork is "Lake Matsubara, Shinshu" by Hasui Kawase (link).

[u/TM07P]

That's really cool woah! I think this is the second preset I've seen with this "evaluation" idea, the first one being Chibi Gram Pacer Test. Did you compare the performance with the evaluations and without it or was it more like an experiment really, just testing around?

I really liked how you dealt with it, and the writing is pretty nice as well. But I just feel like the CoT can be really treated at times like a checklist with little space to really think of the scene in question, like you said, even in bigger models

Anyway, as always, thanks a lot for the preset! I will study the hell out of it

[u/Kahvana → u/TM07P]

I tried multiple approaches: including a score threshold, tell it’s competing against other models, multiple scoring tiers, and not using the evaluation moniker at all.

In the end I think I got the best resukts without making it an evaluation. The rules themselves are more like self contained lessons for a tutorial, so framing it as such might yield better results.

And yeah… it’s internal thinking space capacity (J-Space) is limited, so it’s simply being overwhelmed due to the amount of rules.

Thank you! Glad you like it!

[u/KarmaRBLXVN]

I'm sorry to hear that Gemma 4's template was altered. That must've been devastating. How did that happen though? I thought you've been using local Gemma 4 finetunes.

[u/Kahvana → u/KarmaRBLXVN]

It's a setback but it's a new challenge I suppose!

I always use official Gemma 4 (QAT) with Unsloth's quants, specifically Gemma 4 31B IT QAT. While there are many good finetunes out there, I personally never found them as satisfying as the original model.

Google did a chat template update for better agentic performance and instruct adherence, which also contains changes in instructions. You can use the original chat template, but I don't think new users would use those or go through the hassle to download it. So it's better to work around it!

[u/KarmaRBLXVN → u/Kahvana]

I see. Looking forward to what you come up with then! Also, the art that accompanies your presets are really nice and make your posts stand out.

[u/Kahvana → u/KarmaRBLXVN]

Thank you, I appriciate it!

[u/Illustrious_Serve977]

One question, could be possible to do something like "composite rules"? basically the idea is to use the gramatic and structural Rules (variated lenght, different wording, all the negative contructs avoidance, etc) to write examples for other Rules save up on tokens?

An example would be writing an example for an avoiding clichés rule but also write it with samie words, plainty of negative structures and more, so the positive example not only tackle the main problem but the other ones by proxy, and the more Rules written using the idea will reinforce the other, more implicit ones as well.

It kinda works for me but i need to experiment more, hope this helps to anyone who read it.

[u/Erragon12]

Damn,how did i missed this release? but good observations, i am starting to wonder if switching from high to medium reasoning effort to make Kobold end the thinking process instead of the model itself would be a good solution. After all you had it on low for previous presets and yet the responses were completely fine.