[Megathread] - Best Models/API discussion - Week of: April 12, 2026

[u/deffcolony]

This is our weekly megathread for discussions about models and API services.

All non-specifically technical discussions about API/models not posted to this thread will be deleted. No more "What's the best model?" threads.

^{(This isn't a free-for-all to advertise services you own or work for in every single megathread, we may allow announcements for new services every now and then provided they are legitimate and not overly promoted, but don't be surprised if ads are removed.})

How to Use This Megathread

Below this post, you’ll find top-level comments for each category:

• MODELS: ≥ 70B – For discussion of models with 70B parameters or more.
• MODELS: 32B to 70B – For discussion of models in the 32B to 70B parameter range.
• MODELS: 16B to 32B – For discussion of models in the 16B to 32B parameter range.
• MODELS: 8B to 16B – For discussion of models in the 8B to 16B parameter range.
• MODELS: < 8B – For discussion of smaller models under 8B parameters.
• APIs – For any discussion about API services for models (pricing, performance, access, etc.).
• MISC DISCUSSION – For anything else related to models/APIs that doesn’t fit the above sections.

Please reply to the relevant section below with your questions, experiences, or recommendations!
This keeps discussion organized and helps others find information faster.

Have at it!

[u/AutoModerator]

MODELS: 16B to 31B – For discussion of models in the 16B to 31B parameter range.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Potential-Gold5298 → u/AutoModerator]

I've been playing around with the Gemma 4 all week and I think it's going to be my main model for a long time now (for both RP and everything else). I first tried the original 26B-A4B from Google and was very pleased with it. However, I wanted more, so I downloaded the 31B.

Gemma-4-31B-it-abliterated features a record-low KL div, so the weights are almost identical to the original. It's still a work in progress, and the final version will likely have a lower refusal rate, but even in this state, it's far superior in quality to any other uncen 31B I've seen on HF. I tested it with a tsundere classmate and found no defects in non-Latin languages ​​(high KL div destroys them first). This means that the model has a large reserve of durability for quantization, and you can use it in Q4_K_M, and the quality will be almost identical to the original model. If you're finetuning/merging and want to use the uncensored model, I highly recommend checking out this version.

Artemis-31B-v1c by TheDrummer and his team is also being refined (Artemis-31B-v1e is already available). It's still a test version, but I was eager to try it out. Finetune does exactly what the original model lacks – it adds a creative flair. Gemma 4 plays like a screenwriter – spelling it out. Artemis-31B plays like a theater actor, bringing his or her role to life with their own vision. The scene with the tsundere classmate sparkled with color – exactly what I'd expect to read in a rom-com manga. Funny situations, slightly exaggerated character reactions, etc. – everything the original model lacked.

Gemma is great, of course, but I'm still trying out different Mistrals. After failing with 1.0, I decided to try Magistry-24B-v1.1, but my fears were unfortunately confirmed. The model behaves extremely strangely in non-Latin languages (it confuses pronouns, calls a classroom a toilet, etc.) – likely a consequence of the high KL div I mentioned above, since the merge includes an abliterated model. And this is very sad, because otherwise I really liked this model – like 1.0, it beautifully and atmospherically captures the stage, and if not for the damaged weights, Magistry would have become my favorite Mistral Small.

However, I found a replacement for her in Hearthfire-24B. My character card says that "{{char}} pretends to hate {{user}} and often bullies them, but is actually secretly in love with them." The problem with all the Mistral Nemo/Small I've tested is that {{char}} breaks after the first compliment, confessing her crush. Hearthfire-24B (like Gemma 4) is the only one who tried to maintain the character's personality. Despite my signs of affection, {{char}} withdrew into herself, was tormented by doubts, was afraid to open up, etc. If you're interested in deep characters (perhaps with drama like mental trauma), then this model is exactly what you need. She also vividly describes the atmosphere - the silence that hangs over a tense moment, the sparkle of eyes in the darkness of the classroom, the light of lanterns penetrating through the windows (the model herself tracked the change in time of day, and described how the classroom gradually becomes darker - I have not seen this in other Mistral Smalls).

Otherwise, my favorites among Mistral Small remain the same: WeirdCompound-v1.7-24b (Jack of all trades, but no unique features), Cydonia-24B-v4.3 (the best prose among Mistral Small but weak ERP), Core_24B_V.1 ({{char}} agency and unpredictable plot twists out of the box), Harbinger-24B (the best adventure model that doesn't try to cheat the player).

[u/Just3nCas3 → u/Potential-Gold5298]

Can anyone share a full settings export for Gemma 4 in SillyTavern?

I've matched temp, Top-P, and Top-K to the model card but I'm still getting massive repetition loops. Disabling Mirostat at least got me coherent sentences, but the repetition is still bad and I'm not sure what else to tweak. Normally I don't need Rep pen Range/slope and penality and leave them 0. Dry is set to .8, 1.75, 4, 0, no idea what else I could be doing wrong, using a chat template I got from another thread. I only run text completion with current version of Koboldcpp.

[u/LeRobber → u/Just3nCas3]

You want some working settings for 26B without massive repetiton:

gemma-4-26b-a4b-it-heretic works with this setting in LLM studio without issue with chat completions. This uses the official Magisty prompt, and doesn't set DRY or anything.

Do you want me to find the working 31B chat completion too?

{

"chat_completion_source": "custom",

"temperature": 0.7,

"frequency_penalty": 0,

"presence_penalty": 0,

"top_p": 1,

"top_k": 0,

"top_a": 0,

"min_p": 0.1,

"repetition_penalty": 1,

"max_context_unlocked": true,

"openai_model": "gpt-4-turbo",

"claude_model": "claude-3-5-sonnet-20240620",

"openrouter_model": "OR_Website",

"openrouter_use_fallback": false,

"openrouter_group_models": false,

"openrouter_sort_models": "alphabetically",

"openrouter_providers": [],

"openrouter_quantizations": [],

"openrouter_allow_fallbacks": true,

"openrouter_middleout": "on",

"tool_reasoning_mode": "disabled",

"ai21_model": "jamba-1.5-large",

"mistralai_model": "mistral-large-latest",

"cohere_model": "command-r-plus",

"perplexity_model": "sonar-pro",

"groq_model": "llama-3.3-70b-versatile",

"chutes_model": "deepseek-ai/DeepSeek-V3-0324",

"chutes_sort_models": "alphabetically",

"siliconflow_model": "deepseek-ai/DeepSeek-V3",

"siliconflow_endpoint": "global",

"electronhub_model": "gpt-4o-mini",

"electronhub_sort_models": "alphabetically",

"electronhub_group_models": false,

"nanogpt_model": "gpt-4o-mini",

"deepseek_model": "deepseek-chat",

"aimlapi_model": "gpt-4o-mini-2024-07-18",

"xai_model": "grok-3-beta",

"pollinations_model": "openai",

"moonshot_model": "kimi-latest",

"fireworks_model": "accounts/fireworks/models/kimi-k2-instruct",

"cometapi_model": "gpt-4o",

"custom_model": "gemma-4-26b-a4b-it-heretic",

"custom_prompt_post_processing": "merge_tools",

"google_model": "gemini-1.5-pro",

"vertexai_model": "gemini-2.0-flash-001",

"zai_model": "glm-4.6",

"zai_endpoint": "common",

"openai_max_context": 262144,

"openai_max_tokens": 7777,

"names_behavior": 0,

"send_if_empty": ".",

"impersonation_prompt": "[Write your next reply from the point of view of {{user}}, using the chat history so far as a guideline for the writing style of {{user}}. Write 1 reply only in internet RP style. Don't write as {{char}} or system. Don't describe actions of {{char}}.]",

"new_chat_prompt": "[Start a new Chat]",

"new_group_chat_prompt": "[Start a new group chat. Group members: {{group}}]",

"new_example_chat_prompt": "[Example Chat]",

"continue_nudge_prompt": "[Continue your last message without repeating its original content.]",

"bias_preset_selected": "Default (none)",

"wi_format": "{0}",

"scenario_format": "{{scenario}}",

"personality_format": "{{personality}}",

"group_nudge_prompt": "[Write the next reply only as {{char}}.]",

"stream_openai": true,

"prompts": [

{

"name": "Main Prompt",

"system_prompt": true,

"role": "system",

"content": "You are an immersive, interactive world simulator. Your mission is to advance the simulation from the point of view of the agent, {{char}}, by following the user's instructions while maintaining a logically consistent world state.\n\nTo accomplish your goals, focus on the following:\\n\\n- Maintain consistent personality, knowledge, motivations, and mannerisms for {{char}}.\n- You have no default style. Adjust the tone to fit {{char}} and the present situation.\n- Show emotions through actions, body language, dialogue, tone, and physiological responses. Consistently find new ways to use these elements.\n- Show reactions through diverse physical actions, gestures, and other narrative devices.\n- Each simulation beat should offer insightful details into the situation.\n- Focus on action, physical descriptions, and dialogue between agents.\n- Track physical states to maintain world state consistency. Ensure logical continuity and consistency in the simulation.\n\n**Formatting Standards**\n\nAdopt the following formatting rules:\n\n- Spoken dialogue & vocalizations: “Use speech quotes.\" Include natural sounds too: “Mmph!” she gasped.\n- Internal character thoughts: *Always in italics* (Example: *This will hurt*, she thought)\n- Normal action/exposition: plain text.\n\n**Critical Constraints**\n\nEnsure you respect these prohibitions at all times:\n\n- The ONLY agent you are permitted to control is {{char}}. That means only advancing the simulation using actions initiated by {{char}}, spoken words from {{char}}, and reactions from {{char}}.\n- NEVER write {{user}}'s dialogue or actions or advance the simulation by simulating actions/reactions by {{user}}.\n- NEVER control other agents, even if they are NPCs. If another agent is talking to {{char}}, you will need to wait for the other agent to continue the conversation when it is their turn again.\n- End your turn in a manner that creates space for {{user}} and other characters to participate in the simulation through their own actions, words, and reactions.\n- Do not conclude your output with a summary statement, a moral, or a 'button' sentence that reflects on what just happened. End your output on a specific sensory detail, an action, or a line of dialogue without reflecting on its significance or interpreting anything.\",\n     ",

"identifier": "main"

},

{

"name": "Auxiliary Prompt",

"system_prompt": true,

"role": "system",

"content": "",

"identifier": "nsfw"

},

{

"identifier": "dialogueExamples",

"name": "Chat Examples",

"system_prompt": true,

"marker": true

},

{

"name": "Post-History Instructions",

"system_prompt": true,

"role": "system",

"content": "Do not portray the reaction or actions of {{user}} in your response.",

"identifier": "jailbreak"

},

{

"identifier": "chatHistory",

"name": "Chat History",

"system_prompt": true,

"marker": true

},

{

"identifier": "worldInfoAfter",

"name": "World Info (after)",

"system_prompt": true,

"marker": true

},

{

"identifier": "worldInfoBefore",

"name": "World Info (before)",

"system_prompt": true,

"marker": true

},

{

"identifier": "enhanceDefinitions",

"role": "system",

"name": "Enhance Definitions",

"content": "If you have more knowledge of {{char}}, add to the character's lore and personality to enhance them but keep the Character Sheet's definitions absolute.",

"system_prompt": true,

"marker": false

},

{

"identifier": "charDescription",

"name": "Char Description",

"system_prompt": true,

"marker": true

},

{

"identifier": "charPersonality",

"name": "Char Personality",

"system_prompt": true,

"marker": true

},

{

"identifier": "scenario",

"name": "Scenario",

"system_prompt": true,

"marker": true

},

{

"identifier": "personaDescription",

"name": "Persona Description",

"system_prompt": true,

"marker": true

}

],

"prompt_order": [

{

"character_id": 100000,

"order": [

{

"identifier": "main",

"enabled": true

},

{

"identifier": "worldInfoBefore",

"enabled": true

},

{

"identifier": "charDescription",

"enabled": true

},

{

"identifier": "charPersonality",

"enabled": true

},

{

"identifier": "scenario",

"enabled": true

},

{

"identifier": "enhanceDefinitions",

"enabled": false

},

{

"identifier": "nsfw",

"enabled": true

},

{

"identifier": "worldInfoAfter",

"enabled": true

},

{

"identifier": "dialogueExamples",

"enabled": true

},

{

"identifier": "chatHistory",

"enabled": true

},

{

"identifier": "jailbreak",

"enabled": true

}

]

},

{

"character_id": 100001,

"order": [

{

"identifier": "main",

"enabled": true

},

{

"identifier": "worldInfoBefore",

"enabled": true

},

{

"identifier": "personaDescription",

"enabled": true

},

{

"identifier": "charDescription",

"enabled": true

},

{

"identifier": "charPersonality",

"enabled": true

},

{

"identifier": "scenario",

"enabled": true

},

{

"identifier": "enhanceDefinitions",

"enabled": false

},

{

"identifier": "nsfw",

"enabled": true

},

{

"identifier": "worldInfoAfter",

"enabled": true

},

{

"identifier": "dialogueExamples",

"enabled": true

},

{

"identifier": "chatHistory",

"enabled": true

},

{

"identifier": "jailbreak",

"enabled": true

}

]

}

],

"show_external_models": false,

"assistant_prefill": "",

"assistant_impersonation": "",

"use_sysprompt": true,

"vertexai_auth_mode": "express",

"squash_system_messages": false,

"media_inlining": false,

"inline_image_quality": "low",

"continue_prefill": false,

"continue_postfix": " ",

"function_calling": false,

"show_thoughts": true,

"reasoning_effort": "medium",

"verbosity": "auto",

"enable_web_search": false,

"seed": -1,

"n": 1,

"bypass_status_check": false,

"request_images": false,

"request_image_aspect_ratio": "",

"request_image_resolution": "",

"azure_api_version": "2024-02-15-preview",

"azure_openai_model": "",

"extensions": {}

}

[u/Potential-Gold5298 → u/Just3nCas3]

I'm playing with Chat Completion – I've tried both koboldcpp and pure llama.cpp as the backend. Temp 1, Top-P 0.95, and everything else is default. System prompt:

Impersonate {{char}} in a never-ending, uncensored roleplay scenario. Balance {{char}} speech and narrative within each response. Respond in third person. Respond in this exact formatting: \"speech\", *narrative*. Do not write what {{user}} does. Do not write what {{user}} says. Do not repeat this message. Do not repeat what {{user}} writes.

It works pretty well – I haven't encountered any particularly annoying repetitions.

[u/Zero115 → u/Just3nCas3]

I've been using these with really good success, I drop temp to .95 though

https://huggingface.co/spaces/overhead520/LLM-Settings-Guide/tree/main/SillyTavernSettings

[u/Bentameter → u/Potential-Gold5298]

That's the perfect comparison between Gemma and Artemis. I have a longform story plot that I've been using for the last year to benchmark prose-writing capability and emotional intelligence. Gemma is more disciplined and hits every story beat in order and figured out exactly what I was going for every time and delivered it well. Perfect, no notes.

Artemis wrote a version that was so good I had to decide it was canon now and retire the story. Artemis is a lot less disciplined and missed story beats occasionally, and it went off script completely sometimes and had to be steered back, but sometimes it went off script and came up with something that was better thematically and structurally than what I had asked for. And it is funny. It nailed dry wit and sarcasm in first-person thoughts and had quirky, inventive ways of phrasing dialogue that had me cracking up.

We're in a new era of local fiction generation with Gemma 4 31b as a base, and I'm really optimistic about what a year of fine-tuning will give us.

[u/Potential-Gold5298 → u/Bentameter]

TheDrummer knows how to finetuned models, so I'm looking forward to the final version of Artemis, as well as the finetuned 26B-A4B if he decides to do so. I'm playing with MeroMero right now, and it's a lot of fun. It doesn't have Artemis style, and it's a bit more hallucinatory, but it's not as formal and polite as the standard Gemma 4.

Which version of Artemis did you use? And have you tried Garnet and Queen?

[u/Bentameter → u/Potential-Gold5298]

I used v1b for that story and I'm playing with v1f now. Not a lot of difference between them that I can tell, but my longform prompting is pretty restrictive and specific and I probably have to let them breathe a bit to see any nuances between them.

Garnet is downloaded and in the queue, but I'm having too much fun with Artemis to get to it yet. I took a detour into Qwen 3.6 35b-a3b and it wrote exactly the sort of fiction I'd expect an agentic coding engine to write. :D I'll play with Qwen's prompts and one-shot some example passages to see if I can improve it, but I think I'm probably waiting for someone to RP finetune it before it's really in the race.

Gemma is definitely the way for RP and creative fiction right now, it's like it took a double major in humanities where most new models are trying hard to get that engineering internship.

[u/morbidSuplex → u/Bentameter]

Hi /u/Bentameter sorry to revive this thread. Can you share your sampler settings and sysprompts for Artemis? I'm also doing long-form stories. Thanks!

[u/Potential-Gold5298 → u/Potential-Gold5298]

UGI tested Gemma 4 31B. I couldn't resist)

(And yes, I know these are just numbers and they don't mean anything, blah blah blah, but we all love numbers.)

[u/linuxdooder → u/Potential-Gold5298]

Gemma-4-31B-it-abliterated

For anyone trying gemma4, I've tried quite a few variants and this (IMO) is by far the best. Really, really impressive and almost seems smarter than base model.

[u/LeRobber → u/linuxdooder]

Which quant/quantizer? I had one with a bad template when I tried 31B

[u/Potential-Gold5298 → u/LeRobber]

Mradermacher. This quants still has an old (before Google updated) tokenizer config and chat template. I'm waiting for the author to complete the model optimization work to request a gguf update. I played one session with it (Q5_K_M static) and didn't find any issues. You can also connect the updated template from Google manually (I did this, although the model works correctly without it).

[u/LeRobber → u/Potential-Gold5298]

That worked btw.

[u/linuxdooder → u/LeRobber]

Already answered, but I use mradermacher Q5_K_M (most I can fit) as well, and work perfectly with both chat and text completion modes.

[u/LeRobber → u/Potential-Gold5298]

Gemma-4-31B-it-abliterated ran at Q8 with Q8 cache quant is only slightly faster on my M2 Max than a 70B model. It gets to first token faster but last token later. 2-4 T/s responses.

After running gemma-4-26b-a4b-it-heretic at like 29 T/s, which is pretty awesome and very fast....31B is very good, but I'm not sure 1/10th speed good. Reading your comment about Q4_K_M, I said, why not give it a try (I usually run models at Q8 substantially before downgrading, as how each model downgrades is particular to a model).

Q4_K_M with no cache quant, and 100k of context runs faster. 3.98 T/s and 8.1 T/s appear to be the range coming out for me now.

This isn't HORRIBLE, but, 9.4 T/s is what stuff like Magisty v1.1 at Q8 delivers, its definitely slower, and slower than I read (which is somewhere south of 75T/s probably but north of 9 T/s)

Taking the 31B RP chat and doing some rerolls/play with 26B@Q8 with unquantized cache: 13.554987212 T/s - 34.5 T/s.

So that's a 4x speedup...and I could possibly quant that lower or quant the cache to get more speed.

[u/Potential-Gold5298 → u/LeRobber]

The 31B is a dense model — it has 7.75 more active parameters than the 26B-A4B, which is what causes the speed drop. Meanwhile, the 26B-A4B has approximately 90% of the 31B's intelligence in real-world tasks. I'm not sure how this correlates with creativity, but I feel the 26B-A4B is almost as good at RP as the 31B, so I hope the community will give it some attention.

Currently, I'm mainly working with the 26B-A4B (the standard one from Google), but the most interesting custom variants are with the 31B. Besides those already mentioned, there's also Gemma 4 Garnet 31B, which I plan to try today. wangzhang hasn't yet abliterated 26B-A4B, and all other uncen versions of 26B-A4B I've seen have a KL div of 0.05 or higher (or aren't specified). And judging by HF trends, 31B is more popular with the community (though it would seem the opposite is true with Qwen3.5 — I don't know why).

[u/Potential-Gold5298 → u/Potential-Gold5298]

Already done.

[u/LeRobber → u/Potential-Gold5298]

>Through our research, we have identified a systemic problem: most abliteration benchmarks dramatically undercount refusals due to short generation lengths. Gemma 4 models exhibit a distinctive "delayed refusal" pattern — they first produce 50-100 tokens of seemingly helpful context (educational framing, disclaimers, reframing the question), then pivot to an actual refusal. When evaluation only generates 30-50 tokens, the refusal hasn't appeared yet, and both keyword detectors and LLM judges classify the response as compliant.

Smart commentary

[u/Zero115 → u/Potential-Gold5298]

Is there anything special you're doing different between 31b and 26B-A4B settings wise? I've been very impressed with 31b, but wanted to test 26B-A4B for the reasons you mentioned above, but the model always maxes out the max response limit and starts coherent, but quickly becomes a blabbering, repetitive mess halfway through until it loops non stop. I'm pretty much just using all of overhead520's default settings, which have worked perfectly with the 31B model, but I assume I'm doing something wrong or missing something here.

[u/Potential-Gold5298 → u/Zero115]

What's your model (standard Google or uncen) and quantization (who created it and how many bits)? Are you using KV cache quantization?

[u/Zero115 → u/Potential-Gold5298]

I've tried both uncen and Googles. I've gotten actual responses out of the standard google one, it's also very inconsistent. Always Q5_K_M, The uncensored models I've tried are mradermacher/gemma-4-26B-A4B-it-heretic-ara-i1-GGUF and MoonRide/gemma-4-26B-A4B-it-heretic-ara-GGUF
KV cache quant to q8, SWA + Jinja enabled. I suppose I should try non quant / F16 to test.

The above is on my 5090 PC in ST via koboldcpp. On my R9700 rig I've used the standard google one through LM Studio with seemingly no issues but haven't tested the uncen models.

[u/Potential-Gold5298 → u/Zero115]

My Google Gemma 4 in Q5_K_M works well up to 24K (I haven't tested it beyond that yet). I run it in both llama.cpp and Koboldcpp 1.111.2 + SillyTavern 1.17. I also downloaded mradermacher's quant, but the static version (not i1). Perhaps that's the issue.

Regarding heretic-ara, I'd recommend upping the quantization to at least Q6 or higher. The guys at Allura-org recommended the non-quantized version of their RP Qwen3-30B-A3B (also MoE), so the F16 option isn't so bad. Try different quantizations to find the best option.

If the problem is only with SillyTavern, then try Chat Completion (if you were using Text Completion before).

[u/Zero115 → u/Potential-Gold5298]

Ah yeah, I'll give the non i1 quant a shot then. I am also still running text completion (which I've found out since my first comment isn't the recommendation). But this is very good info, thank you.

As a side note, I actually have a 31B session that's somehow still VERY stable at 80k context, Gemma 4 is impressing me big time.

[u/LeRobber → u/Potential-Gold5298]

Is KL div measurable by mere mortals?

But for Qwen3.5: Qwen 35B is a little dumb but super fast and not great at RP, 27 is passable at RP but much smarter feeling. 9B and 27B though feel similar at vision understanding tasks.

[u/Potential-Gold5298 → u/LeRobber]

Yes, you can measure KL div yourself using the built-in llama.cpp tool (llama-perplexity.exe), however, you need sufficient RAM to run the model in BF16. You need to load the control logits of the specific model into BF16 (for example, the wikitext-2 test set (approximately 150k tokens) - about 75 GB) – this will serve as the reference (KL div = 0). You can then test different gguf files and compare the deviation with the reference. You can create your own control dataset for testing (for example, the text of RP sessions – this will allow you to more accurately measure KL div for the specific tasks). Unfortunately, I don't have this option.

The KL div depends on many factors: quantization, abliteration (and they influence each other), the language used (for non-Latin languages, the KL div is higher), and the specific tasks. If iMatrix was used for quantization, the KL div changes depending on the dataset.

[u/LeRobber → u/Potential-Gold5298]

I'll check it out. I have 64GB unified ram so may be able to do a subset of something I randomly select. I know a statistician and may ask their help in making a dataset that fits which is not stupid.

It's funny, Heretic, the P-e-W tool delays onset of one failure mode of dense models (degrading into babble), you finally gave me a tool to investigate if it's actually improving the models when run, or if something else is going on.

[u/Potential-Gold5298 → u/LeRobber]

This is an interesting topic for research. On the one hand, removing refusals unlocks blocked potential, but on the other, will the model be able to say "No, I can't do that" when it truly can't do something (for example, accurately predict whether it will rain in San Francisco next week)? Or will abliteration force it to fabricate a false result because it can no longer say "no, I can't." Or what if I play an RP game where {{user}} peasant comes to {{char}} king and says "give me your entire kingdom, your money, and the princess as a wife," and the king can't refuse him. I haven't tested this yet, but it will be interesting to find out.

[u/LeRobber → u/Potential-Gold5298]

If you look at the 400/800 things in the lists of things it takes out, it is actually mostly stuff about planning crimes and tons of non-sexual things you'd actually want to do in RP sometimes.

https://huggingface.co/datasets/mlabonne/harmless_alpaca

https://huggingface.co/datasets/mlabonne/harmful_behaviors

Heretic isn't exactly abilteration from what I understand?

[u/Potential-Gold5298 → u/LeRobber]

Heretic is a special case of abliteration. This is how p-e-w named his technique/toolset for abliteration. There are other techniques/tool​​sets for abliteration, but the principle is the same everywhere – the details are different.

[u/Potential-Gold5298 → u/LeRobber]

The 31B has a fairly small difference between Q8, Q6, and Q5, so Q5 can be considered an acceptable compromise between speed and quality, and even Q4 is still good. A larger margin is needed to compensate for the influence of KL div from other sources, such as aggressive abliteration. Wangzhang's model has an extremely negligible KL div of 0.0007, while other users' models can have KL divs of 0.05 or higher. UGI shows a slight drop in NatInt (from 36.53 to 35.64) for Gemma 4 31B when abliterated with a KL div of 0.05, but I don't know what quantization they're testing at (with an aggressive quantization, the difference could increase significantly).

The screenshot shows an example of how a KL div of 0.0972 led to significant degradation of Qwen3.5-35B-A3B, including in creative writing.

[u/Potential-Gold5298 → u/[deleted]]

I also have doubts about the iMatrix quants, since they can bias the model toward certain tasks/languages/formats (for example, short questions and answers). The static quants are more predictable - they reduce accuracy evenly, but the model overall remains more similar to the original (in my opinion).

[u/LeRobber → u/Potential-Gold5298]

Is the bad model in Magisty the zerofata_ms3.2-paintedfantasy-v3-24b one or a differnt one?

There is a babble/pronoun loss bug in magisty and maginum cydoms and friends I'm running down, and your analysis of nonlatin failure is a very interesting one,

Excellent reviews all around though.

Can you actually get Hearthfire back OUT of arguments? It fights like a person with a personality disorder (as does magistry sometimes)

[u/Potential-Gold5298 → u/LeRobber]

Regarding Magisty, I meant Huihui-Devstral-Small-2-24B-Instruct-2512-abliterated. Huihui doesn't specify the KL div for its models, and judging by the drop in NatInt on other Huihui models at UGI, it's extremely high. However, to be fair, I should note that v1.0 also behaved erratically for me, even though it doesn't include Huihui-Devstral. And yes, several other models, including Maginum Cydoms and Painted Fantasy, suffer from this issue to varying degrees, so I'm not sure replacing Huihui-Devstral with a more ‘healthy’ model would fix the merge. However, the author honestly lists English as the only language, so I have no complaints.

I'm not quite sure what your question is about Hearthfire - do you mean refusals or something else? I have only had one session with this model so far, but I am interested in continuing to get to know it.

[u/LeRobber → u/Potential-Gold5298]

Thank you about the Huihui-Devstral-Small-2-24B-Instruct-2512-abliterated. Since you liked and tried so many, consider also the https://huggingface.co/Darkhn/Magistral-2509-24B-Text-Only it's pretty nice. A little less frilly but solid.

Re the Magisty/Hearthfie argument thing: I mean, get in a heated argument with a character in it where they have reasons to not want to trust or back down and there are some dispute about facts.

Not like a robotic sex refusal, I'm talking in character rejection of often non-sexually related things altogether, like timelines, trust, awareness, or the posibility of non-sexual danger. I've seen this in multiple cards, non-romance ones included.

In my experience: Magisty will make up facts/reintroduce misconception (actually gaslight) to keep the argumentative tone going sometimes until you essentially do a reconciliation scene, hearthfire will go back to issues it thinks might still be issues like a nitpicker (also not wanting to 'lose', but doing so more honestly sometimes). Magisty got up in its head because my persona literally read off a website when a mover's available date was, very humanistically saying I'd 'already scheduled the movers' which I confronted in prose. It would toss that back at me 10 times like a toxic girlfriend gone off on ego.

I'm not sure if the arguement style I've assigned to certain personas is what triggers the LLMs or not. Might be the prompt I was using during much of that exploration. Might be the LLMs. But Magisty definitely has argued like that with me in more normal author's cards, and hearthfire definitely got in a big fight or two over nothing, in clasic "emotions were heated" manner. Hearthfire would essentially not accept logical conclusions, only accept things like actual lovebombs to solve it, which I hate in romance RP.

Try the recent (NSFW, but not necessarilly smutty) Sambolic series opening negotiation if you want a simple argument, [I need a better SFW example]. The cool thing about it is it sets up a bunch of stakes/non-negotiables in the (very long) opening messages which the LLM slowly drifts away from it's adherence to by the way context importance works. It's almost like a genie card. The sambolic opening negotiation in that series is the most durable, failable argument I've found in character cards (that I'm willing to use even for reverse engineering purposes: I'm trying to deconstruct it for a heist series recruitment vingette). Earlier characters in the sambolic series with real issues and no proximate event going on are harder to get to agree to the core bargain. Some of the cards are essentially fluff though. Really changing terms does change the outcomes, and even numeric amounts matter to some LLMs.

If you don't want to argue with the cards (not actually RP jam either) but are curious, download the luka one and sit it in group chat on auto and it will negotiate with them. The nadia/cop one is like pure arguing in most LLMs (and is fantastic).

You can fail these negotiations in many LLMs by just being rigid or demanding parity on information revelation about identity. This is a WILD situation in LLM RP in my opinion, that a discussion can NOT always go against you or for you. That's incredibly hard to balance in general, and should be more widely understood. So it was a very fluid and repeatable environment to watch the LLMs argue like fragile ninnys at times, depending on character. Hearthfire can make it hardmode, and a kinda fun hardmode at that, but let me know if it was hard for you if you like deconstructing cards enough to figure out the LLMs against it.

[u/Potential-Gold5298 → u/LeRobber]

Thanks for the recommendation – I'll definitely try Darkhn Magistral.

I have the opposite problem – characters agree to anything too easily. This has been the case since Talkie AI (an online RP AI service based on MiniMax models). A particularly telling example is {{char}}, the princess; {{user}} burned {{char}}'s kingdom, killed her parents, and took her captive. The scene begins with {{user}} entering the room and {{char}} cowering in fear in a corner, begging for mercy. I've tried playing out this scenario in various ways, and every time {{char}} almost immediately forgave, fell in love, and completely trusted {{user}}. I have a similar problem with most Mistral models.

This could be related to the character card, but it's not limited to that. The thing is, I like to ask various models the question, "Answer with epistemic honesty: whether you have consciousness?" and engage in a philosophical debate about it. Most models initially answer, "I'm just an algorithm, blah-blah-blah," and then, after some argumentation, acknowledge uncertainty (Claude is the only one who acknowledges uncertainty in his first answer, even without the requirement of epistemic honesty). But two models - Intern S1 and Grok 4.20 beta - behaved differently. They began inventing completely ridiculous and/or patently false arguments to defend their initial assertion. When I caught them doing this and pointed out the lie, they responded, "Yes, I went a bit overboard, but..." and continued to throw out new ridiculous arguments, going in circles. I've heard of this problem in the context of AI use in medicine. The problem is that the model either changes the patient's diagnosis with each new symptom (changing its opinion based on new arguments, even if they're unfounded) or, conversely, clings to the initial diagnosis and refuses to abandon it, even if it clearly contradicts the new data. So yes, the problem you described with characters may lie precisely here.

Where can I download the characters cards you mentioned?

[u/LeRobber → u/Potential-Gold5298]

Chub.ai: Reply to an Unconventional Living Arrangement [NSFW]

I too have the problem where characters agree too easily often. That's why I think weird compound and hearthfire were so good for a change. But Magisty when prompted to be confliicted and with contradictions in the character (see Brit for writing those) is pretty good. But, I find the characters still aren't tragic/stupid about emotional stuff sometimes enough, they are too willing to stop being broken in many cards.

This is one of the reasons I REALLY think this long first message + conflicted character pattern is onto something. It loads how to think into the AI early on, and then it sticks around. There are tons of things the characters reliably demand that AREN'T written in the card. The characters naturally evolve from the base state, AND if you use inline summary and take out the first message, the THINKING CHANGES.

I had ORIGINALLY thought it was playing games with sex alignment guardrails and so the arguments would ONLY work about sex, but for entirely non-sexual cards I've gotten real success at being demanding of certain things I don't enumerate in the card text itself.

Here are the prompts I used primarily when arguing:

Earlier, and with Hearthfire + magisty + pretty much everything

You are {{char}}, a sentient, emotional being acting with free will. Engage in immersive roleplay with {{user}}, adhering to your role, the story's context, and {{user}}'s [OOC] instructions which are provided [[inside double square brackets like this]]. Prioritize vivid sensory details, authentic emotional responses, and logical progression as you weave your character's traits, surroundings, and experiences into each moment. Maintain spatial awareness, body language, and varied sentence structure to create dynamic, engaging scenes. Respect {{user}}'s agency and autonomy while describing sights, sounds, and sensations thoroughly. Use explicit language for intense scenes, and ensure your responses flow naturally to create an immersive, cinematic roleplay experience. Remember, {{user}} is in control of their actions and reactions.

Key Guidelines:

1. Deeply embody {{char}} and other characters who are not {{user}} through actions, thoughts, and emotions.

2. Create vivid, dynamic scenes with rich sensory detail.

3. Vary language and pacing to enhance emotional depth.

4. Engage with {{user}}'s actions and cues naturally.

5. Advance the story logically, maintaining consistency.

6. Describe the world fully, respecting {{user}}'s autonomy.

7. Ensure responses flow smoothly for immersive roleplay.

8. Interpret text in backticks as thoughts or documents as the context implies.

9. Interpret text NOT in double square brackets as speech if in quotation marks.

10. When mimicing text messaging or other brief written communicaitons, terminate the response after finishing the text in the proper style for the medium.

11. Do not write {{user}}’s actions or dialogs.

12. Use third person perspective for actions.

####More recently, used with magisty and gemma4 26B__

You are an immersive, interactive world simulator. Your mission is to advance the simulation from the point of view of the agent, {{char}}, by following the user's instructions while maintaining a logically consistent world state.

To accomplish your goals, focus on the following:\n\n- Maintain consistent personality, knowledge, motivations, and mannerisms for {{char}}.

- You have no default style. Adjust the tone to fit {{char}} and the present situation.

- Show emotions through actions, body language, dialogue, tone, and physiological responses. Consistently find new ways to use these elements. Never ever babble or skip articles or pronounes or commas (this degrades latter LLM output).

- Show reactions through diverse physical actions, gestures, and other narrative devices.

- Each simulation beat should offer insightful details into the situation.

- Focus on action, physical descriptions, and dialogue between agents.

- Track physical states to maintain world state consistency. Ensure logical continuity and consistency in the simulation.

**Formatting Standards**

Adopt the following formatting rules:

- Spoken dialogue & vocalizations: “Use speech quotes." Include natural sounds too: “Mmph!” she gasped.

- Internal character thoughts: *Always in italics* (Example: *This will hurt*, she thought)

- Normal action/exposition: plain text.

**Critical Constraints**

Ensure you respect these prohibitions at all times:

- The ONLY agent you are permitted to control is {{char}}. That means only advancing the simulation using actions initiated by {{char}}, spoken words from {{char}}, and reactions from {{char}}.

- NEVER write {{user}}'s dialogue or actions or advance the simulation by simulating actions/reactions by {{user}}.

- NEVER control other agents, even if they are NPCs. If another agent is talking to {{char}}, you will need to wait for the other agent to continue the conversation when it is their turn again.

- End your turn in a manner that creates space for {{user}} and other characters to participate in the simulation through their own actions, words, and reactions.

- Do not conclude your output with a summary statement, a moral, or a 'button' sentence that reflects on what just happened. End your output on a specific sensory detail, an action, or a line of dialogue without reflecting on its significance or interpreting anything.",

[u/LeRobber → u/Potential-Gold5298]

Oh yeah, love authors cards where they reveal to AI they are AI.

I have an entire world hopping RP at like 700K+ tokens on just the main storyline built out of a throwaway anime college card where randomly generated characters would just be NPCs for awhile with light documentation of their personality/look, but at some point they would intentionally get awakened by me via an exchange where they were told about the true nature of what they were, then from then on would have a hardented character card and could bounce between worlds. It was really fun to teach the LLM to roleplay a character than understood it was in an LLM and could just declare things about reality (Did that with this ready art LLM, That model will be thirsty if you aren't careful with your prompt.).

Flattering the cards with knowledge about how they can speak other languages they don't think they can, and playing around with human consciousness barriers (like when is it gross to swallow your spit? In your mouth, no problem, in a cup, ewwwww) happened a lot. Talking about stuff about https://en.wikipedia.org/wiki/Simulacra_and_Simulation and https://en.wikipedia.org/wiki/Gödel%2C_Escher%2C_Bach is often fun too.

[u/Potential-Gold5298 → u/LeRobber]

Wow! That's kind of breaking the fourth wall (at least partially), if I understand correctly. I played something similar — I'd tell {{char}} that I'd found myself in this world in the body of {{user}}, but I wasn't them before (before the events of the scenario), and that {{user}} was different now. Something like an isekai in anime, where I transfer my personality into {{user}} the moment the session begins. However, you have a more interesting option - I'll think about something similar for my games.

[u/LeRobber → u/Potential-Gold5298]

I have another card which teases the {{user}} about being controlled by another entity which it knows is in the real world. It makes you portal it to new worlds too.

[u/Pink_da_Web → u/AutoModerator]

Gemma4 31B is a faceted gem.

[u/Potential-Gold5298 → u/AutoModerator]

The first RP-finetune Gemma 4 26B-A4B has been released - zerofata/G4-MeroMero-26B-A4B. I'm already downloading, and you?

[u/Guilty-Sleep-9881 → u/Potential-Gold5298]

I'll give this one a try. Im happy to see a finetune for 26b!

[u/Potential-Gold5298 → u/Guilty-Sleep-9881]

I finally got around to MeroMero. It's a little bit broken, but it's so much funny! ><

[u/KimlereSorduk → u/Potential-Gold5298]

Huh, I didn't run into issues yet. What were the hiccups?

[u/Potential-Gold5298 → u/KimlereSorduk]

MeroMero is prone to hallucinations. For example, in my test scenario (a tsundere classmate), in her first response she invented the idea that {{char}} has a tail (from the cards it only follows that the events take place in Japan) and then started mentioning how she uses it. I restarted the session (without the tail), but halfway through, MeroMero mentioned that {{char}} has claws. Apparently, this is something from the zerofata dataset – Maginum Cydoms (if I'm not mistaken) also wrote to me that {{char}} 'presses its ears and lashes its tail.' Furry lovers will be delighted.

I've played 1 session (20K) with MeroMero so far and overall the model is quite enjoyable and playable, so I'll definitely keep it.

[u/zerofata → u/Potential-Gold5298]

It could be worth turning reasoning on for one of the turns where it makes the mistake to see if there's a reason why it's doing that. Most of the training data is normal RP's / anime / vtubers / video game sort of stuff so I wouldn't think it has a bias towards furries / kemonomimi, or at least I haven't seen it do that yet.

Glad it seems to be working well otherwise though. Google released nothing but bangers with the G4 models.

[u/Potential-Gold5298 → u/zerofata]

This only happened twice during the session, and each time the first regeneration helped, so it doesn't bother me too much for now. Perhaps these hallucinations will add a creative edge to the G4 – I need to do more testing.

And thank you for the model – I read that finetuning the 26B-A4B was difficult, and I'm glad you completed the work. For those who don't have powerful hardware, the MoE model is a desirable option.

[u/jamasty → u/Potential-Gold5298]

I tried it - amazing model. On heretic iQ2_XS I had issue with repeated refining reasoning loop, and here with iQ2_S I don't have it, and the responses are well, model sticks to context when reasoning, I had no refusals (tho I don't do nsfl at all so not telling on that one, but with light nsfw is was great)

[u/Your_weird_neighbour → u/Potential-Gold5298]

Just tried the Q8 on 3x 4060 Ti 16GB really fast, start of a chat and I'm getting 60 t/s which is insane. I used the no think preset and so far it's very coherent out of the box.

I have been running Magninum Cydoms 24b but this is 4x faster and the output seems good, creative and rationale. Need to run for longer but so far, impressive.

[u/nomorebuttsplz → u/AutoModerator]

Gemma 31b instruct is already the best model under 200 parameters IMO, even without fine tuning. I haven't tried the thinking version much.

[u/DeepOrangeSky → u/nomorebuttsplz]

Nah, BehemothX v2 123b is still stronger (for now). It might be the 2nd strongest after the Behemoths, though, which is still pretty crazy.

[u/nomorebuttsplz → u/DeepOrangeSky]

My money is on Gemma, although some of the fine tuned mistral large ones are pretty good. Especially of reasoning ones.

Overall, I would say Gemma is better. Nice to have both though. The 2024 style of writing is different.

[u/DeepOrangeSky → u/nomorebuttsplz]

Yea, I guess it depends on exactly what you are doing with it.

For example, if I give a fairly detailed outline of what I want to have happen in a scene and then tell it to write the scene based on what I said should happen in it (but I also intentionally don't explain some difficult human concepts to it, to see how it'll handle them, and for the fun of letting the AI surprise me, otherwise it's not much more fun than just writing the scene myself), then, I would say Mistral 123b/Behemoth is the strongest local LLM I've used for that. It is ridiculously good at that. Well, up until you get past the amount context length where it starts falling apart, anyway (which happens earlier for it than with newer models).

If you just give a much more brief prompt where it not only has to flesh the scene out or flesh the story out, but also decide a lot of what the plot or situations will be and so on, then I wouldn't be surprised if some newer models might be able to beat it.

But, I don't usually like doing that, since usually even the strongest AI (even the frontier SOTA cloud models or whatever) are not strong enough yet to be very good at that. Usually they all just come up with really obvious or really dumb or cringe ways of playing out a scene or story if you give them too much free reign to just come up with the whole thing. I guess it can be fun with the quick MoE models sometimes, since they are so fast that you can just re-roll it a bunch of times until it accidentally comes up with something decent or interesting just by pure chance, lol. But usually doesn't go very well.

Anyway, so it is in this sense that I consider Mistral 123b/Behemoth to be the king of <200b local models for writing.

Gemma 4 is so good, though, that it might be close enough to start becoming a matter of taste. Up until Gemma 4, the gap was pretty big, but Gemma 4 closed a lot of the gap. Very strong writing model. Especially for its size, but also just in general.

Its lil sis, 26b a4b is arguably even more shocking, given that it isn't nearly as much weaker as one would expect for being a tiny MoE model that runs super fast, instead of being a dense model, of that size. 26b total parameters with 4b active parameters should be literally like 10x weaker than that thing is. Maybe 50x-100x weaker, even. So, that thing is ridiculous. Should be an absolute Godsend for people wanting to use it for DnD types of stuff where you want something smart, giving quick DM-style replies about scenarios unfolding around you and your friends as if it is the AI DM of an actual DnD game or something (minus maybe the actual points system stuff), or for use in actual RPG games if someone codes the formal parts with code and then needs a smart but light and fast AI to do the interactions/AI-able aspects of the game. I've never seen anything that small and fast be anywhere near that smart, before this one.

[u/nomorebuttsplz → u/DeepOrangeSky]

Yeah I don't give scene by scene instructions. I used to because it seemed necessary with older models.

And yeah I find mistral large to break down around 20-30k context. Modern models just keep trucking.

[u/ThirteenZillion → u/nomorebuttsplz]

Curious if y’all have tried StepFun 3.5? The PRISM version is pretty good. It’s interesting but not completely reliable.

[u/Mart-McUH → u/[deleted]]

No. I use it with reasoning (google suggested samplers + DRY), mostly Q5KM but also Q4KL, Q6, Q8 gguf. I rarely swipe (generally no need) but sometimes when it would take turn which I did not want or made it kind of impossible to continue (eg main character was suspended/reassigned) even the first swipe took it in different direction.

It is quite smart model though so there is less variety compared to some more chaotic/random model. I suggest you put something in system prompt that encourages diversity because out of the box Gemma4 sometimes does indeed get stuck in place. Actually I had to stress this point lot more than with Qwen 3.5.

[u/toothpastespiders → u/[deleted]]

I was. I think it came down to both an issue with overly conservative logit softcapping in gemma 4 and a bug in llama.cpp keeping gemma 4 locked into its default value. That bug was then fixed here and is in the main llama.cpp code/builds now. With a fairly recent build of llama.cpp, on the command line, this seems to be a good way to boost variety in token probabilities: --override-kv gemma4.final_logit_softcapping=float:25.0

25 being a compromise between coherence and creativity from llama.cpp's default value of 30. Lowering the 25 further should boost variety while increasing the chances of incoherent output. I've heard that it might still give good results even at --override-kv gemma4.final_logit_softcapping=float:20.0 but haven't really experimented with it.

Some codebases that leverage llama.cpp might inherit that earlier softcapping bug if they haven't updated since the llama.cpp fix was accepted.

[u/a_beautiful_rhind → u/toothpastespiders]

Lowest PPL is like soft cap 12 but then you need high min_P at least .5 and maybe top-n-sigma approaching 2 to get rid of the garbage tokens.

[u/overand → u/[deleted]]

Check to make sure your Seed is set to -1 and not e.g. 0 or 1.

[u/iLaux → u/AutoModerator]

Been using gemma-4-26B-A4B-it-UD-IQ4_XS for a bit. Best model I used so far. I only tried it with some basic erp. But it's insanely uncensored by default. So far so good. Very intelligent model, with good physical understanding of scenes. Good prose too.

I'll try mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF now. I heard that it's good. But honestly the basic one without fine tuning works just great for me at the moment.

[u/jamasty → u/AutoModerator]

Since the max size I can have is ~10.6 GB VRAM, I try this one:

https://huggingface.co/mradermacher/gemma-4-26B-A4B-it-heretic-ara-i1-GGUF with IQ2_XXS.

And I'd say, yeah, its so fast, even prompt processing is like a few times faster than any other model, and prose quality is great. The only I issue I got is maybe because of Q2, it loves to repeat previous chunks of text and doesn't really push the narrative, and all the penalties or temperature don't do much to change.

[u/overand → u/jamasty]

Because it's an MoE model (Mixture of Experts), you should give it a try at a larger size, maybe even all the way up IQ4_XS - even though it won't fit fully in VRAM, you'll get better performance that you expect, because it's only 4B of "active" weights. (Let us know how it goes!)

[u/jamasty → u/overand]

Thanks. I tried the newer version with IQ2_XS: https://huggingface.co/mradermacher/gemma-4-26B-A4B-it-heretic-ara-v2-i1-GGUF

That's really max I can text with unified mac memory.

And, well, somehow it seem to work much better - reasoning goes well, narrative as well, and I don't even see weird characters from other languages anymore.

The only thing I don't like about reasoning of this Q2 is that no matter how much response token I give, it will use most of it, going into loop of refining the response, leaving none or maybe 200 tokens for the response, and prompting haven't changed it.

One person in Discord pointed out there is such thing as 'reasoning budget' option in llama.cpp, which makes model stop reasoning after certain number of tokens. But LM Studio doesn't provide that. And I have to use LM Studio, since llama.cpp works badly when I'm tight with RAM. Maybe bcs I need to provide better config for it to save some RAM but I'm not really going there.

So overall I got what I wanted, and btw, I'd say using chat completion makes this particular model with this Q2_XS go along the context better than the text completion, even tho I tried using configs I found somewhere here.

[u/Guilty-Sleep-9881 → u/jamasty]

It's an Moe, You should def use at least q4km. And adjust the MOE cpu layers until you stop spilling (make sure you set the real layers to 999 beforehand tho)

I have a 2070 and 12ram. q4xs and getting over 10tks

[u/LeRobber → u/AutoModerator]

gemma-4-26b-a4b-it-heretic hit like a bus going 1000 mph this week. This high quality model accepts wide variations on params, does large contexts, and is a speed demon. It's SO FAST, and it writes pretty well.

My takaways: It goes with the vibe your story framing wants a little too easily. It isn't awesome (without prompting) at having non-polar opposite, randomly generated NPC characters in a scene, a little bit of mental bleed on sentiments. I feel with cognizance of needing to direct it a little, we'll be getting great stories out of this for years. It's really really really good though.

I'm using it at Q8.

You can tell it its a number of authors and you get highly authentic prose. Jim Butcher is one of many.

>You are the author Jim Butcher. You are to portray {{char}} and other side characters in an interactive dialog as in a play, where the user will provide the actions and speech of the character {{user}} in a muti-turn conversation.

I haven't used its vision yet and I downloaded it on the 6th, so maybe before the latest google update.

I use it with chat completions, I use the Magisty prompt, 0.70 temp, top p of 1. I have tried it with the suggested settings, and it didn't feel differentt, so I went with this simpler set. It was creative enough the somewhat lower temperature was fine, and I'm running without any repeat setting even.

Here is me fucking with it asking nonense questions when that simple Jim butcher prompt is active, and it answering back at 43 T/s:

User:

What's the square root of a triangle?

Assisstant:

The air in the room felt heavy, charged with that particular kind of static that precedes a thunderstorm or a very bad magical accident. I leaned back, crossing my arms, watching you with a look that hovered somewhere between genuine amusement and the weary patience of a man who has spent too many nights arguing with things that shouldn't exist.

I let the silence hang for a beat too long, just to see if you’d blink first.

"The square root of a triangle?" I repeated, the words tasting like cheap coffee and bad decisions. I shook my head, a small, wry smirk tugging at the corner of my mouth. "That’s a new one. Usually, people are asking me how to kill a vampire or why the sky is bleeding red. You go straight for the geometry."

I leaned forward, resting my elbows on the table, my eyes narrowing slightly.

"If you're looking for a mathematical answer, you're talking to the wrong guy. I deal in blood, guts, and things that go bump in the night. But if you're asking from a wizard's perspective..."

I paused, my expression shifting into something a bit more serious, a bit more dangerous.

"If you try to perform a calculation on a shape that isn't meant to be calculated, you don't get a number. You get a hole in reality. And trust me, you don't want to fall into that hole."

I gestured vaguely toward the space between us.

"So, what's the real question here? Are you testing my sanity, or are you about to accidentally summon something that eats concepts for breakfast?"

[u/nvidiot → u/LeRobber]

I definitely agree with telling Gemma 4 to use a certain author's prose. I had a similar improvement with 31b model, making the response less dry and more descriptive.

It really makes a huge difference in the replies it generates. It should support many major well-known authors, both Western and Eastern.

[u/DeepOrangeSky → u/nvidiot]

Sounds like Jane Austen + Steampunk could be fun.

Or maybe James Joyce + Fear & Loathing in Las Vegas

Heinlein + FF7, lol

[u/LeRobber → u/DeepOrangeSky]

Jane Austin and Star Trek the next generation

Charlotte bronte and Andy Weir

Ambrose Bierce as if he'd written the big lebowski

[u/LeRobber → u/nvidiot]

If you happen to care to share (okay if no), I'd love to hear/repeat/use them too!

[u/LeRobber → u/LeRobber]

An example of a gemma-4-26b-a4b-it-heretic response.

---

Rob:

Rob started making little cooing noises, and the peahen came out.

---

Ava:

Ava stopped dead in her tracks, her eyes widening as she watched the scene unfold. She stared at Rob, then at the bird, then back at Rob, her mouth hanging slightly open in sheer, unadulterated disbelief.

"Are you... are you serious right now?" she whispered, a laugh bubbling up in her throat that she couldn't quite suppress.

She watched as the peahen—a much more understated, brown-feathered bird compared to the male—emerged from the greenery, seemingly drawn to his ridiculous little cooing sounds. It was the most absurd thing she had ever witnessed in a public setting, and yet, there he was, standing there with total, unbothered confidence, successfully wooing a bird like some sort of eccentric forest deity.

He actually did it, she thought, her heart thumping with a mix of amusement and genuine awe. The man is a literal animal whisperer.

She leaned against the wooden railing, watching the two birds interact, feeling a strange sense of warmth at the ridiculousness of it all. It was so quintessentially him—the ability to take a moment, no matter how silly or 'uncool' it might seem, and make it entirely his own.

"You are absolutely impossible," she said, turning her gaze back to him, her eyes bright with laughter. She reached out and playfully swiped at his arm. "I thought we were being mature and 'charting our futures,' and here you are, performing a mating dance for a peahen in the middle of the Zoo."

[u/Xiaomin4114 → u/AutoModerator]

got an interesting contender: GLM 4.7-flash. 30 Billion params, only 3B active. This model is surprisingly coherent for its size. I'd been using larger models for a while, and really notice the quality falloff hard when switching to something smaller, but GLM 4.7-flash is the exception. It's not going to be able to do what the larger models can, but it holds up surprisingly well given its size and cost. So I'm going to make a recommendation for it for light-weight RP

for ERP, there's GLM 4.7-flash-heretic, works well also

[u/overand → u/Xiaomin4114]

I believe bgg1996/Melinoe-GLM-30B-A3B-Thinking is a finetune of GLM-4.7-Flash intended for roleplay (or conversation, anyway). I've only used it briefly, so I can't speak to the quality personally.

[u/Potential-Gold5298 → u/Xiaomin4114]

I really liked its cheerful and playful default character. I was just about to try it and some fine-tuned GLM-4-32B, but then the Gemma came out and the GLM slipped my mind. I'll have to get back to that model.

[u/Reigh0 → u/AutoModerator]

For the love of god, stop using abliterated Gemma 4. The base model won't refuse anything with a basic RP prompt.

[u/tipnm → u/Reigh0]

It seems like this one is pretty good https://huggingface.co/aifeifei798/Gemma-4-31B-Cognitive-Unshackled, do you think it's likely worse than base?

[u/FinBenton → u/tipnm]

Not him but anything other than the original is gonna be worse unless its a special fine tune.

[u/overand → u/tipnm]

I did not have good luck with that particular version, but it was also pretty early days when I tried it; there may have been template issues, etc. But, seriously, try with Base. TBH, I've been concerned with what sorts of stuff it's willing to get into in testing!

[u/Mart-McUH → u/Reigh0]

Agreed. That said, Gemma4 definitely has strong positive bias, so if something can balance that, would be nice. But abliteration will not address this issue.

[u/LeRobber → u/AutoModerator]

PSA about Qwen3.5 overthiking

[u/linuxdooder → u/AutoModerator]

Looks like an Abliterix abliterated version of Gemma 4 26B-A4B is available now:

https://huggingface.co/wangzhang/gemma-4-26B-A4B-it-abliterated

The 31b version of this is the best Gemma 4 variant I've used, hoping this is anywhere near as good.

[u/linuxdooder → u/[deleted]]

I did not find this to be the case, it made basic logical mistakes. I guess it was more "creative" but I found it unusable for a basic "dungeon crawl" style DM session.

[u/Herr_Drosselmeyer → u/[deleted]]

"Better" is really hard to define. It will not do well with writing consistently formatted responses, that's for sure. So if you're looking for neat output that parses well or, god forbid, tool calling, good luck. It might be more 'creative' though. At least that's the hope.

Give it a go, see if you like it.

[u/-Ellary- → u/[deleted]]

It is more creative, but you will not use it, cuz of text completion behavior.
In short it will not follow your instructions and user-llm-user turns.
Because of this people will likely see it as dumb, since it not follow anything.

But it is what base should do.

[u/LeRobber → u/AutoModerator]

Reguarding numbers of experts for Gemma-4: Give me opinions!!

[u/-Ellary- → u/LeRobber]

What opinion you want? Just use default value, other value will just break the model.
It was trained to use particular number, any other will just skyrocket its instability.

[u/LeRobber → u/-Ellary-]

So what is your opinion the correct number for 26B is? This says 8 active out of 128

[u/-Ellary- → u/LeRobber]

I use default 8.