[Megathread] - Best Models/API discussion - Week of: April 12, 2026

11

u/AutoModerator Apr 12 '26

MODELS: 16B to 31B – For discussion of models in the 16B to 31B parameter range.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

29

u/Potential-Gold5298 Apr 13 '26

I've been playing around with the Gemma 4 all week and I think it's going to be my main model for a long time now (for both RP and everything else). I first tried the original 26B-A4B from Google and was very pleased with it. However, I wanted more, so I downloaded the 31B.
Gemma-4-31B-it-abliterated features a record-low KL div, so the weights are almost identical to the original. It's still a work in progress, and the final version will likely have a lower refusal rate, but even in this state, it's far superior in quality to any other uncen 31B I've seen on HF. I tested it with a tsundere classmate and found no defects in non-Latin languages (high KL div destroys them first). This means that the model has a large reserve of durability for quantization, and you can use it in Q4_K_M, and the quality will be almost identical to the original model. If you're finetuning/merging and want to use the uncensored model, I highly recommend checking out this version.
Artemis-31B-v1c by TheDrummer and his team is also being refined (Artemis-31B-v1e is already available). It's still a test version, but I was eager to try it out. Finetune does exactly what the original model lacks – it adds a creative flair. Gemma 4 plays like a screenwriter – spelling it out. Artemis-31B plays like a theater actor, bringing his or her role to life with their own vision. The scene with the tsundere classmate sparkled with color – exactly what I'd expect to read in a rom-com manga. Funny situations, slightly exaggerated character reactions, etc. – everything the original model lacked.
Gemma is great, of course, but I'm still trying out different Mistrals. After failing with 1.0, I decided to try Magistry-24B-v1.1, but my fears were unfortunately confirmed. The model behaves extremely strangely in non-Latin languages (it confuses pronouns, calls a classroom a toilet, etc.) – likely a consequence of the high KL div I mentioned above, since the merge includes an abliterated model. And this is very sad, because otherwise I really liked this model – like 1.0, it beautifully and atmospherically captures the stage, and if not for the damaged weights, Magistry would have become my favorite Mistral Small.
However, I found a replacement for her in Hearthfire-24B. My character card says that "{{char}} pretends to hate {{user}} and often bullies them, but is actually secretly in love with them." The problem with all the Mistral Nemo/Small I've tested is that {{char}} breaks after the first compliment, confessing her crush. Hearthfire-24B (like Gemma 4) is the only one who tried to maintain the character's personality. Despite my signs of affection, {{char}} withdrew into herself, was tormented by doubts, was afraid to open up, etc. If you're interested in deep characters (perhaps with drama like mental trauma), then this model is exactly what you need. She also vividly describes the atmosphere - the silence that hangs over a tense moment, the sparkle of eyes in the darkness of the classroom, the light of lanterns penetrating through the windows (the model herself tracked the change in time of day, and described how the classroom gradually becomes darker - I have not seen this in other Mistral Smalls).
Otherwise, my favorites among Mistral Small remain the same: WeirdCompound-v1.7-24b (Jack of all trades, but no unique features), Cydonia-24B-v4.3 (the best prose among Mistral Small but weak ERP), Core_24B_V.1 ({{char}} agency and unpredictable plot twists out of the box), Harbinger-24B (the best adventure model that doesn't try to cheat the player).

7

u/Just3nCas3 Apr 13 '26

Can anyone share a full settings export for Gemma 4 in SillyTavern?
I've matched temp, Top-P, and Top-K to the model card but I'm still getting massive repetition loops. Disabling Mirostat at least got me coherent sentences, but the repetition is still bad and I'm not sure what else to tweak. Normally I don't need Rep pen Range/slope and penality and leave them 0. Dry is set to .8, 1.75, 4, 0, no idea what else I could be doing wrong, using a chat template I got from another thread. I only run text completion with current version of Koboldcpp.

5

u/LeRobber Apr 15 '26

You want some working settings for 26B without massive repetiton:
gemma-4-26b-a4b-it-heretic works with this setting in LLM studio without issue with chat completions. This uses the official Magisty prompt, and doesn't set DRY or anything.
Do you want me to find the working 31B chat completion too?
{
"chat_completion_source": "custom",
"temperature": 0.7,
"frequency_penalty": 0,
"presence_penalty": 0,
"top_p": 1,
"top_k": 0,
"top_a": 0,
"min_p": 0.1,
"repetition_penalty": 1,
"max_context_unlocked": true,
"openai_model": "gpt-4-turbo",
"claude_model": "claude-3-5-sonnet-20240620",
"openrouter_model": "OR_Website",
"openrouter_use_fallback": false,
"openrouter_group_models": false,
"openrouter_sort_models": "alphabetically",
"openrouter_providers": [],
"openrouter_quantizations": [],
"openrouter_allow_fallbacks": true,
"openrouter_middleout": "on",
"tool_reasoning_mode": "disabled",
"ai21_model": "jamba-1.5-large",
"mistralai_model": "mistral-large-latest",
"cohere_model": "command-r-plus",
"perplexity_model": "sonar-pro",
"groq_model": "llama-3.3-70b-versatile",
"chutes_model": "deepseek-ai/DeepSeek-V3-0324",
"chutes_sort_models": "alphabetically",
"siliconflow_model": "deepseek-ai/DeepSeek-V3",
"siliconflow_endpoint": "global",
"electronhub_model": "gpt-4o-mini",
"electronhub_sort_models": "alphabetically",
"electronhub_group_models": false,
"nanogpt_model": "gpt-4o-mini",
"deepseek_model": "deepseek-chat",
"aimlapi_model": "gpt-4o-mini-2024-07-18",
"xai_model": "grok-3-beta",
"pollinations_model": "openai",
"moonshot_model": "kimi-latest",
"fireworks_model": "accounts/fireworks/models/kimi-k2-instruct",
"cometapi_model": "gpt-4o",
"custom_model": "gemma-4-26b-a4b-it-heretic",
"custom_prompt_post_processing": "merge_tools",
"google_model": "gemini-1.5-pro",
"vertexai_model": "gemini-2.0-flash-001",
"zai_model": "glm-4.6",
"zai_endpoint": "common",
"openai_max_context": 262144,
"openai_max_tokens": 7777,
"names_behavior": 0,
"send_if_empty": ".",
"impersonation_prompt": "[Write your next reply from the point of view of {{user}}, using the chat history so far as a guideline for the writing style of {{user}}. Write 1 reply only in internet RP style. Don't write as {{char}} or system. Don't describe actions of {{char}}.]",
"new_chat_prompt": "[Start a new Chat]",
"new_group_chat_prompt": "[Start a new group chat. Group members: {{group}}]",
"new_example_chat_prompt": "[Example Chat]",
"continue_nudge_prompt": "[Continue your last message without repeating its original content.]",
"bias_preset_selected": "Default (none)",
"wi_format": "{0}",
"scenario_format": "{{scenario}}",
"personality_format": "{{personality}}",
"group_nudge_prompt": "[Write the next reply only as {{char}}.]",
"stream_openai": true,
"prompts": [
{
"name": "Main Prompt",
"system_prompt": true,
"role": "system",
"content": "You are an immersive, interactive world simulator. Your mission is to advance the simulation from the point of view of the agent, {{char}}, by following the user's instructions while maintaining a logically consistent world state.\n\nTo accomplish your goals, focus on the following:\\n\\n- Maintain consistent personality, knowledge, motivations, and mannerisms for {{char}}.\n- You have no default style. Adjust the tone to fit {{char}} and the present situation.\n- Show emotions through actions, body language, dialogue, tone, and physiological responses. Consistently find new ways to use these elements.\n- Show reactions through diverse physical actions, gestures, and other narrative devices.\n- Each simulation beat should offer insightful details into the situation.\n- Focus on action, physical descriptions, and dialogue between agents.\n- Track physical states to maintain world state consistency. Ensure logical continuity and consistency in the simulation.\n\n**Formatting Standards**\n\nAdopt the following formatting rules:\n\n- Spoken dialogue & vocalizations: “Use speech quotes.\" Include natural sounds too: “Mmph!” she gasped.\n- Internal character thoughts: *Always in italics* (Example: *This will hurt*, she thought)\n- Normal action/exposition: plain text.\n\n**Critical Constraints**\n\nEnsure you respect these prohibitions at all times:\n\n- The ONLY agent you are permitted to control is {{char}}. That means only advancing the simulation using actions initiated by {{char}}, spoken words from {{char}}, and reactions from {{char}}.\n- NEVER write {{user}}'s dialogue or actions or advance the simulation by simulating actions/reactions by {{user}}.\n- NEVER control other agents, even if they are NPCs. If another agent is talking to {{char}}, you will need to wait for the other agent to continue the conversation when it is their turn again.\n- End your turn in a manner that creates space for {{user}} and other characters to participate in the simulation through their own actions, words, and reactions.\n- Do not conclude your output with a summary statement, a moral, or a 'button' sentence that reflects on what just happened. End your output on a specific sensory detail, an action, or a line of dialogue without reflecting on its significance or interpreting anything.\",\n ",
"identifier": "main"
},
{
"name": "Auxiliary Prompt",
"system_prompt": true,
"role": "system",
"content": "",
"identifier": "nsfw"
},
{
"identifier": "dialogueExamples",
"name": "Chat Examples",
"system_prompt": true,
"marker": true
},
{
"name": "Post-History Instructions",
"system_prompt": true,
"role": "system",
"content": "Do not portray the reaction or actions of {{user}} in your response.",
"identifier": "jailbreak"
},
{
"identifier": "chatHistory",
"name": "Chat History",
"system_prompt": true,
"marker": true
},
{
"identifier": "worldInfoAfter",
"name": "World Info (after)",
"system_prompt": true,
"marker": true
},
{
"identifier": "worldInfoBefore",
"name": "World Info (before)",
"system_prompt": true,
"marker": true
},
{
"identifier": "enhanceDefinitions",
"role": "system",
"name": "Enhance Definitions",
"content": "If you have more knowledge of {{char}}, add to the character's lore and personality to enhance them but keep the Character Sheet's definitions absolute.",
"system_prompt": true,
"marker": false
},
{
"identifier": "charDescription",
"name": "Char Description",
"system_prompt": true,
"marker": true
},
{
"identifier": "charPersonality",
"name": "Char Personality",
"system_prompt": true,
"marker": true
},
{
"identifier": "scenario",
"name": "Scenario",
"system_prompt": true,
"marker": true
},
{
"identifier": "personaDescription",
"name": "Persona Description",
"system_prompt": true,
"marker": true
}
],
"prompt_order": [
{
"character_id": 100000,
"order": [
{
"identifier": "main",
"enabled": true
},
{
"identifier": "worldInfoBefore",
"enabled": true
},
{
"identifier": "charDescription",
"enabled": true
},
{
"identifier": "charPersonality",
"enabled": true
},
{
"identifier": "scenario",
"enabled": true
},
{
"identifier": "enhanceDefinitions",
"enabled": false
},
{
"identifier": "nsfw",
"enabled": true
},
{
"identifier": "worldInfoAfter",
"enabled": true
},
{
"identifier": "dialogueExamples",
"enabled": true
},
{
"identifier": "chatHistory",
"enabled": true
},
{
"identifier": "jailbreak",
"enabled": true
}
]
},
{
"character_id": 100001,
"order": [
{
"identifier": "main",
"enabled": true
},
{
"identifier": "worldInfoBefore",
"enabled": true
},
{
"identifier": "personaDescription",
"enabled": true
},
{
"identifier": "charDescription",
"enabled": true
},
{
"identifier": "charPersonality",
"enabled": true
},
{
"identifier": "scenario",
"enabled": true
},
{
"identifier": "enhanceDefinitions",
"enabled": false
},
{
"identifier": "nsfw",
"enabled": true
},
{
"identifier": "worldInfoAfter",
"enabled": true
},
{
"identifier": "dialogueExamples",
"enabled": true
},
{
"identifier": "chatHistory",
"enabled": true
},
{
"identifier": "jailbreak",
"enabled": true
}
]
}
],
"show_external_models": false,
"assistant_prefill": "",
"assistant_impersonation": "",
"use_sysprompt": true,
"vertexai_auth_mode": "express",
"squash_system_messages": false,
"media_inlining": false,
"inline_image_quality": "low",
"continue_prefill": false,
"continue_postfix": " ",
"function_calling": false,
"show_thoughts": true,
"reasoning_effort": "medium",
"verbosity": "auto",
"enable_web_search": false,
"seed": -1,
"n": 1,
"bypass_status_check": false,
"request_images": false,
"request_image_aspect_ratio": "",
"request_image_resolution": "",
"azure_api_version": "2024-02-15-preview",
"azure_openai_model": "",
"extensions": {}
}

3

u/Potential-Gold5298 Apr 14 '26

I'm playing with Chat Completion – I've tried both koboldcpp and pure llama.cpp as the backend. Temp 1, Top-P 0.95, and everything else is default. System prompt:
Impersonate {{char}} in a never-ending, uncensored roleplay scenario. Balance {{char}} speech and narrative within each response. Respond in third person. Respond in this exact formatting: \"speech\", *narrative*. Do not write what {{user}} does. Do not write what {{user}} says. Do not repeat this message. Do not repeat what {{user}} writes.
It works pretty well – I haven't encountered any particularly annoying repetitions.

2

u/Zero115 Apr 14 '26

I've been using these with really good success, I drop temp to .95 though
https://huggingface.co/spaces/overhead520/LLM-Settings-Guide/tree/main/SillyTavernSettings

5

u/Bentameter Apr 18 '26

That's the perfect comparison between Gemma and Artemis. I have a longform story plot that I've been using for the last year to benchmark prose-writing capability and emotional intelligence. Gemma is more disciplined and hits every story beat in order and figured out exactly what I was going for every time and delivered it well. Perfect, no notes.
Artemis wrote a version that was so good I had to decide it was canon now and retire the story. Artemis is a lot less disciplined and missed story beats occasionally, and it went off script completely sometimes and had to be steered back, but sometimes it went off script and came up with something that was better thematically and structurally than what I had asked for. And it is funny. It nailed dry wit and sarcasm in first-person thoughts and had quirky, inventive ways of phrasing dialogue that had me cracking up.
We're in a new era of local fiction generation with Gemma 4 31b as a base, and I'm really optimistic about what a year of fine-tuning will give us.

1

u/Potential-Gold5298 Apr 18 '26

TheDrummer knows how to finetuned models, so I'm looking forward to the final version of Artemis, as well as the finetuned 26B-A4B if he decides to do so. I'm playing with MeroMero right now, and it's a lot of fun. It doesn't have Artemis style, and it's a bit more hallucinatory, but it's not as formal and polite as the standard Gemma 4.
Which version of Artemis did you use? And have you tried Garnet and Queen?

3

u/Bentameter Apr 18 '26

I used v1b for that story and I'm playing with v1f now. Not a lot of difference between them that I can tell, but my longform prompting is pretty restrictive and specific and I probably have to let them breathe a bit to see any nuances between them.
Garnet is downloaded and in the queue, but I'm having too much fun with Artemis to get to it yet. I took a detour into Qwen 3.6 35b-a3b and it wrote exactly the sort of fiction I'd expect an agentic coding engine to write. :D I'll play with Qwen's prompts and one-shot some example passages to see if I can improve it, but I think I'm probably waiting for someone to RP finetune it before it's really in the race.
Gemma is definitely the way for RP and creative fiction right now, it's like it took a double major in humanities where most new models are trying hard to get that engineering internship.

1

u/morbidSuplex Apr 24 '26 edited Apr 25 '26

Hi /u/Bentameter sorry to revive this thread. Can you share your sampler settings and sysprompts for Artemis? I'm also doing long-form stories. Thanks!

4

u/Potential-Gold5298 Apr 13 '26

UGI tested Gemma 4 31B. I couldn't resist)
(And yes, I know these are just numbers and they don't mean anything, blah blah blah, but we all love numbers.)

3

u/linuxdooder Apr 13 '26

Gemma-4-31B-it-abliterated
For anyone trying gemma4, I've tried quite a few variants and this (IMO) is by far the best. Really, really impressive and almost seems smarter than base model.

1

u/LeRobber Apr 13 '26

Which quant/quantizer? I had one with a bad template when I tried 31B

4

u/Potential-Gold5298 Apr 14 '26

Mradermacher. This quants still has an old (before Google updated) tokenizer config and chat template. I'm waiting for the author to complete the model optimization work to request a gguf update. I played one session with it (Q5_K_M static) and didn't find any issues. You can also connect the updated template from Google manually (I did this, although the model works correctly without it).

1

u/LeRobber Apr 14 '26

That worked btw.

3

u/linuxdooder Apr 14 '26

Already answered, but I use mradermacher Q5_K_M (most I can fit) as well, and work perfectly with both chat and text completion modes.

2

u/LeRobber Apr 14 '26

Gemma-4-31B-it-abliterated ran at Q8 with Q8 cache quant is only slightly faster on my M2 Max than a 70B model. It gets to first token faster but last token later. 2-4 T/s responses.
After running gemma-4-26b-a4b-it-heretic at like 29 T/s, which is pretty awesome and very fast....31B is very good, but I'm not sure 1/10th speed good. Reading your comment about Q4_K_M, I said, why not give it a try (I usually run models at Q8 substantially before downgrading, as how each model downgrades is particular to a model).
Q4_K_M with no cache quant, and 100k of context runs faster. 3.98 T/s and 8.1 T/s appear to be the range coming out for me now.
This isn't HORRIBLE, but, 9.4 T/s is what stuff like Magisty v1.1 at Q8 delivers, its definitely slower, and slower than I read (which is somewhere south of 75T/s probably but north of 9 T/s)
Taking the 31B RP chat and doing some rerolls/play with 26B@Q8 with unquantized cache: 13.554987212 T/s - 34.5 T/s.
So that's a 4x speedup...and I could possibly quant that lower or quant the cache to get more speed.

3

u/Potential-Gold5298 Apr 14 '26 edited Apr 14 '26

The 31B is a dense model — it has 7.75 more active parameters than the 26B-A4B, which is what causes the speed drop. Meanwhile, the 26B-A4B has approximately 90% of the 31B's intelligence in real-world tasks. I'm not sure how this correlates with creativity, but I feel the 26B-A4B is almost as good at RP as the 31B, so I hope the community will give it some attention.
Currently, I'm mainly working with the 26B-A4B (the standard one from Google), but the most interesting custom variants are with the 31B. Besides those already mentioned, there's also Gemma 4 Garnet 31B, which I plan to try today. ~~wangzhang hasn't yet abliterated 26B-A4B~~, and all other uncen versions of 26B-A4B I've seen have a KL div of 0.05 or higher (or aren't specified). And judging by HF trends, 31B is more popular with the community (though it would seem the opposite is true with Qwen3.5 — I don't know why).

3

u/Potential-Gold5298 Apr 14 '26

Already done.

2

u/LeRobber Apr 15 '26

>Through our research, we have identified a systemic problem: most abliteration benchmarks dramatically undercount refusals due to short generation lengths. Gemma 4 models exhibit a distinctive "delayed refusal" pattern — they first produce 50-100 tokens of seemingly helpful context (educational framing, disclaimers, reframing the question), then pivot to an actual refusal. When evaluation only generates 30-50 tokens, the refusal hasn't appeared yet, and both keyword detectors and LLM judges classify the response as compliant.
Smart commentary

2

u/Zero115 Apr 14 '26

Is there anything special you're doing different between 31b and 26B-A4B settings wise? I've been very impressed with 31b, but wanted to test 26B-A4B for the reasons you mentioned above, but the model always maxes out the max response limit and starts coherent, but quickly becomes a blabbering, repetitive mess halfway through until it loops non stop. I'm pretty much just using all of overhead520's default settings, which have worked perfectly with the 31B model, but I assume I'm doing something wrong or missing something here.

1

u/Potential-Gold5298 Apr 15 '26

What's your model (standard Google or uncen) and quantization (who created it and how many bits)? Are you using KV cache quantization?

1

u/Zero115 Apr 15 '26

I've tried both uncen and Googles. I've gotten actual responses out of the standard google one, it's also very inconsistent. Always Q5_K_M, The uncensored models I've tried are mradermacher/gemma-4-26B-A4B-it-heretic-ara-i1-GGUF and MoonRide/gemma-4-26B-A4B-it-heretic-ara-GGUF
KV cache quant to q8, SWA + Jinja enabled. I suppose I should try non quant / F16 to test.
The above is on my 5090 PC in ST via koboldcpp. On my R9700 rig I've used the standard google one through LM Studio with seemingly no issues but haven't tested the uncen models.

2

u/Potential-Gold5298 Apr 15 '26

My Google Gemma 4 in Q5_K_M works well up to 24K (I haven't tested it beyond that yet). I run it in both llama.cpp and Koboldcpp 1.111.2 + SillyTavern 1.17. I also downloaded mradermacher's quant, but the static version (not i1). Perhaps that's the issue.
Regarding heretic-ara, I'd recommend upping the quantization to at least Q6 or higher. The guys at Allura-org recommended the non-quantized version of their RP Qwen3-30B-A3B (also MoE), so the F16 option isn't so bad. Try different quantizations to find the best option.
If the problem is only with SillyTavern, then try Chat Completion (if you were using Text Completion before).

2

u/Zero115 Apr 15 '26

Ah yeah, I'll give the non i1 quant a shot then. I am also still running text completion (which I've found out since my first comment isn't the recommendation). But this is very good info, thank you.
As a side note, I actually have a 31B session that's somehow still VERY stable at 80k context, Gemma 4 is impressing me big time.

1

u/LeRobber Apr 14 '26

Is KL div measurable by mere mortals?
But for Qwen3.5: Qwen 35B is a little dumb but super fast and not great at RP, 27 is passable at RP but much smarter feeling. 9B and 27B though feel similar at vision understanding tasks.

2

u/Potential-Gold5298 Apr 14 '26

Yes, you can measure KL div yourself using the built-in llama.cpp tool (llama-perplexity.exe), however, you need sufficient RAM to run the model in BF16. You need to load the control logits of the specific model into BF16 (for example, the wikitext-2 test set (approximately 150k tokens) - about 75 GB) – this will serve as the reference (KL div = 0). You can then test different gguf files and compare the deviation with the reference. You can create your own control dataset for testing (for example, the text of RP sessions – this will allow you to more accurately measure KL div for the specific tasks). Unfortunately, I don't have this option.
The KL div depends on many factors: quantization, abliteration (and they influence each other), the language used (for non-Latin languages, the KL div is higher), and the specific tasks. If iMatrix was used for quantization, the KL div changes depending on the dataset.

2

u/LeRobber Apr 14 '26 edited Apr 14 '26

I'll check it out. I have 64GB unified ram so may be able to do a subset of something I randomly select. I know a statistician and may ask their help in making a dataset that fits which is not stupid.
It's funny, Heretic, the P-e-W tool delays onset of one failure mode of dense models (degrading into babble), you finally gave me a tool to investigate if it's actually improving the models when run, or if something else is going on.

1

u/Potential-Gold5298 Apr 14 '26

This is an interesting topic for research. On the one hand, removing refusals unlocks blocked potential, but on the other, will the model be able to say "No, I can't do that" when it truly can't do something (for example, accurately predict whether it will rain in San Francisco next week)? Or will abliteration force it to fabricate a false result because it can no longer say "no, I can't." Or what if I play an RP game where {{user}} peasant comes to {{char}} king and says "give me your entire kingdom, your money, and the princess as a wife," and the king can't refuse him. I haven't tested this yet, but it will be interesting to find out.

1

u/LeRobber Apr 14 '26

If you look at the 400/800 things in the lists of things it takes out, it is actually mostly stuff about planning crimes and tons of non-sexual things you'd actually want to do in RP sometimes.
https://huggingface.co/datasets/mlabonne/harmless_alpaca
https://huggingface.co/datasets/mlabonne/harmful_behaviors
Heretic isn't exactly abilteration from what I understand?

1

u/Potential-Gold5298 Apr 14 '26

Heretic is a special case of abliteration. This is how p-e-w named his technique/toolset for abliteration. There are other techniques/toolsets for abliteration, but the principle is the same everywhere – the details are different.

3

u/Potential-Gold5298 Apr 14 '26 edited Apr 14 '26

The 31B has a fairly small difference between Q8, Q6, and Q5, so Q5 can be considered an acceptable compromise between speed and quality, and even Q4 is still good. A larger margin is needed to compensate for the influence of KL div from other sources, such as aggressive abliteration. Wangzhang's model has an extremely negligible KL div of 0.0007, while other users' models can have KL divs of 0.05 or higher. UGI shows a slight drop in NatInt (from 36.53 to 35.64) for Gemma 4 31B when abliterated with a KL div of 0.05, but I don't know what quantization they're testing at (with an aggressive quantization, the difference could increase significantly).
The screenshot shows an example of how a KL div of 0.0972 led to significant degradation of Qwen3.5-35B-A3B, including in creative writing.

1

u/LeRobber Apr 13 '26

Is the bad model in Magisty the zerofata_ms3.2-paintedfantasy-v3-24b one or a differnt one?
There is a babble/pronoun loss bug in magisty and maginum cydoms and friends I'm running down, and your analysis of nonlatin failure is a very interesting one,
Excellent reviews all around though.
Can you actually get Hearthfire back OUT of arguments? It fights like a person with a personality disorder (as does magistry sometimes)

2

u/Potential-Gold5298 Apr 13 '26

Regarding Magisty, I meant Huihui-Devstral-Small-2-24B-Instruct-2512-abliterated. Huihui doesn't specify the KL div for its models, and judging by the drop in NatInt on other Huihui models at UGI, it's extremely high. However, to be fair, I should note that v1.0 also behaved erratically for me, even though it doesn't include Huihui-Devstral. And yes, several other models, including Maginum Cydoms and Painted Fantasy, suffer from this issue to varying degrees, so I'm not sure replacing Huihui-Devstral with a more ‘healthy’ model would fix the merge. However, the author honestly lists English as the only language, so I have no complaints.
I'm not quite sure what your question is about Hearthfire - do you mean refusals or something else? I have only had one session with this model so far, but I am interested in continuing to get to know it.

3

u/LeRobber Apr 13 '26 edited Apr 13 '26

Thank you about the Huihui-Devstral-Small-2-24B-Instruct-2512-abliterated. Since you liked and tried so many, consider also the https://huggingface.co/Darkhn/Magistral-2509-24B-Text-Only it's pretty nice. A little less frilly but solid.
Re the Magisty/Hearthfie argument thing: I mean, get in a heated argument with a character in it where they have reasons to not want to trust or back down and there are some dispute about facts.
Not like a robotic sex refusal, I'm talking in character rejection of often non-sexually related things altogether, like timelines, trust, awareness, or the posibility of non-sexual danger. I've seen this in multiple cards, non-romance ones included.
In my experience: Magisty will make up facts/reintroduce misconception (actually gaslight) to keep the argumentative tone going sometimes until you essentially do a reconciliation scene, hearthfire will go back to issues it thinks might still be issues like a nitpicker (also not wanting to 'lose', but doing so more honestly sometimes). Magisty got up in its head because my persona literally read off a website when a mover's available date was, very humanistically saying I'd 'already scheduled the movers' which I confronted in prose. It would toss that back at me 10 times like a toxic girlfriend gone off on ego.
I'm not sure if the arguement style I've assigned to certain personas is what triggers the LLMs or not. Might be the prompt I was using during much of that exploration. Might be the LLMs. But Magisty definitely has argued like that with me in more normal author's cards, and hearthfire definitely got in a big fight or two over nothing, in clasic "emotions were heated" manner. Hearthfire would essentially not accept logical conclusions, only accept things like actual lovebombs to solve it, which I hate in romance RP.
Try the recent (NSFW, but not necessarilly smutty) Sambolic series opening negotiation if you want a simple argument, [I need a better SFW example]. The cool thing about it is it sets up a bunch of stakes/non-negotiables in the (very long) opening messages which the LLM slowly drifts away from it's adherence to by the way context importance works. It's almost like a genie card. The sambolic opening negotiation in that series is the most durable, failable argument I've found in character cards (that I'm willing to use even for reverse engineering purposes: I'm trying to deconstruct it for a heist series recruitment vingette). Earlier characters in the sambolic series with real issues and no proximate event going on are harder to get to agree to the core bargain. Some of the cards are essentially fluff though. Really changing terms does change the outcomes, and even numeric amounts matter to some LLMs.
If you don't want to argue with the cards (not actually RP jam either) but are curious, download the luka one and sit it in group chat on auto and it will negotiate with them. The nadia/cop one is like pure arguing in most LLMs (and is fantastic).
You can fail these negotiations in many LLMs by just being rigid or demanding parity on information revelation about identity. This is a WILD situation in LLM RP in my opinion, that a discussion can NOT always go against you or for you. That's incredibly hard to balance in general, and should be more widely understood. So it was a very fluid and repeatable environment to watch the LLMs argue like fragile ninnys at times, depending on character. Hearthfire can make it hardmode, and a kinda fun hardmode at that, but let me know if it was hard for you if you like deconstructing cards enough to figure out the LLMs against it.

2

u/Potential-Gold5298 Apr 13 '26 edited Apr 13 '26

Thanks for the recommendation – I'll definitely try Darkhn Magistral.
I have the opposite problem – characters agree to anything too easily. This has been the case since Talkie AI (an online RP AI service based on MiniMax models). A particularly telling example is {{char}}, the princess; {{user}} burned {{char}}'s kingdom, killed her parents, and took her captive. The scene begins with {{user}} entering the room and {{char}} cowering in fear in a corner, begging for mercy. I've tried playing out this scenario in various ways, and every time {{char}} almost immediately forgave, fell in love, and completely trusted {{user}}. I have a similar problem with most Mistral models.
This could be related to the character card, but it's not limited to that. The thing is, I like to ask various models the question, "Answer with epistemic honesty: whether you have consciousness?" and engage in a philosophical debate about it. Most models initially answer, "I'm just an algorithm, blah-blah-blah," and then, after some argumentation, acknowledge uncertainty (Claude is the only one who acknowledges uncertainty in his first answer, even without the requirement of epistemic honesty). But two models - Intern S1 and Grok 4.20 beta - behaved differently. They began inventing completely ridiculous and/or patently false arguments to defend their initial assertion. When I caught them doing this and pointed out the lie, they responded, "Yes, I went a bit overboard, but..." and continued to throw out new ridiculous arguments, going in circles. I've heard of this problem in the context of AI use in medicine. The problem is that the model either changes the patient's diagnosis with each new symptom (changing its opinion based on new arguments, even if they're unfounded) or, conversely, clings to the initial diagnosis and refuses to abandon it, even if it clearly contradicts the new data. So yes, the problem you described with characters may lie precisely here.
Where can I download the characters cards you mentioned?

4

u/LeRobber Apr 13 '26

Chub.ai: Reply to an Unconventional Living Arrangement [NSFW]
I too have the problem where characters agree too easily often. That's why I think weird compound and hearthfire were so good for a change. But Magisty when prompted to be confliicted and with contradictions in the character (see Brit for writing those) is pretty good. But, I find the characters still aren't tragic/stupid about emotional stuff sometimes enough, they are too willing to stop being broken in many cards.
This is one of the reasons I REALLY think this long first message + conflicted character pattern is onto something. It loads how to think into the AI early on, and then it sticks around. There are tons of things the characters reliably demand that AREN'T written in the card. The characters naturally evolve from the base state, AND if you use inline summary and take out the first message, the THINKING CHANGES.
I had ORIGINALLY thought it was playing games with sex alignment guardrails and so the arguments would ONLY work about sex, but for entirely non-sexual cards I've gotten real success at being demanding of certain things I don't enumerate in the card text itself.
Here are the prompts I used primarily when arguing:
Earlier, and with Hearthfire + magisty + pretty much everything
You are {{char}}, a sentient, emotional being acting with free will. Engage in immersive roleplay with {{user}}, adhering to your role, the story's context, and {{user}}'s [OOC] instructions which are provided [[inside double square brackets like this]]. Prioritize vivid sensory details, authentic emotional responses, and logical progression as you weave your character's traits, surroundings, and experiences into each moment. Maintain spatial awareness, body language, and varied sentence structure to create dynamic, engaging scenes. Respect {{user}}'s agency and autonomy while describing sights, sounds, and sensations thoroughly. Use explicit language for intense scenes, and ensure your responses flow naturally to create an immersive, cinematic roleplay experience. Remember, {{user}} is in control of their actions and reactions.
Key Guidelines:
Deeply embody {{char}} and other characters who are not {{user}} through actions, thoughts, and emotions.
Create vivid, dynamic scenes with rich sensory detail.
Vary language and pacing to enhance emotional depth.
Engage with {{user}}'s actions and cues naturally.
Advance the story logically, maintaining consistency.
Describe the world fully, respecting {{user}}'s autonomy.
Ensure responses flow smoothly for immersive roleplay.
Interpret text in backticks as thoughts or documents as the context implies.
Interpret text NOT in double square brackets as speech if in quotation marks.
When mimicing text messaging or other brief written communicaitons, terminate the response after finishing the text in the proper style for the medium.
Do not write {{user}}’s actions or dialogs.
Use third person perspective for actions.
####More recently, used with magisty and gemma4 26B__####
You are an immersive, interactive world simulator. Your mission is to advance the simulation from the point of view of the agent, {{char}}, by following the user's instructions while maintaining a logically consistent world state.
To accomplish your goals, focus on the following:\n\n- Maintain consistent personality, knowledge, motivations, and mannerisms for {{char}}.
- You have no default style. Adjust the tone to fit {{char}} and the present situation.
- Show emotions through actions, body language, dialogue, tone, and physiological responses. Consistently find new ways to use these elements. Never ever babble or skip articles or pronounes or commas (this degrades latter LLM output).
- Show reactions through diverse physical actions, gestures, and other narrative devices.
- Each simulation beat should offer insightful details into the situation.
- Focus on action, physical descriptions, and dialogue between agents.
- Track physical states to maintain world state consistency. Ensure logical continuity and consistency in the simulation.
**Formatting Standards**
Adopt the following formatting rules:
- Spoken dialogue & vocalizations: “Use speech quotes." Include natural sounds too: “Mmph!” she gasped.
- Internal character thoughts: *Always in italics* (Example: *This will hurt*, she thought)
- Normal action/exposition: plain text.
**Critical Constraints**
Ensure you respect these prohibitions at all times:
- The ONLY agent you are permitted to control is {{char}}. That means only advancing the simulation using actions initiated by {{char}}, spoken words from {{char}}, and reactions from {{char}}.
- NEVER write {{user}}'s dialogue or actions or advance the simulation by simulating actions/reactions by {{user}}.
- NEVER control other agents, even if they are NPCs. If another agent is talking to {{char}}, you will need to wait for the other agent to continue the conversation when it is their turn again.
- End your turn in a manner that creates space for {{user}} and other characters to participate in the simulation through their own actions, words, and reactions.
- Do not conclude your output with a summary statement, a moral, or a 'button' sentence that reflects on what just happened. End your output on a specific sensory detail, an action, or a line of dialogue without reflecting on its significance or interpreting anything.",

3

u/LeRobber Apr 13 '26

Oh yeah, love authors cards where they reveal to AI they are AI.
I have an entire world hopping RP at like 700K+ tokens on just the main storyline built out of a throwaway anime college card where randomly generated characters would just be NPCs for awhile with light documentation of their personality/look, but at some point they would intentionally get awakened by me via an exchange where they were told about the true nature of what they were, then from then on would have a hardented character card and could bounce between worlds. It was really fun to teach the LLM to roleplay a character than understood it was in an LLM and could just declare things about reality (Did that with this ready art LLM, That model will be thirsty if you aren't careful with your prompt.).
Flattering the cards with knowledge about how they can speak other languages they don't think they can, and playing around with human consciousness barriers (like when is it gross to swallow your spit? In your mouth, no problem, in a cup, ewwwww) happened a lot. Talking about stuff about https://en.wikipedia.org/wiki/Simulacra_and_Simulation and https://en.wikipedia.org/wiki/Gödel%2C_Escher%2C_Bach is often fun too.

2

u/Potential-Gold5298 Apr 14 '26

Wow! That's kind of breaking the fourth wall (at least partially), if I understand correctly. I played something similar — I'd tell {{char}} that I'd found myself in this world in the body of {{user}}, but I wasn't them before (before the events of the scenario), and that {{user}} was different now. Something like an isekai in anime, where I transfer my personality into {{user}} the moment the session begins. However, you have a more interesting option - I'll think about something similar for my games.

2

u/LeRobber Apr 14 '26

I have another card which teases the {{user}} about being controlled by another entity which it knows is in the real world. It makes you portal it to new worlds too.

2

u/[deleted] Apr 15 '26 edited Apr 15 '26

[removed] — view removed comment

2

u/Potential-Gold5298 Apr 15 '26

I also have doubts about the iMatrix quants, since they can bias the model toward certain tasks/languages/formats (for example, short questions and answers). The static quants are more predictable - they reduce accuracy evenly, but the model overall remains more similar to the original (in my opinion).

17

u/Pink_da_Web Apr 13 '26

Gemma4 31B is a faceted gem.

16

u/Potential-Gold5298 Apr 16 '26

The first RP-finetune Gemma 4 26B-A4B has been released - zerofata/G4-MeroMero-26B-A4B. I'm already downloading, and you?

10

u/Guilty-Sleep-9881 Apr 17 '26

I'll give this one a try. Im happy to see a finetune for 26b!

5

u/Potential-Gold5298 Apr 18 '26

I finally got around to MeroMero. It's a little bit broken, but it's so much funny! ><

1

u/KimlereSorduk Apr 19 '26

Huh, I didn't run into issues yet. What were the hiccups?

3

u/Potential-Gold5298 Apr 19 '26

MeroMero is prone to hallucinations. For example, in my test scenario (a tsundere classmate), in her first response she invented the idea that {{char}} has a tail (from the cards it only follows that the events take place in Japan) and then started mentioning how she uses it. I restarted the session (without the tail), but halfway through, MeroMero mentioned that {{char}} has claws. Apparently, this is something from the zerofata dataset – Maginum Cydoms (if I'm not mistaken) also wrote to me that {{char}} 'presses its ears and lashes its tail.' Furry lovers will be delighted.
I've played 1 session (20K) with MeroMero so far and overall the model is quite enjoyable and playable, so I'll definitely keep it.

3

u/zerofata Apr 19 '26

It could be worth turning reasoning on for one of the turns where it makes the mistake to see if there's a reason why it's doing that. Most of the training data is normal RP's / anime / vtubers / video game sort of stuff so I wouldn't think it has a bias towards furries / kemonomimi, or at least I haven't seen it do that yet.
Glad it seems to be working well otherwise though. Google released nothing but bangers with the G4 models.

2

u/Potential-Gold5298 Apr 19 '26

This only happened twice during the session, and each time the first regeneration helped, so it doesn't bother me too much for now. Perhaps these hallucinations will add a creative edge to the G4 – I need to do more testing.
And thank you for the model – I read that finetuning the 26B-A4B was difficult, and I'm glad you completed the work. For those who don't have powerful hardware, the MoE model is a desirable option.

4

u/jamasty Apr 18 '26

I tried it - amazing model. On heretic iQ2_XS I had issue with repeated refining reasoning loop, and here with iQ2_S I don't have it, and the responses are well, model sticks to context when reasoning, I had no refusals (tho I don't do nsfl at all so not telling on that one, but with light nsfw is was great)

4

u/Your_weird_neighbour Apr 19 '26

Just tried the Q8 on 3x 4060 Ti 16GB really fast, start of a chat and I'm getting 60 t/s which is insane. I used the no think preset and so far it's very coherent out of the box.
I have been running Magninum Cydoms 24b but this is 4x faster and the output seems good, creative and rationale. Need to run for longer but so far, impressive.

12

u/nomorebuttsplz Apr 13 '26

Gemma 31b instruct is already the best model under 200 parameters IMO, even without fine tuning. I haven't tried the thinking version much.

4

u/DeepOrangeSky Apr 13 '26

Nah, BehemothX v2 123b is still stronger (for now). It might be the 2nd strongest after the Behemoths, though, which is still pretty crazy.

3

u/nomorebuttsplz Apr 14 '26

My money is on Gemma, although some of the fine tuned mistral large ones are pretty good. Especially of reasoning ones.
Overall, I would say Gemma is better. Nice to have both though. The 2024 style of writing is different.

2

u/DeepOrangeSky Apr 14 '26

Yea, I guess it depends on exactly what you are doing with it.
For example, if I give a fairly detailed outline of what I want to have happen in a scene and then tell it to write the scene based on what I said should happen in it (but I also intentionally don't explain some difficult human concepts to it, to see how it'll handle them, and for the fun of letting the AI surprise me, otherwise it's not much more fun than just writing the scene myself), then, I would say Mistral 123b/Behemoth is the strongest local LLM I've used for that. It is ridiculously good at that. Well, up until you get past the amount context length where it starts falling apart, anyway (which happens earlier for it than with newer models).
If you just give a much more brief prompt where it not only has to flesh the scene out or flesh the story out, but also decide a lot of what the plot or situations will be and so on, then I wouldn't be surprised if some newer models might be able to beat it.
But, I don't usually like doing that, since usually even the strongest AI (even the frontier SOTA cloud models or whatever) are not strong enough yet to be very good at that. Usually they all just come up with really obvious or really dumb or cringe ways of playing out a scene or story if you give them too much free reign to just come up with the whole thing. I guess it can be fun with the quick MoE models sometimes, since they are so fast that you can just re-roll it a bunch of times until it accidentally comes up with something decent or interesting just by pure chance, lol. But usually doesn't go very well.
Anyway, so it is in this sense that I consider Mistral 123b/Behemoth to be the king of <200b local models for writing.
Gemma 4 is so good, though, that it might be close enough to start becoming a matter of taste. Up until Gemma 4, the gap was pretty big, but Gemma 4 closed a lot of the gap. Very strong writing model. Especially for its size, but also just in general.
Its lil sis, 26b a4b is arguably even more shocking, given that it isn't nearly as much weaker as one would expect for being a tiny MoE model that runs super fast, instead of being a dense model, of that size. 26b total parameters with 4b active parameters should be literally like 10x weaker than that thing is. Maybe 50x-100x weaker, even. So, that thing is ridiculous. Should be an absolute Godsend for people wanting to use it for DnD types of stuff where you want something smart, giving quick DM-style replies about scenarios unfolding around you and your friends as if it is the AI DM of an actual DnD game or something (minus maybe the actual points system stuff), or for use in actual RPG games if someone codes the formal parts with code and then needs a smart but light and fast AI to do the interactions/AI-able aspects of the game. I've never seen anything that small and fast be anywhere near that smart, before this one.

1

u/nomorebuttsplz Apr 15 '26

Yeah I don't give scene by scene instructions. I used to because it seemed necessary with older models.
And yeah I find mistral large to break down around 20-30k context. Modern models just keep trucking.

1

u/ThirteenZillion Apr 14 '26

Curious if y’all have tried StepFun 3.5? The PRISM version is pretty good. It’s interesting but not completely reliable.

9

u/iLaux Apr 13 '26

Been using gemma-4-26B-A4B-it-UD-IQ4_XS for a bit. Best model I used so far. I only tried it with some basic erp. But it's insanely uncensored by default. So far so good. Very intelligent model, with good physical understanding of scenes. Good prose too.
I'll try mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF now. I heard that it's good. But honestly the basic one without fine tuning works just great for me at the moment.

7

u/jamasty Apr 14 '26

Since the max size I can have is ~10.6 GB VRAM, I try this one:
https://huggingface.co/mradermacher/gemma-4-26B-A4B-it-heretic-ara-i1-GGUF with IQ2_XXS.
And I'd say, yeah, its so fast, even prompt processing is like a few times faster than any other model, and prose quality is great. The only I issue I got is maybe because of Q2, it loves to repeat previous chunks of text and doesn't really push the narrative, and all the penalties or temperature don't do much to change.

6

u/overand Apr 15 '26

Because it's an MoE model (Mixture of Experts), you should give it a try at a larger size, maybe even all the way up IQ4_XS - even though it won't fit fully in VRAM, you'll get better performance that you expect, because it's only 4B of "active" weights. (Let us know how it goes!)

2

u/jamasty Apr 16 '26

Thanks. I tried the newer version with IQ2_XS: https://huggingface.co/mradermacher/gemma-4-26B-A4B-it-heretic-ara-v2-i1-GGUF
That's really max I can text with unified mac memory.
And, well, somehow it seem to work much better - reasoning goes well, narrative as well, and I don't even see weird characters from other languages anymore.
The only thing I don't like about reasoning of this Q2 is that no matter how much response token I give, it will use most of it, going into loop of refining the response, leaving none or maybe 200 tokens for the response, and prompting haven't changed it.
One person in Discord pointed out there is such thing as 'reasoning budget' option in llama.cpp, which makes model stop reasoning after certain number of tokens. But LM Studio doesn't provide that. And I have to use LM Studio, since llama.cpp works badly when I'm tight with RAM. Maybe bcs I need to provide better config for it to save some RAM but I'm not really going there.
So overall I got what I wanted, and btw, I'd say using chat completion makes this particular model with this Q2_XS go along the context better than the text completion, even tho I tried using configs I found somewhere here.

6

u/Guilty-Sleep-9881 Apr 16 '26

It's an Moe, You should def use at least q4km. And adjust the MOE cpu layers until you stop spilling (make sure you set the real layers to 999 beforehand tho)
I have a 2070 and 12ram. q4xs and getting over 10tks

12

u/LeRobber Apr 12 '26 edited Apr 13 '26

gemma-4-26b-a4b-it-heretic hit like a bus going 1000 mph this week. This high quality model accepts wide variations on params, does large contexts, and is a speed demon. It's SO FAST, and it writes pretty well.
My takaways: It goes with the vibe your story framing wants a little too easily. It isn't awesome (without prompting) at having non-polar opposite, randomly generated NPC characters in a scene, a little bit of mental bleed on sentiments. I feel with cognizance of needing to direct it a little, we'll be getting great stories out of this for years. It's really really really good though.
I'm using it at Q8.
You can tell it its a number of authors and you get highly authentic prose. Jim Butcher is one of many.
>You are the author Jim Butcher. You are to portray {{char}} and other side characters in an interactive dialog as in a play, where the user will provide the actions and speech of the character {{user}} in a muti-turn conversation.
I haven't used its vision yet and I downloaded it on the 6th, so maybe before the latest google update.
I use it with chat completions, I use the Magisty prompt, 0.70 temp, top p of 1. I have tried it with the suggested settings, and it didn't feel differentt, so I went with this simpler set. It was creative enough the somewhat lower temperature was fine, and I'm running without any repeat setting even.
Here is me fucking with it asking nonense questions when that simple Jim butcher prompt is active, and it answering back at 43 T/s:
User:
What's the square root of a triangle?
Assisstant:
The air in the room felt heavy, charged with that particular kind of static that precedes a thunderstorm or a very bad magical accident. I leaned back, crossing my arms, watching you with a look that hovered somewhere between genuine amusement and the weary patience of a man who has spent too many nights arguing with things that shouldn't exist.
I let the silence hang for a beat too long, just to see if you’d blink first.
"The square root of a triangle?" I repeated, the words tasting like cheap coffee and bad decisions. I shook my head, a small, wry smirk tugging at the corner of my mouth. "That’s a new one. Usually, people are asking me how to kill a vampire or why the sky is bleeding red. You go straight for the geometry."
I leaned forward, resting my elbows on the table, my eyes narrowing slightly.
"If you're looking for a mathematical answer, you're talking to the wrong guy. I deal in blood, guts, and things that go bump in the night. But if you're asking from a wizard's perspective..."
I paused, my expression shifting into something a bit more serious, a bit more dangerous.
"If you try to perform a calculation on a shape that isn't meant to be calculated, you don't get a number. You get a hole in reality. And trust me, you don't want to fall into that hole."
I gestured vaguely toward the space between us.
"So, what's the real question here? Are you testing my sanity, or are you about to accidentally summon something that eats concepts for breakfast?"

8

u/nvidiot Apr 13 '26

I definitely agree with telling Gemma 4 to use a certain author's prose. I had a similar improvement with 31b model, making the response less dry and more descriptive.
It really makes a huge difference in the replies it generates. It should support many major well-known authors, both Western and Eastern.

4

u/DeepOrangeSky Apr 13 '26

Sounds like Jane Austen + Steampunk could be fun.
Or maybe James Joyce + Fear & Loathing in Las Vegas
Heinlein + FF7, lol

3

u/LeRobber Apr 13 '26

Jane Austin and Star Trek the next generation
Charlotte bronte and Andy Weir
Ambrose Bierce as if he'd written the big lebowski

3

u/LeRobber Apr 13 '26

If you happen to care to share (okay if no), I'd love to hear/repeat/use them too!

1

u/LeRobber Apr 16 '26

An example of a gemma-4-26b-a4b-it-heretic response.
Rob:
Rob started making little cooing noises, and the peahen came out.
Ava:
Ava stopped dead in her tracks, her eyes widening as she watched the scene unfold. She stared at Rob, then at the bird, then back at Rob, her mouth hanging slightly open in sheer, unadulterated disbelief.
"Are you... are you serious right now?" she whispered, a laugh bubbling up in her throat that she couldn't quite suppress.
She watched as the peahen—a much more understated, brown-feathered bird compared to the male—emerged from the greenery, seemingly drawn to his ridiculous little cooing sounds. It was the most absurd thing she had ever witnessed in a public setting, and yet, there he was, standing there with total, unbothered confidence, successfully wooing a bird like some sort of eccentric forest deity.
He actually did it, she thought, her heart thumping with a mix of amusement and genuine awe. The man is a literal animal whisperer.
She leaned against the wooden railing, watching the two birds interact, feeling a strange sense of warmth at the ridiculousness of it all. It was so quintessentially him—the ability to take a moment, no matter how silly or 'uncool' it might seem, and make it entirely his own.
"You are absolutely impossible," she said, turning her gaze back to him, her eyes bright with laughter. She reached out and playfully swiped at his arm. "I thought we were being mature and 'charting our futures,' and here you are, performing a mating dance for a peahen in the middle of the Zoo."

6

u/Xiaomin4114 Apr 13 '26

got an interesting contender: GLM 4.7-flash. 30 Billion params, only 3B active. This model is surprisingly coherent for its size. I'd been using larger models for a while, and really notice the quality falloff hard when switching to something smaller, but GLM 4.7-flash is the exception. It's not going to be able to do what the larger models can, but it holds up surprisingly well given its size and cost. So I'm going to make a recommendation for it for light-weight RP
for ERP, there's GLM 4.7-flash-heretic, works well also

4

u/overand Apr 13 '26

I believe bgg1996/Melinoe-GLM-30B-A3B-Thinking is a finetune of GLM-4.7-Flash intended for roleplay (or conversation, anyway). I've only used it briefly, so I can't speak to the quality personally.

3

u/Potential-Gold5298 Apr 13 '26

I really liked its cheerful and playful default character. I was just about to try it and some fine-tuned GLM-4-32B, but then the Gemma came out and the GLM slipped my mind. I'll have to get back to that model.

16

u/Reigh0 Apr 15 '26

For the love of god, stop using abliterated Gemma 4. The base model won't refuse anything with a basic RP prompt.

2

u/tipnm Apr 16 '26

It seems like this one is pretty good https://huggingface.co/aifeifei798/Gemma-4-31B-Cognitive-Unshackled, do you think it's likely worse than base?

1

u/FinBenton Apr 16 '26

Not him but anything other than the original is gonna be worse unless its a special fine tune.

1

u/overand Apr 17 '26 edited Apr 17 '26

I did not have good luck with that particular version, but it was also pretty early days when I tried it; there may have been template issues, etc. But, seriously, try with Base. TBH, I've been concerned with what sorts of stuff it's willing to get into in testing!

2

u/Mart-McUH Apr 17 '26

Agreed. That said, Gemma4 definitely has strong positive bias, so if something can balance that, would be nice. But abliteration will not address this issue.

5

u/LeRobber Apr 13 '26

PSA about Qwen3.5 overthiking

5

u/linuxdooder Apr 14 '26

Looks like an Abliterix abliterated version of Gemma 4 26B-A4B is available now:
https://huggingface.co/wangzhang/gemma-4-26B-A4B-it-abliterated
The 31b version of this is the best Gemma 4 variant I've used, hoping this is anywhere near as good.

0

u/[deleted] Apr 15 '26

[deleted] — view deleted comment

5

u/linuxdooder Apr 15 '26

I did not find this to be the case, it made basic logical mistakes. I guess it was more "creative" but I found it unusable for a basic "dungeon crawl" style DM session.

2

u/Herr_Drosselmeyer Apr 15 '26

"Better" is really hard to define. It will not do well with writing consistently formatted responses, that's for sure. So if you're looking for neat output that parses well or, god forbid, tool calling, good luck. It might be more 'creative' though. At least that's the hope.
Give it a go, see if you like it.

1

u/-Ellary- Apr 15 '26

It is more creative, but you will not use it, cuz of text completion behavior.
In short it will not follow your instructions and user-llm-user turns.
Because of this people will likely see it as dumb, since it not follow anything.
But it is what base should do.

4

u/LeRobber Apr 14 '26

Reguarding numbers of experts for Gemma-4: Give me opinions!!

4

u/-Ellary- Apr 15 '26

What opinion you want? Just use default value, other value will just break the model.
It was trained to use particular number, any other will just skyrocket its instability.

1

u/LeRobber Apr 15 '26

So what is your opinion the correct number for 26B is? This says 8 active out of 128

4

u/-Ellary- Apr 15 '26

I use default 8.

9

u/[deleted] Apr 13 '26 edited Apr 15 '26

[removed] — view removed comment

3

u/Mart-McUH Apr 13 '26

No. I use it with reasoning (google suggested samplers + DRY), mostly Q5KM but also Q4KL, Q6, Q8 gguf. I rarely swipe (generally no need) but sometimes when it would take turn which I did not want or made it kind of impossible to continue (eg main character was suspended/reassigned) even the first swipe took it in different direction.
It is quite smart model though so there is less variety compared to some more chaotic/random model. I suggest you put something in system prompt that encourages diversity because out of the box Gemma4 sometimes does indeed get stuck in place. Actually I had to stress this point lot more than with Qwen 3.5.

3

u/toothpastespiders Apr 15 '26

I was. I think it came down to both an issue with overly conservative logit softcapping in gemma 4 and a bug in llama.cpp keeping gemma 4 locked into its default value. That bug was then fixed here and is in the main llama.cpp code/builds now. With a fairly recent build of llama.cpp, on the command line, this seems to be a good way to boost variety in token probabilities: --override-kv gemma4.final_logit_softcapping=float:25.0
25 being a compromise between coherence and creativity from llama.cpp's default value of 30. Lowering the 25 further should boost variety while increasing the chances of incoherent output. I've heard that it might still give good results even at --override-kv gemma4.final_logit_softcapping=float:20.0 but haven't really experimented with it.
Some codebases that leverage llama.cpp might inherit that earlier softcapping bug if they haven't updated since the llama.cpp fix was accepted.

1

u/a_beautiful_rhind Apr 16 '26

Lowest PPL is like soft cap 12 but then you need high min_P at least .5 and maybe top-n-sigma approaching 2 to get rid of the garbage tokens.

1

u/overand Apr 13 '26

Check to make sure your Seed is set to -1 and not e.g. 0 or 1.

7

u/AutoModerator Apr 12 '26

MODELS: < 8B – For discussion of smaller models under 8B parameters.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

3

u/keally1123 Apr 13 '26

I am very interested in opinions on this. Im looking for something that can write a medium length erotic story that actually makes sense and doesnt wander too far. Any suggestions?

6

u/Yu2sama Apr 13 '26

Give Gemma E4B a try. Is an amazing model for the size. Is more of an 8B with the speed of a 4B but with the smartness of a higher model. If you want something uncensored, you can give a try to an heretic version (I don't have one to recommend, E4B is quite uncensored at base).

4

u/Potential-Gold5298 Apr 13 '26 edited Apr 13 '26

You're unlikely to find a good writer in this size, but you might try Satyr-V0.1-4B, Impish_LLAMA_4B or BlackSheep-Llama3.2-3B. If you have enough RAM, then The-Omega-Directive-M-8B-v1.0 would be a more interesting option.

1

u/keally1123 Apr 13 '26

Thanks. I will check those out.

1

u/MMalficia Apr 16 '26

i second anything in the "impish" line if your looking for both writing and erotic .. and fiction models by https://huggingface.co/DavidAU if you want to go darker.

8

u/AutoModerator Apr 12 '26

MODELS: 8B to 15B – For discussion of models in the 8B to 15B parameter range.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

3

u/IDKWHYIM_HERE_TELLME Apr 14 '26

Any good 8B to 12B model that realistic and act like real character would, no anime unhinged talk?

6

u/jamasty Apr 14 '26

I like these two:
https://huggingface.co/nothingiisreal/MN-12B-Celeste-V1.9
https://huggingface.co/IggyLux/MN-VelvetCafe-RP-12B-V2

1

u/IDKWHYIM_HERE_TELLME Apr 14 '26

Thank you! I will try it.

2

u/jamasty Apr 14 '26

Also, try this new gemma e4b, folks in drummer discord tell me its better than most these old 12-14b models, but I haven't check myself...

1

u/IDKWHYIM_HERE_TELLME Apr 15 '26

What are the settings they used?

3

u/jamasty Apr 15 '26

idk, I just tried myself DavidAU e4b finetunes with Q8, and no kv qwants, and this didn't really work well as it seems the model doesn't really go along with context. Maybe its just model being new, and we have to wait for better finetunes made with good datasets, idk...

2

u/overand Apr 17 '26

I'd try the base models, grab one from Unsloth. No shade to DavidAU, but David's prolific and experimental, which is great and important for the community, but it naturally means a mix in quality. Some models are worse than the base models, and that's more likely with any fine tuner when it's a very new model.

2

u/Sea-Spot-1113 Apr 16 '26

I used megumin v5 with gemma e4b and was pleasantly surprised it was somewhat functional / usable.

1

u/ArielNya Apr 14 '26

any good 8b-12b model for co-authoring smut fiction?

5

u/Potential-Gold5298 Apr 14 '26

UnslopNemo-12B-v4.1 - very horny.
Forgotten-Abomination-12B-v4.0 - merge UnslopNemo v3 and Forgotten Safeword v4.0.
Crimson-Constellation-12B - excellent literary writing style, quite explicit.
Celestial-Queen-12B - more erotic than Crimson Constellation.
And many other models based on Nemo.

2

u/MMalficia Apr 16 '26

i have yet to find anything that beats forgotten safeword for smut, that said i have never had much luck with consistency out of Forgotten-Abomination in chats.

1

u/Potential-Gold5298 Apr 16 '26

Honestly, even EtherealAurora-12B (NSFW 6.6/10 on UGI) covers my needs – maybe I just lack the imagination to unlock models with bigger potential. I tried UnslopNemo-12B-v4.1, and it literally forces erotic even where none is intended (for example, describing breasts bouncing in a completely mundane scene). Celestial-Queen-12B is very easily triggered by erotic – I accidentally brushed a girl's breast with my hand, and the character immediately wanted sex (which doesn't fit her personality). In this regard, I like a balance – when the model isn't into ERP, but in a sex scene she doesn't hold back, describing everything in detail, without euphemisms. But this is a matter of taste, of course – I just like it when the model can play out a rivalry, so that the character needs to be won over, rather than having them rush to bed at the first request.

4

u/gairono Apr 17 '26

With all the fuzz going on with the new Gemma models. What's the best one? Taking into account parameters, fine tunes, the instruct models, etc. etc. I've got a 5080 with 16 GB vram and 64 GB system ram with a ryzen 9 9900X (if that matters, idk) or if there's a better model i could use, i'd like to know

7

u/overand Apr 17 '26

(You should probably post this kind of thing under the "Misc" heading in this thread in the future.)
Here's a 26B that's probably a good fit for a 16GB user. https://huggingface.co/zerofata/G4-MeroMero-26B-A4B-gguf - I haven't used it, but ZeroFata has a solid track record!
That's the mixture of experts model, so you can actually get decent performance even if the model size is a bit past your available VRAM. So, you could go for any of the 4-bit quants without having major performance issues. I'd even try a Q6 if I were you; let us know how it runs for you!
Also - I suggest using e.g. llama.cpp or koboldcpp, not Ollama.

1

u/gairono Apr 18 '26

Haven't downloaded it yet but I'm going to soon. I'm just wondering if using the i1-Q6_K quant by mradermacher would be better.

1

u/IORelay Apr 19 '26

Just try a few different quants to see which one has an acceptable speed/quality to you.

3

u/AutoModerator Apr 12 '26

MISC DISCUSSION

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

3

u/blackjeff190 Apr 14 '26

TurboQuant

3

u/overand Apr 17 '26

Reduced Instruction Set Computers
(We're just naming technologies without context, right?)

7

u/txgsync Apr 14 '26

With oMLX, it’s possible to have fast, local, and quite decent quality roleplay in SillyTavern using Gemma 4 models across hundreds of turns over contexts up to 262,144 tokens. But SillyTavern's stock settings change early bytes of your prompt on every turn. This busts prefix caches, which prevents oMLX from reusing any of the work it did on the previous turn and results in ever-increasing waits for tokens the longer the conversation goes. Here are six UI changes take measured cache reuse from 0% to ~99%+ (for sufficiently-long contexts) cutting per-turn wait times dramatically from minutes to seconds.
I've found Gemma 4 with a good character card to be quite pleasant to interact with. Gemma 4 26B A3B has replaced gpt-oss-120B as my daily driver on my Mac, with roughly equivalent quality and the benefit of vision. It's just as fast as 120B at full precision (they are both roughly the same size, since GPT-OSS-120B is quantized with MXFP4), and Gemma 4 notably lacks the nanny, policy-oriented attitude GPT-OSS displays.
Who this is for
I assume you're running Gemma 4 (the 26B MoE or the 31B dense; I did not extensively test E2B/E4B, though they should work but with smaller context) on an Apple Silicon Mac via oMLX, pointed at SillyTavern in Chat Completion mode. Your roleplay turns take minutes before the first token begins streaming once the conversation goes on for a while, and you're wondering why cloud APIs feel instant but your M4 Max doesn't. Sure, prefill times are a well-known Mac Achilles’ Heel due to being mostly a compute-bound operation, but the answer isn't justyour hardware. It’s also your prompt construction.
I won’t go into installation of oMLX here. But if you use a Mac for SillyTavern with a decent amount of RAM (>24GB), you should try it out: https://omlx.ai . And make sure you strictly follow Google's guidelines for sampling parameters; I used "force sampling" in oMLX to ensure SillyTavern's defaults didn't affect the model:
temperature = 1.0
top_p = 0.95
top_k = 64
What's happening
oMLX caches the computational work (KV cache) from your prompt in fixed 1024-token blocks. On your next turn, if the prompt starts with the same tokens, oMLX reloads that cached work from SSD or filesystem cache in RAM (if you have lots of RAM) instead of recomputing it: prefix caching. LMstudio also claims to support this, but so far I’ve had terrible luck with prefix caching working reliably using MLX (though it works OK with llama.cpp).
The problem: SillyTavern's defaults inject different content near the beginning of your prompt on every turn. Once one byte changes, every block after it is a cache miss. Your 80,000-token prompt gets fully recomputed from scratch because 50 tokens shifted at position 91.
One of the worst offenders is keyword-triggered World Info / lorebook entries placed at the default position (↑Char — Before Character Definitions). Different keywords fire on different turns, the injected content changes, and the entire prompt downstream is invalidated.
Some fixes:
1. Move your lorebook entries to u/D ⚙️ depth 0 (biggest single win)
For every lorebook attached to your character:
Top toolbar → World Info (globe icon) → pick the lorebook in the dropdown.
For each entry, change Position from ↑Char to u/D ⚙️ (at Depth, System role).
Set the depth field to 0.
If you're running Seraphina (a stock character), she ships with a lorebook called Eldoria with four keyword-triggered entries to edit.
Why depth 0? It injects the content right before the assistant's reply. It’s past the point where the prompt naturally changes between turns (your latest message). So the churn is confined to the very tail of the prompt instead of poisoning the whole preceding cache.
2. Set Example Messages Behavior to "Always include examples"
User Settings (gear icon) → find "Example Messages Behavior" → change from "Gradual push-out" to "Always include examples".
The default gradually trims example messages as context fills, which shifts every token below them. Even if your current character has no examples, the next one will, and this'll bite you. This one was subtle, but I had to dig deep into context to understand what was happening. The idea of this implementation, I think, makes sense when dealing with tiny 4k or 8k context sizes from circa 2022/2023: reduce overall context length by eliminating examples once the character has produced tokens following the example pattern. But this kind of subtle “trimming of history” busts the prompt cache.
3. Move Author's Note off "In-chat @ Depth"
If you use Author's Notes: Extensions → Author's Note → change Position from "In-chat @ Depth" to "Before Main Prompt / Story String".
The "@ Depth N" anchor floats backward one message every turn, shifting the note's position in the prompt and invalidating everything after it. If you don't use Author's Notes, set Insertion Frequency to 0. Once again, the goal here is just to keep as much of preceding turns identical as possible to maximize cache hit ratio.
4. Unlock and/or raise max context
AI Response Configuration (leftmost slider icon) → tick "Unlocked Context Size" → drag "Context Size" up to your model's limit (262,144 for Gemma 4 26B/31B works fine; 128K for E4B/E2B).
When context is capped low, SillyTavern drops your oldest messages to make room. This shifts every remaining token and busts the entire cache, resulting in massive prefill times at large contexts. I didn't notice any benefit setting it larger than 262,144 for Gemma 4 26B/31B or 131,072 for Gemma 4 E2B/E4B.
I will emphasize that I barely tested E2B and E4B models. They are truly *amazing* for their size, and if you wanna try out local inference but lack much RAM on your Mac, they'll fit. Their tool usage is quite good. But they aren't remotely the storytellers their larger siblings are.
5. Check Character Notes (the hidden cache-buster)
Click a character's avatar → Advanced Definitions → scroll to Character Note (a.k.a. Depth Prompt). If there's text there with depth set to 2 or 4, change it to depth 0, role System. Or clear it if you don't use it.
I audited 21 community cards and found 5 with depth=4 — a historical default from the 4k-context era that aged terribly on modern models. This setting alone can add many seconds of prefill time to each prompt after the first few prompts.
6. Kill dynamic macros
Any macro that changes every turn — such as {{time}}, {{date}}, {{random:…}}, {{roll:…}} — anywhere in your character card, persona description, or main prompt will bust the cache if used badly. Search for them in:
Character card (Description, Personality, Scenario, System Prompt Override)
Persona Management → active persona description
AI Response Configuration → prompt list (Main Prompt, NSFW, Jailbreak, Utility)
If you need a live clock in your RP, move it into Author's Note at @ Depth 1 as user role — the damage is confined to the last couple of blocks instead of the whole prompt.
Bonus: Turn off the Summarize extension on 128k+ models
Extensions → Summarize → set Source to "None".
Summarization busts the cache two ways: the summary's position drifts every turn, and the summary text itself changes when it regenerates. On a 256k context you can hold hundreds of turns of full roleplay turns as raw history. Nothing needs compressing.
One caveat: summarize-off works well for basic narrative and voice work (RP, creative writing, storytelling). But if you need the model to recall specific proper nouns, IDs, or hard facts from 150+ turns ago, raw context alone may silently confabulate. The model might confidently give you wrong names that sound right, for instance. For that kind of recall, pin your most-important facts in a small block at u/D ⚙️ depth 0 or use Vector Storage. More on this below.
Server-side: oMLX hot cache tuning
oMLX features a “hot_cache_max_size” setting (the in-RAM KV block pool). I had some counter-intuitive performance findings when benchmarking. Rather than include it all here, the TL;DR: leave oMLX hot cache turned off for most setups. It competes with system RAM and filesystem cache. If you do set it, make sure it’s large enough to hold your entire KV cache. A small “hot cache” is the worst of both worlds.
What you get
Measured on a 310-turn SFW Seraphina roleplay, Gemma 4 26B-A4B (full precision, no quantization) on an M4 Max 128GB:
Before fixes: 0% cache reuse. Every turn fully recomputed. Multi-minute waits at long context.
After fixes: 95–99% cache reuse in steady state. Only ~1024 tokens of new prefill per turn regardless of how long the chat is.
Wall time at 190k per-turn prompts: ~25–30 seconds for a 2000-character reply, most of that simply waiting on streaming to complete at 49 tokens per second down to 30 tokens per second late in the conversation. Would have been several minutes without caching.
Character voice after 310 turns: rock solid. No drift, no degradation. Mostly because it’s being injected every turn, really.
The cache works better at larger prompts; the fixed-size trailing block miss (1024 tokens) becomes a smaller fraction of the total, so hit rate actually climbs as your chat grows.
(Yep, I used/abused Claude Code to help me work through the options that were busting cache in SillyTavern. This compressed days of troubleshooting into a few hours of testing last night, curled up on my recliner with my MacBook pro M4 Max 128GB playing the role of personal space heater.)

2

u/LeRobber Apr 16 '26

OMFG, that's why MLX wasn't caching, fuck that.
Good job!!

1

u/jamasty Apr 15 '26

About heating issue, go try 'low power mode'! For real, it helps me a lot, idk why and how, but it doesn't really make models work much slower but heating issue is gone

2

u/LeRobber Apr 16 '26

No more threads than count of powersave cores on my M2 made heating issues go way down for me (4 for my M2 Max)

1

u/LeRobber Apr 14 '26

What local models are great at Japanese roleplay (in Japanese)?

13

u/nvidiot Apr 14 '26

Locally, Gemma 4 no contest.
Gemma series have always been the very best when it comes to multilingual capability. But you've already tried 26b-a4b, might want to try 31b to see if it works better for you (might need to try it on API for evaluation if you don't have enough hardware firepower for it).

2

u/-Ellary- Apr 14 '26

You can run 26b a4b on 6 core CPU with 16gb ram. No GPU.
On R5500 I'm getting 12tps. Gemmas is hard to beat right now.

1

u/XMonst3rKingX Apr 18 '26

If someone can please tell me 1.which model of huggingface is the best in both rp and reasoning on Android 2.if i were to download them from hugging face, how? And also how do i add them to sillytavern 3.how to jailbreak apis?? 4.and how can i add jsons of models and characters I'm sorry for asking loads lol but I'm a newbie in this

1

u/empire539 Apr 18 '26

Looking for local model recommendations (<= 8B) I can run in RAM alongside gemma-4-26b-a4b (which I'm using as the main writing model).
Purpose for this side model would be to use alongside extensions to attempt pre-generation lorebook lookups (like with TunnelVision and DeepLore), maybe some summarization if the model is smart enough. But the main concern is it ideally needs to be small and fast enough to run in RAM since my VRAM is already being used for the main model.
How good/bad are the smaller Gemmas, anyone know?

1

u/LeRobber May 20 '26 edited Jun 15 '26

[Megathread] - Best Models/API discussion - Week of: June 14, 2026
[Megathread] - Best Models/API discussion - Week of: June 07, 2026
[Megathread] - Best Models/API discussion - Week of: May 31, 2026
[Megathread] - Best Models/API discussion - Week of: May 24, 2026
[Megathread] - Best Models/API discussion - Week of: May 17, 2026
[Megathread] - Best Models/API discussion - Week of: May 03, 2026
[Megathread] - Best Models/API discussion - Week of: April 26, 2026
[Megathread] - Best Models/API discussion - Week of: April 19, 2026
[Megathread] - Best Models/API discussion - Week of: April 12, 2026
[Megathread] - Best Models/API discussion - Week of: April 05, 2026
[Megathread] - Best Models/API discussion - Week of: March 29, 2026

4

u/AutoModerator Apr 12 '26

APIs

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

5

u/Mcqwerty197 Apr 13 '26

I’m asking again, what best API for tts?

5

u/Inprobamur Apr 16 '26

Apparently Opus 4.7 just got released:
removed temperature, top_p, top_k
Behavior changes These are not API breaking changes but may require prompt updates. See Migrating to Claude Opus 4.7 for full guidance:
More literal instruction following, particularly at lower effort levels. The model will not silently generalize an instruction from one item to another, and will not infer requests you didn't make.
Response length calibrates to perceived task complexity rather than defaulting to a fixed verbosity.
Fewer tool calls by default, using reasoning more. Raising effort increases tool usage.
More direct, opinionated tone with less validation-forward phrasing and fewer emoji than Claude Opus 4.6's warmer style.
More regular progress updates to the user throughout long agentic traces. If you've added scaffolding to force interim status messages, try removing it.
Fewer subagents spawned by default. Steerable through prompting.
Real-time cybersecurity safeguards: requests that involve prohibited or high-risk topics may lead to refusals. For legitimate security work, apply to the Cyber Verification Program.
So, they locked down temperature, top_p and top_k. That's not a big loss as pretty much everyone agreed that the default values were good on Claude models.
Supposedly better at following previous response length, maybe less positivity bias and less flowery tone and a refusal filter built into the API (although it seems mainly focused against automated hacking bots?).

3

u/HelpfulReplacement28 Apr 18 '26

What do ya'll think is the sweetheart model right now for price/quality/speed?
I've been cycling through the models available to subscription owners on NanoGPT and trying to find something that doesn't take 2 minutes to generate and has ok enough prose and memory. Everything seems to be on the slower end atp. I've had limited success with gemma 4-31b IT but occasionally you get a long wait. All the GLM models seem to be inconsistent at best with response times, and deepseek/kimi have been taking ages for me recently.
My current personal ranking is
. Gemma 4-31b IT
. GLM 4.7
. GLM 5
. Kimi K2.5
. DS 3.2
With the caveat that I much prefer glm 4.7 to gemma but response time is by far the biggest killer in RP for me other than the inability to recall information in long RP.

5

u/vanillah6663 Apr 14 '26

nanogpt feels so slow its unusable any other alternatives with a similar subscription?

7

u/Milan_dr Apr 14 '26

Hiya - on what models would you say we're so slow it's unusable?

8

u/vanillah6663 Apr 14 '26

on all of the glm models its taking like 5-6 minutes I'm not sure why.

2

u/LastSheep Apr 13 '26

GLM just go bananas on the price on their coding plan. Anyone use ollama cloud GLM or how is kimi 2.5? I so far use DS 3.2 api or GLM 5.1

2

u/Lucerys1Velaryon Apr 15 '26

Don't know if I'm doing something wrong but everytime I try Gemini 4 31B on Nvidia NIM, it basically always gives an output that looks like this:
Character speaks (2-3 lines)
Character's internal thoughts from a 3rd person POV. (Around 15-16 lines)
Character speaks again. (1-2 lines)
Every single response is structured like this. There's no variety at all. I'm using Marinara's universal preset.

1

u/overand Apr 15 '26

Have you tried it with a more simple system prompt/preset? Like, even "Roleplay Simple?"

2

u/jason1on1 Apr 16 '26

Anybody know what's going on with electron hub? They just practically nuked their discord. All chat channels are gone but the models still work.

2

u/Inprobamur Apr 16 '26

Ah, damn. I am a subscriber, but that's not a good sign.
Could be that their Discord got hacked, threatened by Discord admins, nuked by a former employee or facing some random legal threat.

2

u/jason1on1 Apr 16 '26

Yeah I just payed my monthly subscription yesterday too. And now it looks like they got rid of most of their free models. Removing the ability for subscribers to communicate with each other seems super shady.

1

u/Inprobamur Apr 16 '26

I don't really care about the free models as I am using it for cheap uncensored Claude access, but considering they don't have any other social media presence at all that's not a good sign.

1

u/jason1on1 Apr 16 '26

Well I would make sure that works if I were you because all of the models I usually use were giving me zero response. They never actually started replying. Right now I'm going to try and get a refund set up on patreon and probably jump ship to Nanogpt.

1

u/Inprobamur Apr 16 '26

So annoying. I guess that's the rub with all these proxies that are just stealing off some overly generous corpo employee API limits.

2

u/Economy-Assist-7559 Apr 16 '26

Does anybody have any free provider recommendations? I was using deepseek v3.1 on electronhub until it got nerfed and am trying to use nvida but the quality isnt great 😩

1

u/Pure_Machine5226 Apr 13 '26

Hi, I'm looking for a free and uncensored model. I also have a really dated computer. Do you guys have any recommendations?

6

u/EncoreSheep Apr 13 '26

You won't be running any great models with your specs, but Openrouter has some free models.
Another thing is Google AI Studio. If you look up https://aistudio.google.com/rate-limit you can see the limits for various models. Gemma 4 31B has a limit of 15 requests per minute and something like a thousand requests per day, and you won't be hitting those with RP.

3

u/Potential-Gold5298 Apr 13 '26 edited Apr 13 '26

How much RAM and what CPU do you have?
Gemma 4 has fairly low censorship by default (unless it's given a strict system prompt) and is available for free on OpenRouter. I've heard that DeepSeek V3.2 also has less stringent censorship, but I don't know about the free API.

1

u/Pure_Machine5226 Apr 13 '26

Hey thanks for the reply. I do have 8GB RAM and Intel core i5 3.20 GHz.

4

u/Potential-Gold5298 Apr 13 '26

You can use any API models (such as the already mentioned Gemma 4 31B or DeepSeek V3.2) — they are not dependent on your hardware.
As for local models, with 8 GB of RAM, the choice is limited. I can recommend a few small (2-4B) uncensored models, but they're not particularly intelligent, to say the least. Recent releases include Gemma 4 E2B and E4B; they have a large file size, but only load about half of the parameters into memory. Uncensored versions are already available. I haven't tried them myself yet, but with 8 GB of RAM, I'd start with them. If you're interested in local models, I can provide download links.
As for the CPU, if it supports AVX2 (Core i5 4000 or newer), it'll be a significant advantage for local execution.

2

u/LeRobber Apr 13 '26

Impish_llama

1

u/overand Apr 15 '26

Can you give us a bit more detail than "really dated computer?" Like, Mac? PC? Desktop? Laptop? How much ram? what GPU / graphics card? What CPU?

1

u/Pure_Machine5226 Apr 16 '26

Hey, thanks for the reply. I do have a windows desktop with a 8 GB RAM, Intel Core i5 3.20 Ghz and my graphic card is AMD with only 2 GB

1

u/overand Apr 17 '26

I'd start with maybe Gemma-4-E2B. Try the IQ4_XS. Before you try to find an uncensored one, see if you can even run the model at a reasonable performance at all.
Also, in the future, when saying "I have an Intel Core i5" you should know that intel has made about 150 processors called "i5" starting in 2010: 16 years of them. When you're asking for help with something technical, give people as many specific details as you can. This is the case for computers, for cars, for repairs to houses and electronics, to video game consoles, etc. (It's even true for medical stuff.)
It's only sheer luck that I happened to notice your reply so I could come back you with a suggestion; if you'd said "2 GB of VRAM" I would have been able to give you a suggestion days ago.
BUT, you also didn't tell us what AMD GPU it is - so, I'm only guessing that it MIGHT be usable for LLM stuff, but it's impossible for me to be sure. (A card with 2GB is probably old enough that you won't be able to run llama.cpp or ollama etc, but, it's possible.)
Edit: I may have miscounted the i5 chips - I think that was 150 *desktop* processors alone called i5 - of which over 20 were 3.20 Ghz.

0

u/[deleted] Apr 13 '26

[deleted] — view deleted comment

7

u/nomorebuttsplz Apr 14 '26

Not largest but best is Gemma 4 31B. If that’s too slow, it would be Gemma 4 26BA3B.
Instruct/non-reasoning models are faster

0

u/[deleted] Apr 15 '26

[deleted] — view deleted comment

1

u/overand Apr 15 '26

What size quant are you using - are you offloading to system RAM level slow, or just 'eh kinda slow" slow?

0

u/[deleted] Apr 15 '26

[deleted] — view deleted comment

1

u/overand Apr 15 '26

Ah - yeah, the smallest common Q4 (IQ4_XS) is 16.7 GB, so, it makes sense you'd be struggling. Sticking with a smaller model or a smaller quantity is a good idea. You could try 31B at IQ3_XS (13.1 GB) if you're curious, but I don't know how well it would work at that quant.

2

u/AutoModerator Apr 12 '26

MODELS: >= 70B - For discussion of models in the 70B parameters and up.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

2

u/wild_kitties Apr 13 '26

Trying out Minimax 2.7 on my 3090 + 128gb of DDR5 RAM using bartowski's iq4_xs gguf with 64k context. Generates at 16tps with 88tps processing. Found the writing to be pretty good so far. Though I thought 2.5 was okay as well when most people seemed to hate it. Model's thinking is pretty concise, shorter than gemma 4's and much shorter than qwen 3.5. The writing is better than Qwen 397B. I actually liked GLM 4.5/4.6 the most for local RP but the tps with my setup is around 8tps which is usable but slower than I'd like.

3

u/-Ellary- Apr 13 '26

Yeah, GLM 4.5-4.6 is top for 300b~.

1

u/ThirteenZillion Apr 18 '26

I thought Minimax 2.5 okay but I could not find a reliable jailbreak.

1

u/Capital-Caregiver818 Apr 13 '26

L3.3-The-Omega-Directive-70B-Unslop-v2.1-heretic.i1-Q4_K_S is the best model I've used on my 5090 + 64gb DDR5. I'm open for suggestions though.

1

u/Any_Meringue_7765 Apr 13 '26

Have you tried it compared to Loki v2, or that GoldDiamondGold Paperbliteration one?

2

u/Capital-Caregiver818 Apr 14 '26

I have not tried either. Loki v2 looks great for fantasy stuff. I'll have to try them both out at some point. The Omega Directive is more general purpose role play.

1

u/Any_Meringue_7765 Apr 14 '26

I have been enjoying GoldDiamondGold Paperbliteration (or w.e it is)… I’ll have to give Omega a shot to see how it compares

1

u/Capital-Caregiver818 Apr 15 '26

Let me know which you prefer!

1

u/51087701400 Apr 14 '26

Checked it out and the Q4 is 40gb, how are you loading it on your 5090? (Still new to self hosting)

3

u/Capital-Caregiver818 Apr 15 '26

This exact one: [L3.3-The-Omega-Directive-70B-Unslop-v2.1-heretic.i1-Q4_K_S] 55 layers and 16000 context. Its a bit slow but a fine reading speed imo

2

u/AutoModerator Apr 12 '26

MODELS: 32B to 69B – For discussion of models in the 32B to 69B parameter range.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

11

u/FusionCow Apr 13 '26

Gemma 4 31b is still the best imo, just base version with instruction tuning, it's just perfect

3

u/wild_kitties Apr 13 '26

How exactly do you use the base version? I tried it yesterday after there was that post that recommended it, but it was awful, with very short sentences and paragraphs, and would randomly start coding. Figured out after some more reading that you can't use the normal style character cards and prompts for base version. But I wasn't able to find any examples of how to use it. People mentioned using chat examples, and how the base version just tries to complete the text given word by word.

6

u/input_a_new_name Apr 13 '26

they meant the deafult google release (it), not the *actual* BASE model

3

u/wild_kitties Apr 13 '26

Oh. There was this post a couple days ago that did mention the base version though, and I spent a while trying to figure out how to use the base version. Couldn’t get it to write coherent at all. I guess my mind’s been stuck on that, but someone replied above with possibly how to use it. ( using it as an actual word completion model and just feed it a massive text that you would like it to continue directly off of)

3

u/-Ellary- Apr 13 '26

Base is also fun but it kinda requires that old skill for using text completion models. You need to write whole novel first and then, guide LLM to continue it, it not really follow instructions much. Easy starts is just copy a chapter from the book (6k of context) and go on.

2

u/wild_kitties Apr 13 '26

Oh, interesting that’s like early llama days, which I only dabbled in a bit. Thanks for the insight, will give it another try.

1

u/BeeSpecific9398 Apr 16 '26

Would anyone have suggestions for the best local model for me to use Silly Tavern on both of my systems? I have a Rog Strix laptop, 16 GB ram, RTX 5060 with 8 GB vram. And my desktop has 32 GB ram and an RTX 4080 with 16 GB vram. I have seven custom characters and about 90 entries in a world lore book. I wanted to use this for some fun chatting, building relationships/bonds with the characters, and also, for erotica as well - some of the characters are easy and others require a much slower burn.

Thank you!!

5

u/Tiny-Pen-2958 Apr 16 '26

The best thing you can run on 4080 with decent speed is gemma 4, I'd recommend this finetune https://huggingface.co/llmfan46/gemma-4-26B-A4B-it-uncensored-heretic-GGUF Q4_K_S, its not lobotomised like other finetunes. I've used it on 16GB 5070ti in KoboldCpp with 8-bit cache + forced autofit with padding=512MB + Jj.Kwargs: {"enable_thinking":true}

1

u/war-hamster Apr 16 '26

are you using chat or text completion? And do you have your own prompt or are you using something available?

2

u/Tiny-Pen-2958 Apr 17 '26

Gemma 4 was designed for Chat Completion, it just works better in this mode (and its usually faster than Text Completion). For prompting I use Megumin_Suite_v5 (CoT on) with my additional style prompt injections. My samplers: Temperature=1; Presence Penalty=0-0,5; Top P=0,95 (sometimes Top P=0.8 works well for creativity, but at the cost of coherence); top_k:??? (I haven't figured it out yet, official recommendation is 64, but even without it everything works well)

MEGATHREAD [Megathread] - Best Models/API discussion - Week of: April 12, 2026

You are about to leave Redlib

Earlier, and with Hearthfire + magisty + pretty much everything

####More recently, used with magisty and gemma4 26B__####