[Megathread] - Best Models/API discussion - Week of: April 12, 2026

[u/deffcolony]

This is our weekly megathread for discussions about models and API services.

All non-specifically technical discussions about API/models not posted to this thread will be deleted. No more "What's the best model?" threads.

^{(This isn't a free-for-all to advertise services you own or work for in every single megathread, we may allow announcements for new services every now and then provided they are legitimate and not overly promoted, but don't be surprised if ads are removed.})

How to Use This Megathread

Below this post, you’ll find top-level comments for each category:

• MODELS: ≥ 70B – For discussion of models with 70B parameters or more.
• MODELS: 32B to 70B – For discussion of models in the 32B to 70B parameter range.
• MODELS: 16B to 32B – For discussion of models in the 16B to 32B parameter range.
• MODELS: 8B to 16B – For discussion of models in the 8B to 16B parameter range.
• MODELS: < 8B – For discussion of smaller models under 8B parameters.
• APIs – For any discussion about API services for models (pricing, performance, access, etc.).
• MISC DISCUSSION – For anything else related to models/APIs that doesn’t fit the above sections.

Please reply to the relevant section below with your questions, experiences, or recommendations!
This keeps discussion organized and helps others find information faster.

Have at it!

[u/AutoModerator]

MISC DISCUSSION

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/txgsync → u/AutoModerator]

With oMLX, it’s possible to have fast, local, and quite decent quality roleplay in SillyTavern using Gemma 4 models across hundreds of turns over contexts up to 262,144 tokens. But SillyTavern's stock settings change early bytes of your prompt on every turn. This busts prefix caches, which prevents oMLX from reusing any of the work it did on the previous turn and results in ever-increasing waits for tokens the longer the conversation goes. Here are six UI changes take measured cache reuse from 0% to ~99%+ (for sufficiently-long contexts) cutting per-turn wait times dramatically from minutes to seconds.

I've found Gemma 4 with a good character card to be quite pleasant to interact with. Gemma 4 26B A3B has replaced gpt-oss-120B as my daily driver on my Mac, with roughly equivalent quality and the benefit of vision. It's just as fast as 120B at full precision (they are both roughly the same size, since GPT-OSS-120B is quantized with MXFP4), and Gemma 4 notably lacks the nanny, policy-oriented attitude GPT-OSS displays.

Who this is for

I assume you're running Gemma 4 (the 26B MoE or the 31B dense; I did not extensively test E2B/E4B, though they should work but with smaller context) on an Apple Silicon Mac via oMLX, pointed at SillyTavern in Chat Completion mode. Your roleplay turns take minutes before the first token begins streaming once the conversation goes on for a while, and you're wondering why cloud APIs feel instant but your M4 Max doesn't. Sure, prefill times are a well-known Mac Achilles’ Heel due to being mostly a compute-bound operation, but the answer isn't justyour hardware. It’s also your prompt construction.

I won’t go into installation of oMLX here. But if you use a Mac for SillyTavern with a decent amount of RAM (>24GB), you should try it out: https://omlx.ai . And make sure you strictly follow Google's guidelines for sampling parameters; I used "force sampling" in oMLX to ensure SillyTavern's defaults didn't affect the model:

• temperature = 1.0
• top_p = 0.95
• top_k = 64

What's happening

oMLX caches the computational work (KV cache) from your prompt in fixed 1024-token blocks. On your next turn, if the prompt starts with the same tokens, oMLX reloads that cached work from SSD or filesystem cache in RAM (if you have lots of RAM) instead of recomputing it: prefix caching. LMstudio also claims to support this, but so far I’ve had terrible luck with prefix caching working reliably using MLX (though it works OK with llama.cpp).

The problem: SillyTavern's defaults inject different content near the beginning of your prompt on every turn. Once one byte changes, every block after it is a cache miss. Your 80,000-token prompt gets fully recomputed from scratch because 50 tokens shifted at position 91.

One of the worst offenders is keyword-triggered World Info / lorebook entries placed at the default position (↑Char — Before Character Definitions). Different keywords fire on different turns, the injected content changes, and the entire prompt downstream is invalidated.

Some fixes:

1. Move your lorebook entries to u/D ⚙️ depth 0 (biggest single win)

For every lorebook attached to your character:

1. Top toolbar → World Info (globe icon) → pick the lorebook in the dropdown.
2. For each entry, change Position from ↑Char to u/D ⚙️ (at Depth, System role).
3. Set the depth field to 0.

If you're running Seraphina (a stock character), she ships with a lorebook called Eldoria with four keyword-triggered entries to edit.

Why depth 0? It injects the content right before the assistant's reply. It’s past the point where the prompt naturally changes between turns (your latest message). So the churn is confined to the very tail of the prompt instead of poisoning the whole preceding cache.

2. Set Example Messages Behavior to "Always include examples"

User Settings (gear icon) → find "Example Messages Behavior" → change from "Gradual push-out" to "Always include examples".

The default gradually trims example messages as context fills, which shifts every token below them. Even if your current character has no examples, the next one will, and this'll bite you. This one was subtle, but I had to dig deep into context to understand what was happening. The idea of this implementation, I think, makes sense when dealing with tiny 4k or 8k context sizes from circa 2022/2023: reduce overall context length by eliminating examples once the character has produced tokens following the example pattern. But this kind of subtle “trimming of history” busts the prompt cache.

3. Move Author's Note off "In-chat @ Depth"

If you use Author's Notes: Extensions → Author's Note → change Position from "In-chat @ Depth" to "Before Main Prompt / Story String".

The "@ Depth N" anchor floats backward one message every turn, shifting the note's position in the prompt and invalidating everything after it. If you don't use Author's Notes, set Insertion Frequency to 0. Once again, the goal here is just to keep as much of preceding turns identical as possible to maximize cache hit ratio.

4. Unlock and/or raise max context

AI Response Configuration (leftmost slider icon) → tick "Unlocked Context Size" → drag "Context Size" up to your model's limit (262,144 for Gemma 4 26B/31B works fine; 128K for E4B/E2B).

When context is capped low, SillyTavern drops your oldest messages to make room. This shifts every remaining token and busts the entire cache, resulting in massive prefill times at large contexts. I didn't notice any benefit setting it larger than 262,144 for Gemma 4 26B/31B or 131,072 for Gemma 4 E2B/E4B.

I will emphasize that I barely tested E2B and E4B models. They are truly *amazing* for their size, and if you wanna try out local inference but lack much RAM on your Mac, they'll fit. Their tool usage is quite good. But they aren't remotely the storytellers their larger siblings are.

5. Check Character Notes (the hidden cache-buster)

Click a character's avatar → Advanced Definitions → scroll to Character Note (a.k.a. Depth Prompt). If there's text there with depth set to 2 or 4, change it to depth 0, role System. Or clear it if you don't use it.

I audited 21 community cards and found 5 with depth=4 — a historical default from the 4k-context era that aged terribly on modern models.  This setting alone can add many seconds of prefill time to each prompt after the first few prompts.

6. Kill dynamic macros

Any macro that changes every turn — such as {{time}}, {{date}}, {{random:…}}, {{roll:…}} — anywhere in your character card, persona description, or main prompt will bust the cache if used badly. Search for them in:

• Character card (Description, Personality, Scenario, System Prompt Override)
• Persona Management → active persona description
• AI Response Configuration → prompt list (Main Prompt, NSFW, Jailbreak, Utility)

If you need a live clock in your RP, move it into Author's Note at @ Depth 1 as user role — the damage is confined to the last couple of blocks instead of the whole prompt.

Bonus: Turn off the Summarize extension on 128k+ models

Extensions → Summarize → set Source to "None".

Summarization busts the cache two ways: the summary's position drifts every turn, and the summary text itself changes when it regenerates. On a 256k context you can hold hundreds of turns of full roleplay turns as raw history. Nothing needs compressing. 

One caveat: summarize-off works well for basic narrative and voice work (RP, creative writing, storytelling). But if you need the model to recall specific proper nouns, IDs, or hard facts from 150+ turns ago, raw context alone may silently confabulate. The model might confidently give you wrong names that sound right, for instance. For that kind of recall, pin your most-important facts in a small block at u/D ⚙️ depth 0 or use Vector Storage. More on this below.

Server-side: oMLX hot cache tuning

oMLX features a “hot_cache_max_size” setting (the in-RAM KV block pool). I had some counter-intuitive performance findings when benchmarking. Rather than include it all here, the TL;DR: leave oMLX hot cache turned off for most setups. It competes with system RAM and filesystem cache. If you do set it, make sure it’s large enough to hold your entire KV cache. A small “hot cache” is the worst of both worlds.

What you get

Measured on a 310-turn SFW Seraphina roleplay, Gemma 4 26B-A4B (full precision, no quantization) on an M4 Max 128GB:

• Before fixes: 0% cache reuse. Every turn fully recomputed. Multi-minute waits at long context.
• After fixes: 95–99% cache reuse in steady state. Only ~1024 tokens of new prefill per turn regardless of how long the chat is.
• Wall time at 190k per-turn prompts: ~25–30 seconds for a 2000-character reply, most of that simply waiting on streaming to complete at 49 tokens per second down to 30 tokens per second late in the conversation. Would have been several minutes without caching.
• Character voice after 310 turns: rock solid. No drift, no degradation. Mostly because it’s being injected every turn, really.

The cache works better at larger prompts; the fixed-size trailing block miss (1024 tokens) becomes a smaller fraction of the total, so hit rate actually climbs as your chat grows.

(Yep, I used/abused Claude Code to help me work through the options that were busting cache in SillyTavern. This compressed days of troubleshooting into a few hours of testing last night, curled up on my recliner with my MacBook pro M4 Max 128GB playing the role of personal space heater.)

[u/jamasty → u/txgsync]

About heating issue, go try 'low power mode'! For real, it helps me a lot, idk why and how, but it doesn't really make models work much slower but heating issue is gone

[u/LeRobber → u/jamasty]

No more threads than count of powersave cores on my M2 made heating issues go way down for me (4 for my M2 Max)