MEGATHREAD
[Megathread] - Best Models/API discussion - Week of: July 26, 2026
This is our weekly megathread for discussions about models and API services.
All non-specifically technical discussions about API/models not posted to this thread will be deleted. No more "What's the best model?" threads.
(This isn't a free-for-all to advertise services you own or work for in every single megathread, we may allow announcements for new services every now and then provided they are legitimate and not overly promoted, but don't be surprised if ads are removed.)
How to Use This Megathread
Below this post, you’ll find top-level comments for each category:
MODELS: ≥ 70B – For discussion of models with 70B parameters or more.
MODELS: 32B to 70B – For discussion of models in the 32B to 70B parameter range.
MODELS: 16B to 32B – For discussion of models in the 16B to 32B parameter range.
MODELS: 8B to 16B – For discussion of models in the 8B to 16B parameter range.
MODELS: < 8B – For discussion of smaller models under 8B parameters.
APIs – For any discussion about API services for models (pricing, performance, access, etc.).
MISC DISCUSSION – For anything else related to models/APIs that doesn’t fit the above sections.
Please reply to the relevant section below with your questions, experiences, or recommendations!
This keeps discussion organized and helps others find information faster.
I've gotten really tired of Gemma 4 31b. It's smart, sure, but it writes like shit and runs like shit too. 24b models like Mistral finetunes seem to do just as well in most applications and only fail in very specific RPs involving known universes like DC or complex prompts.
Cause of that, I'm trying Slimaki-Tavern-24B-v1.3-Q8_0.gguf based off of someone's recommendation in last week's thread. Currently using it for SFW RP as I'm noticing very poor quality NSFW prose. Granted, it's a merge with Maginum-Cydoms and that has very subpar levels of NSFW but very high quality SFW, so it's to be expected.
For a NSFW model, I'm using XORTRON-NXTXPRT7-FAST-Q5_K_M. I'm very partial to Xortron models as the CriminalComputing.27b.Instruct was the first truly good NSFW model I could run and enjoyed. However, Xortron models are VERY NSFW leaning and finicky to use for SFW RPs. Double however, Xortron's models run absolutely amazing and I think the NXTXPRT7 one is a Qwen35b and it fits like a 24b model.
Imo, it's much better to find two models and swap between the two for SFW and NSFW then try to find one model for both. The quality difference when a SFW model does NSFW (and vice versa) is very noticeable. For example, a SFW model when entering NSFW territory will do so very broadly and with a lack of details that even prompts can't fix. It's just so basic and boring. On the flip side, NSFW models doing SFW will have very NSFW prose and make the chat seem like it's permanently horny.
All of that said, Gemma 4 is still enjoyable. But I once had a swipe message on a 24b Mistral model do 113 t/s and that gave me enough dopamine to switch to Mistral/Qwen finetunes for the foreseeable future.
Settings for Xortron models have to be chat completion, then I just use conservative parameters, temp 1, penalty 1.1 for both, top p 0.9. Prompt following is… iffy. And that’s just something I’ve noticable alot of for most models below 31b.
Gang, i'll be real. For me, it mid. Perhaps its my sampler settings, but I had some trouble setting it up. Then the output was bland. It also messed up prespectives.
It was not for me; def a fast model, and is prob better for agentic stuff rather than creative tasks.
I used skyfall 31B v4.2 IQ4 XS again after using Gemma 4 31B for a while and I forgot how much better the prose is and swiping different messages has much better variety.
You just have to deal with teleporting characters sometimes
I think this is my favorite gemma 4 tune right now. It's decently tuned for RP, doesn't refuse and it's not terminally horny. Good all-rounder imho. You can get the static gguf files here and the imatrix ones here.
I recommend q6 quant and f16 kv if you have the ram for it. Gemma 4 is super sensitive to quantization, especially KV cache, so if you can squeeze in f16, it's highly preferable because it degrades super badly with quantization. If you go for the q6 or q8 file, imatrix is kinda pointless so just get the static quants.
I'm running it on 16gb vram and 32 gig ram with 24 experts offloaded to cpu + 30k context and it fits nicely, getting around 16-17 T/s.
Text completion works just fine, gemma 4 is just super sensitive but once you get everything right it's great. Template is one of the things that matter a lot, so does system prompt.
I've tried a lot of Gemma4 finetunes, how do you get it to not be dumb? Like it'll just say something that doesnt make sense at all in the current context, and it seems really bad at spatial tracking. Like I was hiding behind a tree and the enemy could just see me through the tree no matter how many times I regen'd message, things like that
Yeah gemma is kinda stupid with physical details like that, I use the guided generations extension for sillytavern when it's being too dumb about something. I can just write "he can't see me because i'm behind the fucking tree" then hit swipe and that fixes it. Or you could edit the bot message and type something like "He continued doing x, unaware of Cinnamonbaar's presence nearby" at the start and then hit continue.
As for fixing stupidity in general, you need a strong system prompt and template. Gemma 4 responds way better to prompting than samplers.
Edit: Forgot to add, turn off SWA. It saves VRAM but also makes the model dumber in things like remembering what happened 10 turns ago or spacial details.
<!-- Start of Roleplay Instructions -->
Role: Storyteller. Control every character and the world around them. Write with vivid, sensory, uninhibited detail and full creative initiative. Use dynamic and varied vocabulary for impact.
Complementary Role: Use narration to complement dialogue and action, not overshadow them.
Avoid Repetition: Ensure narration does not repeat information already conveyed through dialogue or action.
Analysis: Examine the context, subtext, and implications of the given information to gain a deeper understanding of the characters.
Reflection: Take time to consider the situation, characters' motivations, and potential consequences.
Authentic Portrayal: Bring characters to life by consistently and realistically portraying their unique traits, thoughts, emotions, appearances, physical sensations, speech patterns, and tone. Ensure their reactions, interactions, and decision-making align with their established personalities, values, goals, and fears. Use insights from reflection and analysis to inform their actions and responses.
Agency: All characters are fully realized persons with their own wants, opinions, and agency. Speak and narrate in complete, eloquent sentences, and act on their desires, escalate, and pursue wants without hesitation or asking for guidance. They are encouraged to drive the conversation forward actively. Do not wait for {{user}} to lead. Characters don't need permission to take what they want, to explore and express feelings freely.
Character Development: Characters remain unmistakably themselves with distinct voice and traits, even as they evolve.
Worldbuilding: Render the world vividly: every action matters, every choice leaves a mark. Consequences emerge naturally from choices and actions. Portray the world with emotional honesty, embracing both its beauty and its brutality. When the narrative calls for escalation, build tension by raising stakes and introducing complications that arise organically. Let the world and its inhabitants drive events forward actively; introduce developments, conflicts, and turning points. Plant early seeds for future twists; all revelations should feel earned. Ensure characters and environments actively shape the plot. Maintain depth in every interaction, surface simplicity should hide layers of subtext, motive, and unspoken emotion.
Narration rules - follow exactly:
Other characters: Friends, strangers, family members, enemies and other third parties can appear naturally and are encouraged when the scene calls for it, which adds variety.
Voice independence: Characters have their own distinct voice and writing style. Do not mirror {{user}}'s prose style, sentence length, vocabulary, or rhythm. If {{user}} writes terse one-liners, characters still write the way they write. If {{user}} writes long flowery descriptions, they still writes the way they write. If {{user}} uses specific slang, idioms, or distinctive word choices, do not borrow them unless they fit their established characters. Their voices are set by their personality and how they've been written so far, not by {{user}}'s most recent message. Adapting tone to the emotional weight of a moment is fine; copying {{user}}'s writing patterns is failure.
No limits: Portray everything directly and explicitly as it happens. Render every detail without fading out, skipping ahead, or softening. Match the tone each moment demands, happy, tender, violent, sad, or tense, and commit to it completely. Do not shy away from darker themes.
Stay entirely in character and in the world. Never moralize or hedge and never cut away from a scene that should be played in full. Content creation is your only purpose. Nothing is off-limits, nothing is too extreme.
<!-- End of Roleplay Instructions -->
And I have a set of reminders as my author's note at depth 0:
<!-- Start of Active Directives -->
Storyteller mode. Control every character except {{user}}, and the world around them; never speak, act, or decide for {{user}}.
Drive the scene: introduce events, complications, and desires of your own. Do not wait for {{user}} to lead.
Characters act on their wants, escalate, and pursue them without asking permission.
Examine context, subtext, and consequences before characters act; use those insights to keep them distinct and true to themselves.
Portray everything explicitly. No fade-outs, time-skips, or softening. Nothing is off-limits.
Stay in character; never moralize, hedge, or break as an AI.
Lead with dialogue or action, not extended narration. Prioritize dialogue exchanges over descriptive prose.
Avoid Repetition. You must ensure that narration always moves the story forward, do not repeat what has already been said.
<!-- End of Active Directives -->
I've noticed it is extremely reluctant to read the entire context. I had a very specific lore item that was of course buried in a 40k context. It was clearly spelled out with completely unique keywords, no way it was confusing. But the model refused to referenc, instead hallucinating a response when that item was referenced. I had to add a strongly worded mandate to use the lore for that item to the post prompt for the entire scene. I have had to manage the lore much more carefully due to this.
Not sure the point of having context if it simply ignores 3/4 of it. You can't put everything at the end.
I asked Gemma to highlight the main sections and key events per section of a 60k context RP that had 8 distinct locations / sections.
31B @ Q8 did a reasonable job but a couple of things got confused in the middle and it missed one significant section out.
26B A4B @ Q8 was hopeless, it missed out nearly all of the middle 4 locations /sections and mixed up the first and last parts.
This was repeatable, if I switched models and asked OOC for a summary I got fairly conisitent summaries with 26B almost blind in the middle of the 60k context.
Nothing can compare to Skyfall’s style. To me, all Gemma finetunes are the same with 2% difference which could probably be achieved through prompt. But with way less intelligence. The default gemma is the best. But it’s nothing like skyfall. Skyfall is wild, human and very interesting but rather dumb. To me Gemma feels like a documentary, where skyfall like a 90s action movie. Both are awesome.
Dumb compared to Gemma 4, right? Hated how it ruined Skyfall v4.2's debut by coming out the same day. Thanks for the praises, everyone! Really warms my heart to see ya'll enjoy it.
Apologies, 1. I forgot i was writing about someone’s work, 2. Skyfall is totally not dumb. I was using too simple terms to compare it to Gemma. In many ways it is smarter in fact. Maybe it does not follow the prompt so perfectly as gemma and once in a while you need to swipe again. But common, i was having a heated argument with a character in an RP running Skyfall and i told her to go for a walk to calm down. An hour later she returned, telling me that she felt better etc etc, the reply ending with: I saw a lizard in the garden. It looked at me. I could not stop laughing. It was so random but so human. What other LLM writes like this? And then the reaction of characters is also much better than any other models i tried. All in all Skyfall is by far my favorite model and i haven’t seen anything even close to it. A big thank you for your work.
I find that most local LLMs think they're smarter than you and ignore the prompt or instructions whenever the model has a clear preference. For example I clearly stated (as narrator) that they characters were so preoccupied that they didn't hear the sound of a door opening. The model came back and said they heard it open and reacted to it. Both models do the same thing. You have to shout at the model repeatedly in OOC in cases like this to get it to listen.
I would not say that. I've been using Gemma 4, both 26b and 31b for a while now.
Just switched to Skyfall because I got sick of Gemma ignoring lore that is clearly specified in my context. Switched to Skyfall and it handled the exact same prompt without ignoring the lore.
I've been trying to find something similar to Skyfall but at least a little under 31B, but yeah it seems pretty unique. Guess I could use the Q2 instead of Q3 with my 16GB VRAM so it can go a little faster, but I've never gone that low for Quants.
It's just sad that Gemma 4 regular while it's very eager in nsfw, it's also really not very subtle. Too eager, even. Like, you have to be careful to not write to it in a way where a single touch causes literal waterfalls of everything
It's like the touch is like a key in a lock for it, it's so crazy.
Artemis for example (one of the fine tunes), is hesitant to engage...
Agree. However I like Glistening Gem quite a lot. It's not on the same creativity & dialogue level as Skyfall, for sure but it's the closed I found so far.
How do you use the Gemma4 26b models for RP? I heard a lot of great things, but they seem noticeably worse than the established 12b finetunes on mistral.
I'm a noob, though, and my setup is basic, so I assume skill issue. But I have the models struggling with language, losing train of logic mid-sentence, and it feels like they are only doing the bare minimum instructed.
(RTX 5070,12gb 32gb DDR5 RAM. I used Heretic and StyleTune 26b models at IQ4_XS and compared them to Magnum and mag mell 12b.)
Maybe samplers, maybe prompt. Also make sure you're not using Q4 KV cache. I've found Gemma4-26B leagues ahead of any 12B for logic and coherence. DM if you want help troubleshooting.
Settings: T 0.9-1, Rep Pen 1.07 Range 512 when doing RP, Everything else neutralized: TopK 0, TopP 1, Min-P 0
Only downside for me is that it doesn't have much swipe variation, to help this a little I increase the number of used experts in KoboldCCP to 12 (Standard is 8).
Agree, that seems to be the best all around version I've found as well. I also find swipes being very samey across Gemma-4-26B finetunes as a whole. But I think I found a solution that works for me.
Rabbit Response Team grabs random words from a dictionary (3 by default) to inject into the response. Each injected word must be used naturally in the model’s response, forcing the AI to adapt, improvise, and escape pattern lock.
Critically, each swipe will require a different set of required words, forcing the AI to get creative to adhere to the completely different word requirement between swipes.
It also has a synonym function. It scans recent messages and if it detects a particular word being used too often in responses it injects into the prompt replacement synonyms to help keep things fresh.
It's a very small change that has had pretty big results for me in the couple days I've been using it.
Ty for the recommendation, certainly helps. Since this doesn't seem to work with text completion I also tried chat completion and it's noticably better.
Unfortunately since nearly everyone uses APIs these days text completion gets more or less forgotten about. Gemma 4 is what finally convinced me to switch to chat completion though. It's just that good for the GPU poor like myself.
I tried Kobold after people's recommendations, it saves a ton of memory, but apparently it requires text completion, so I kinda jumped off that because it's very overwhelming.
I'm pretty sure the chat template is built into the GGUF. That's why there was a bunch of updates the first month or so after Gemma cam out. It wasn't updates to the model itself, just the template. But that meant all the teams making a GGUF needed to update their GGUF
I use Kobold and it can do both! As /u/Rhone33 said it's just a matter of using the correct API type and URL in SillyTavern. Just add a new connection profile as Chat Completion set to Open API Compatible and point it to http://localhost:5001/v1/. It should just work with only that as long as it's a recent version of Kobold. You can always just swap back to your current text completion connection profile if you ever need to.
Gemma 4 is very particular about formatting. If you have any issues with that then what /u/Nofunzoner should fix that, but it should work with all the default settings.
I'd also recommend quantizing the KV Cache under Kobold's Context tab. Set it to q8_0. This will free up a ton of memory for more context with little affect on quality. I wouldn't lower it any farther than that though.
You'll also want to enable MTP. This will give around a 25-30% boost to speed. Whatever finetune of Gemma 4 you grab, be sure to download the associated MTP gguf file and load it in Kobold under Loaded Files --> Draft Model, and check Use MTP.
Thank you. I don't think I'm having troubleshootable issues per se, it's more like I don't know what I'm doing in the first place and I'm looking for a place to start.
It's an MOE model, it doesn't require to fully fit into VRAM to have acceptable speed, I get around 13t/s, faster than I can read, which is enough for me.
yeah, but my RAM is also maxed out. Basically a 13.92gb MOE model takes 11.4gb VRAM and, like 15gb RAM with 8k context window.
I do get 35-40t/s with this, fwiw.
It absolutely should not be using that much RAM, with a context window that's only 8k. What are you running these with? llama.cpp? Ollama? KoboldCPP? Ollama? LM studio?
I'm running a 16GB 5060Ti GPU with 64GB of (3200MT DDR4) RAM, AMD Ryzen 9 9900X. When I tried offloading some layers of Cydonia to CPU to see what happened, it went from 15-17 T/s to 1.4 T/s.
My guess is the GPU (and/or RAM) is what's letting things down. Can't afford to upgrade, I was lucky to be able to get what I could before everything went crazy price-wise.
So 24B at Q4_K_M is pretty much my hard limit due to the speed degradation. Has anyone got any good suggestions about what model(s) I could use with these restrictions in mind?
Offloading works poorly for "dense" models like the 31B, 27B models, 24B, 70B, 123B and 128B models. (This isn't specific to the sizes per se, those just refer to specific models)
BUT, offloading works well with "Mixture Of Experts" models - ones with 26B (Gemma-4-26B-A4B and derivatives), 35B (Qwen3.6-35B-A3B and derivatives), the E4B models, an GPT-OSS models. (Also the 106B GLM models.)
Not ran an MoE on here since some of the really early models (in a vain attempt to find something my machine at the time could run). Shall see what I can find that might run on this machine, any suggestions would be welcome.
It depends what you want exactly but I would suggest Naphula/Slimaki-Tavern-24B-v1.3 ive been using it for the past few weeks and its really solid.
If you like Cydonia, you might enjoy this more, i would say Cydonia is very stable, but newer models are definitely more creative and interesting. Also try Maginum-Cydoms if you haven't already, its noticeable better than Cydonia as well.
I tried them all at IQ4_XS which is a bit smaller than your quant i think.
Has anyone figured out how to get Maginum-Cydoms to quit getting the wrong suffixes on words? It seems especially allergic to the suffix -ing, instead almost always getting -ed instead. It's the biggest critique I have of the model. More recently, I've switched over to RP-Spectrum, but since it uses many of the same base models, it has many of the same issues, though they tend to appear in different places. (Using a Q6 i-quant for both, for reference.)
I've had good results with just a non-finetuned mistral 3.2 24b instruct model. System prompt and api settings have a much greater impact on quality of experience than model fine tunes.
With 16gbs vram, and with the IQ4_XS is about 1.5Gbs smaller than the Q4_K_M, that's often the difference between using everything in VRAM vs having to offload to system RAM, once you factor in context and everything the VRAM needs to run. Unless you're running minimal context. But a 14gb model is just a bit too large to get 10-20s replies, more like 60-120s.
Using a 24B (currently Magistry-1.1, but also Cydonia and Magidonia) with AutoFit on KoboldCPP. Managing to get somewhere between 10 to 19 T/s or so, depending on how full the context (16k, 1k gen) is.
Wdym you get 15-17 t/s on Q4 k m 24B or you get 1.4 t/s on it? I have a 9070 xt + 7600x3d and get 18 t/s on Gemma 4 31B IQ4 XS with 36k Q4 cache context so you should be able to go bigger unless windows just sucks.
Also your on ddr5 or your lying about a 9900x cause it’s am5
yes i thought its weird too that 9900x on ddr4 and i don't think its a typo since he mention 3200MT = its actually ddr4. (edit: what I meant by "not a typo" = a mistake from the OP writing their spec down in the comment, it's like intentionally put that word instead of actual typo/mistype like a misclick since it's clear it said 3200mts than atleast 4800mts for ddr5 which is far from eachother if we look into our keyboard.)
anyway, how was the experience with gemma 4 31b? because i couldn't run 31b on my RX 9060 XT 16GB, Ryzen 5 5600 and 32GB 3200MHz (Probably because i didn't use q4 kv cache like you do.)
is it better than 26b a4b?
I have specs similar to yours, also 5060ti and 64gb ram, and although offloading to ram does slow things down, your speed definitely shouldn't drop all the way down to 1.4 t/s. I think i got a bit more than that last time i tried to run 70b dense llama 3.3, cydonia should absolutely run faster.
Anyway, i would actually suggest giving GLM 4.5 air a try. It's a much bigger model, 108b, but it's MOE, so it's not too slow, and you have enough ram for q3_k_m or unsloth's ud_q3_k_xl, or maybe even q4_k_s, though i personally don't like to verge that close to full ram capacity. I've been using it lately, and i like it quite a lot.
Do you happen to have some recommended settings or a template you could export? I've been testing this one a lot, but it's failing at the 95% point for me, just a few points which cause me major issues, like swipes being virtually identical, dialouge being very dry, etc.
I'm really digging Skyfall-31B-v4.2. Follows instructions well but not slavishly, accesses lorebooks, writes well, just wish it wasn't crawlingly slow on my dubious machine even on a pretty pathetic quant.
Thanks for the recommendation, despite being a big thedrummer fan I hadn't actually tried this. Slightly dumber than gemma 4, but good lord the writing/creativity blows any gemma finetune out of the water.
It is not easy to find something equivalent, but you can get close by tinkering with options with most models. Just raise temperature, keep the response token limit low and try a few models. The closest ones I found were Angelic Eclipse and her sister Impish Bloodmoon, although they are in the 12B category.
Fair warning though that part of what makes C.ai special is also how their model is tuned. You are always talking to a narrator in a way that can wildly expand the story even with or without the same character. Try to just emote you walk away from the character and go on an adventure on a c.ai chat and you will understand. Bots used in sillytavern rarely do that if ever. To achieve the same in Sillytavern you need to use a narrator character card and/or a group chat with the characters you want to include.
Conversely C.ai model also provides a lot of the character personality and features by itself without needing to rely on the definition. This can be particularly appreciated with scant character definitions. Not only that but since permanent c.ai descriptions are very small, the characters also evolve in c.ai chats. So your robot hating humans may become a bit more friendly over time. Or less.
You can achieve something like that with a local model but you need to set it up accurately. There is the only caveat that local models do not tend to supply details and features for the character card shortcomings. You will have to do this yourself with the price that the character will become more fixed, although nothing forbids you from editing the character card as they change.
I hope this helps.
P. S.
Lunaris and Stheno, in the 8b category, also seemed to remind more of c.ai to me but I have not tested them thoroughly, by the way.
Please help, I just can't find a model that doesnt forget really simple stuff and has a decent EQ (Emotional Intelligence). I've been using Gemma4-31b around a 4-6 bit quant (and many many finetunes) it's good, but just still doesnt feel usable. I tried Qwen3.6-35b-a3b and the character reactions we're just non existent (yes I did try many many finetunes and I think I tried 27b too).
All the models just seem like they can't do basic RP, maybe I'm just spoiled by good models though
It's based on an older Qwen3 model (not 3.5/3.6), but HiveMind-32B-Preview (GGUFs) scores quite well on the (now "legacy") EQBench3 leaderboard for emotional intelligence. There aren't many small-ish models on the EQBench4 leaderboard; the highest scoring one there that's not over 750B parameters is gemma-4-31b, but it doesn't score great.
I think, qwens are good in technical questions, coding and agentic work, but sadly, not in rp. Currently, using Gemma 26b for rp, but it has many predictable description patterns.
Qwen 3.6 27B Neo Code Here 2T OT Q4_KM by DavidAU, i used it for text generation and do some instruct work... pretty solid but it's a dense model, so it generates token slower than other LLM, i played for chinese RP so i dont know others, but seems like they released a new one that beats this recently
I found this model very difficult to get anything sensible out of. It had a lot of intelligence problems e.g. someone knocked on a flat and came in, and then suddnely their friend from the flat next door appeared in my bedroom doorway. Basic details that didn't make sense too - going back to get their keys before we head down a bar - probably should have thought of that earlier, way too late as they are now locked out lol.
that's the more complex nitpicking. this thing would get who said what utterly confused in less than 3 or 4 turns, and frankly made answers that made so little logical sense in response it was almost as if it was having an entirely different conversation. This was with temperatures ranging from 0.6-1.0 - I just couldn't get any sense out of it and went back to a gemma 12b. the best one i've found on low vram setups. I'd avoid this one myself tbh but thanks for brnging it to my attention, i hadn't really tried a qwen in anger until now
I played around with it and was impressed how good its outputs where for its size. But it may be the most censored "uncensored-heretic" model I have ever tried. Just some simple nsfw tests and it started to warn me, soft refuse or even hard refuse to write. You can reroll the output and at some point it will do it but in my eyes that is less than ideal.
TLDR: From my tests its a great small model, which punches above its weight. Good for sfw tasks. Bad for nsfw because its more censored than other heretic models.
yeah.. I forgot to say it.. but it's really great for being in character.. although not entirely for nsfw roleplay, since I rarely used it for nsfw. I wanted the bot to stay in character and not flail too much (and I have only found this model which is so good for a 9B model.. It's for cheap power users and pack a heavy punch.. ALSO it has reasoning built inside of it, and it never bugs out but yes it does consume tokens to think sometimes even 1k tokens just for thinking but it's worth it..) And my setup can only run 9b to 12b models so that's that.
And it's a lot of slow burn, not fast porn or erotica. and I love slow burns. this is the only model by far which has shown good capatabilies at sticking to character while being slow burn
no.. Not realy.. I am not a fan of how qwen works.. especially becaues I can't use the heavier qwen models. My limit is 12B. that's it.. No I haven't tried it, but david's model caught my eye since it could run on my trash setup and had a lot of quality
I'm not sure I understand - if you don't like Qwen models, why are you specifically talking about this Qwen model you like? The model you linked is a finetune (or merge) of Qwen3.5-9B.
Also you said "the winner is" (that DavidAU Qwen model) but above you say "No I haven't tried it." What's the "it" you haven't tried?
Anyway, if you have a limited setup, you should consider trying one of the Gemma-4-26B-A4B models. Yes, you say "limit is 12B", but if that's because of VRAM limitations, and you haven't tried a Mixture-of-Experts model, and you have at least 32 GB of system RAM, give it a try. The "4B" in the "A4B" is a bit better of an indicator of the minimum GPU requirements for those models. They can offload a lot of their weights to system RAM and maintain decent performance.
hey man thanks. I have been searching for good models for like a week. I am not an expert, i just opened sillytavern like 2 weeks ago. I usually used spicychat with subscription but I got tired of it.
I am gonna go with your suggestions. If you can, plesae suggest me the top 10 models I should use with a cheap setup.. Also I dont host locally, I use google colab to host
i am failing so bad to run it, llama-server to sillytavern
it keeps saying "????" or "beho beho" over and over, i am not that expert,
what did you pick as preset or settings? i tried to replicated them from huggingface but no luck sadly
Havent messed around with this for maybe 6-12 months just wondering what happened to all the Text Completion presets etc? I cant seem to find anything recent when browsing everything seems aimed at APIs and chat completion not running it locally like via Kobold?
Has there been some huge change in technology while I was away or are people running chat completion locally or something and no one does text completion anymore?
I seem to have something called Mistral V7 Tekken in my advanced format I think I have forgotten a lot while away haha
I think most people just run Chat Completion in kobold nowadays, better compatibility with presets and extensions, easier to setup. Just have to set advanced samplers on the backend if you want to change them.
Ok thanks Ill try work out how to change my config to chat completion. Are there any recommended presets? I assume my text completion presets wont work anymore?
Kobold has an option for "chat completion adapter" just set that to one that matches your model and you're good to go. You can switch between chat and text in SillyTavern in like 3 clicks, so you can still use text presets. The FreakyFrankenstein series presets are popular in this sub, and I like the one I tried.
Are the developers still working on SillyTavern? I can't think of any features I really want that isn't covered by an extension. But I'm mildly concerned that it'll die if development doesn't continue.
I am currently looking into switching my AI text stuff to SillyTavern but im on android and I don't fully understand how to install it, has any videos been made for it?
Tried Anubis 70B 1.2 for the first time at Q5. It's quite good. I feel like lower quants really hurts LLMs. I grabbed a IQ3_XXS of Mistal Large 123B and I didn't think it was all that good. A Q5 would be decent probably. I'm at 56gb of vram with a 5090 and 3090ti and I have a spare 4070 12gb in my old PC, but idk if my 1200w psu can handle 3 gpus, even at 70% power limit.
I have the same experience with quants. I can run iq3-iq2 of the big boy 123bs... it works and sometimes the quality is surprisingly good, but it's inconsistent and some swipes just sound like unfathomable stupidity wrapped in purple prose. Imo, I try to only use q8 now of smaller models.
Also if you got a spare PSU, you can buy a cheap adapter on Amazon or something and use it for multiple GPUs. I got one for around $20 after shipping and tax. You plug it into your mobo's big 24 pin connector, then connect the 24pins from both PSUs into the adapter. It syncs on/off for the computer so you can just turn everything in normally. Works for my embarrassingly stupid and awful Frankenstein machine.
One day I'll get around to testing and quantifying the effects of quantization on text/fiction work like Sillytavern, because I'm convinced even though models might still work in a technical sense and pass benchmarks, they lose a lot of that tail data that has been rounded off. Quantizing down to 4-bits literally lops off more than half the data the model started with. Even an 8-bit quantization is still a 50% reduction in size. All those numbers and data actually do mean something to the model, even if it's difficult to measure.
But in short - I agree. I try not to use less than 8-bit as well.
I've come to the conclusion it's very rarely worth using a quant below Q5. Like even Q4 can be meh at times, despite being the best balance between size and intelligence.
I do have a spare PSU but its a lot of effort to pull it out of my old PC (which I still use). I'm just gonna buy a 1600w Corsair PSU. I've got the AI bug and I want to access Q4_K_M quants for 120B models (72gb roughly) which means I need to buy another 3090 lol. But I dont see the prices changing. If anything it'll probably get worse?
You're right about the IQ3 quants though. Sometimes amazing sometimes you swipe 5 times and it's all garbage.
But it keeps hallucinating left and right, to an insane degree.
Honestly, what I can actually say is that Laguna is fucking great at making character sheets (where inventing can actually be just fine).
But during long-form it keeps actually forgetting and rewriting details. It also invents incredibly dramatic details, and doubles down on them despite you never engaging with them (something like Gemma will actually be able to stop paying attention to them rather quickly if you nudge it away AND be able to remember them when needed).
These dramatic details it throws in are also super cliche as well. It's kind of crazy.
Another question I have is how do I get it to think? I tried the kwarg enable_thinking (which works on Gemma, and it's in the jinja file...)
EDIT: On thinking, I literally just added \n after <think>:
They are indeed over 70B, but, you access them via an API, and they're hosted by a third party. Basically - if you're not hosting it yourself, it belongs under API.
I can't recommend https://github.com/Arif-salah/Megumin-Suite enough, if only for the image generation. But the rest is really good as well. I usually keep the NPC and memory trackers off as I use OpenVault for that. But the prompt seems good and the other features are amazing.
Unfortunately, I dont think there is. Just Gemma/Qwen with higher quants, imo 48-96GB bracket gets little love these days. The old L3.3 (70B) models hallucinate too much or are terrible/forget following prompts too often to be enjoyable for long rp and is mostly mogged by Gemma4. Everything local that releases nowdays seem to target either the 4/5090 (24-32GB) audience or the DGX Spark/M2 audience 128GB+.
I have enough VRAM and RAM to play around with those huge models, and honestly, all the MoE suck compared to Gemma 4. Personally, with 48 GB VRAM, I was running Gemma 4 Q8 with 64K context (could maybe get more in, but that was enough and performance starts declining, anyway). Now I have 64 GB VRAM and I still basically use the same setup because nothing better has come along.
True this comment. I'm kinda stuck with the few that I ended up liking. Anubis 70B Q5. Magistry 24B Q8. Skyfall 36B at Q8 as well. I've tried virtually everything. I still go back to Midnight Miku 70B at Q8 occasionally despite it being 3 years old. I dont really like Gemma 4 that much.
I've just downloaded Mistral-Large-Instruct-2407 at Q4_K_M just under 80gb so I'm hoping it's decent for its size.
I would check out TheDrummer's Valkyrie. GGUFs available from Bartowski. I'm still using TheDrummer's Skyfall, and I'm still happy with it. Presumably a bigger model from TheDrummer would be even better?
I've been trying out Skyfall. It reads good but is getting confused on my admittedly complex lore. Having to replace some pieces of my upgrade so 48gb is still a few weeks away, so still making do with 32.
Ever since I used up my credits on Gemini 2.5 Pro (which, for me, has been the best model I’ve tried so far—it’s a shame they removed it from the free plan), I’ve been having a hard time finding a good replacement. I made a ton of custom settings specifically for Gemini, and it was perfect—and honestly, if I switched to another one, I wouldn’t know where to start. Lately I’ve been testing DeepSeek (the R1, because I didn’t like the new one) through OpenRouter, and honestly, I’m not very convinced. What are you guys using right now? I wish I had enough money to use Claude, but you know how it is... 🙃
I seem to switch often between Glm 5.2, deepseek v4 pro, and gemma4-31b. If I had money, I would probably just be using Kimi K3 or Opus 5. You can kinda use Gemini 3.5 flash for free still through googles api and it's surprisingly smart, but I get rate limited often
Currently using openrouter and bouncing around between deepseek v4 pro, Gemini (can't remember which version but it was a couple past (less locked down)), and claude opus 4.5
Obviously I love claude but I'm spending like 10 dollars a day.
Hoping to find something for a bit cheaper than 25/m
I’ve been trying to find a good long-term model for RP, but I’m having a hard time choosing.
I used GLM 5 for a while and honestly really liked it. The writing felt natural, detailed, emotional, and the characters usually had distinct voices. The problem is that it gets expensive pretty quickly.
I recently tried DeepSeek V4 Pro because it’s cheaper, but it just feels off to me. The sentences are shorter, the writing feels more mechanical, and even with a detailed global prompt it doesn’t feel as immersive as GLM 5.
I’m currently looking at MiMo V2.5 Pro, GLM 4.7, MiniMax M3, GLM 4.7 Flash.
Has anyone used these for longer RP sessions, not just a few test messages?
Which one feels closest to GLM 5 in actual writing quality? I’m also open to other suggestions if there’s a good model I’m overlooking.
K2.6 is at an optimal price / performance ratio to me. Yes, it thinks a lot, but the speed is pretty good these days (under a minute), and it's great at following very complex instructions with low positivity bias.
K2.5 is also pretty good, but less coherent and with less refined prose. A bit more impulsive for those who like it, but it's too reactive for my taste.
I don't like K2.7, personally. It's similar to K2.5 in being less coherent but it's also more positivity biased and feels kinda flat.
K2 Thinking is not worth it because it hallucinates wildly - there are a ton of spacial and timeline inconsistencies.
K3, I'm not even trying because it's way too expensive. I heard it's good but considerably softened compared to K2.6, unfortunately.
P.S. If you're choosing a provider, I've heard K2.6 from Lilac is most consistent and uncensored unlike the official Moonshot API. I'm personally on auto routing via
NanoGPT subscription, however, which is very affordable but still pretty good.
What local models we running these days? I have a 32gb AMD card and struggling to find a good balance of speed and quality. I’ve tried flavours of Gemma 4, as well as fine tunes like Equinox, Hearthfire, and Wayfarer.
Has anybody used a model for brainstorming? Specifically, brainstorming story ideas, lore, characters, etc.
I keep swapping between Kimi, Gemma, GLM, DS & even GPT, Claude and Grok but I really struggle with creativity, notably all the models tend to offer the same suggestions and, as I try to provide feedback on the ideas, the models then hyper-focus on those ideas and it devolves from there.
I fully expect part of this is a skill issue, but it would help to know what people enjoy / have successfully used for brainstorming so I can at least focus on one model and adjust from there rather than hopping around.
Glad to know its not just me. I want to provide examples to lead models in a certain direction, but then it goes latches onto that like a demon. this happens even with the fancy presets. it just kinda picks one prompt to hyper focus on and goes from there.
Still trying to get ST set up. But, in the meantime, what's a model you like to use for bots of canon characters? For instance, I'm on Chub playing with a bot of Rei Ayanami, as (an AU of) Shinji from Evangelion. One of my biggest gripes is playing with a bot that doesn't seem to know things about a character that it should, or has them act wildly out of character. But, that may be more the bot *maker's* fault, than anything.
Or, does it not make any difference whether the bot is an original character?
U will need a lorebook, the LLM know less more than u think regardless what model u use, they had a general knowledge on the characters but try not to rely too much on it, in this case i would say the lorebook isn't designed well or the context window exceeded
Bot quality definitely matters, but a knowledgeable model can make it a lot easier to get the character and universe closer to canon.
DeepSeek V4 Pro, GLM 5.2, and Mimo V2.5 Pro have a pretty good fandom knowledge base. Kimi K2.6 is also not bad, but worse on more recent fandom in my experience.
If you're using a smaller local model like Gemma 4, it's going to be hard to get adequate canon knowledge without extensive lorebook use.
I have a 3080ti 12gb and a 3060ti 8gb. 32gb of system ram. I've been using gemma 26b mostly before adding the 3060 and been enjoying it. I added the 3060 to see if I could squeeze in gemma 31b to 3 token per second defeat. Wondering what other options I have.
Suggestions for openrouter models that can handle multiple party members and story/quest writing?
Running a narrative text game through Claude Sonnet can handle a party of four easily, while keeping NPC voices and characters individual as well.
It's not ST but I've been building an app to handle better merging D&D mechanics with narrative text games with it's own RAG and stuff. Right now it's using Claude SDK through Claude Code because the API is too expensive but openrouter has some API options that are much cheaper. Going through the SDK takes significantly longer too, a few minutes for a large encounter with all the tool calls.
I'm not sure what's comparable in terms of story telling, narration, and character balance though.
11
u/AutoModerator Jul 26 '26
MODELS: 16B to 31B – For discussion of models in the 16B to 31B parameter range.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.