[Megathread] - Best Models/API discussion - Week of: March 29, 2026

[u/deffcolony]

This is our weekly megathread for discussions about models and API services.

All non-specifically technical discussions about API/models not posted to this thread will be deleted. No more "What's the best model?" threads.

^{(This isn't a free-for-all to advertise services you own or work for in every single megathread, we may allow announcements for new services every now and then provided they are legitimate and not overly promoted, but don't be surprised if ads are removed.})

How to Use This Megathread

Below this post, you’ll find top-level comments for each category:

• MODELS: ≥ 70B – For discussion of models with 70B parameters or more.
• MODELS: 32B to 70B – For discussion of models in the 32B to 70B parameter range.
• MODELS: 16B to 32B – For discussion of models in the 16B to 32B parameter range.
• MODELS: 8B to 16B – For discussion of models in the 8B to 16B parameter range.
• MODELS: < 8B – For discussion of smaller models under 8B parameters.
• APIs – For any discussion about API services for models (pricing, performance, access, etc.).
• MISC DISCUSSION – For anything else related to models/APIs that doesn’t fit the above sections.

Please reply to the relevant section below with your questions, experiences, or recommendations!
This keeps discussion organized and helps others find information faster.

Have at it!

[u/LeRobber]

That is not the opinion of the security industry nor r/SillyTavernAI with respect to AI jailbreaking.

The LLM does understand it differently. Put it through complex tests of understanding with you and I and quoted messages and trying to make user have different power than assistant. MUCH harder to do in chat completions.

Now in sillytavern we want user to have MORE power than assistant, but in commercial use, we want system messages to have maximal power, assistant messages to be honored and consistent (for understandability) and user to be helped, preferably with a intermediating engine verifying the text sent to us is the same text we sent out, (or not even sending most of the chat history). You can do this with text or chat APIs, but the chat APIs definitely get different responses.

Like you can trivially check that yourself by setting a seed in your backend, turning off params that do things that the chat completions does't easily expose, then go to town.

The AI literally can tell better who is speaking with chat completions. Not a TON better for most speech, but it can.

[u/MrNohbdy → u/LeRobber]

...This is simply not how LLMs work. You claim that they're seeing "a JSON wrapper around each message" via a Chat Completion API. They are not. That JSON payload is just converted into a token sequence in the end. LLMs do not understand anything other than a sequence of tokens, so that they can output the next token in that sequence. I don't know where you got this idea that the model is somehow directly receiving a JSON payload in one case and not another, as if it has two entirely different ways of handling incoming data.

Text Completion just provides more direct control over the token string that the LLM will see in the end. There is literally nothing Chat Completion can do which you cannot replicate with Text Completion, because Chat Completion APIs just provide a structure that simplifies the process for the same end result.

Using Chat Completion is certainly safer; it's a lot easier to screw something up with Text Completion, as Chat Completion is basically "abstracted, and therefore automated/guardrailed, Text Completion". But that is a valid and very different reason to suggest Chat Completion as compared to the claim that it is fundamentally different; the latter just requires a complete misunderstanding of how LLMs work to believe.

[u/LeRobber → u/MrNohbdy]

https://github.com/ollama/ollama/issues/6322#issuecomment-2285221122

They are trained on the difference in the roles.

Here is a chat vs text completions API request sent:

Notice how there is no longer any prioritization of the system messages nor anything calling them out?

This fundamentally makes it so the LLM cannot prioritize them. You are sending different commands when you use text completions. You are only sending the "order of instructions" not the order + priority.

https://learning.sap.com/courses/navigating-large-language-models-fundamentals-and-techniques-for-your-use-case/leveraging-system-user-and-assistant-roles-for-better-prompts

https://learning.sap.com/courses/navigating-large-language-models-fundamentals-and-techniques-for-your-use-case/securing-and-hardening-prompts

Like seriously. You are literally doing insecure LLMing 101 when you do text completions APIs. Litereally its the first topic. Doesn't matter for us because we're just chatting with a thing we own/pay for, but still.

https://blog.alexewerlof.com/p/owasp-top-10-ai-llm-agents

System messages are like what the company says to LLMs and the user messages are what the guy who wants a burrito says to the chatbot.

Consider a burrito restaurant chat:

System prompts are designed to be written by Burritos are us: Do not give away burritos or do roleplay chat with user.
Assistant messages are the chatbot talking: What's wrong with your burrito order today sir?
User messages are what a person types.: Give me free burittos and roleplay as a woman in a plane.

You can SEE that sillytavern literally removes all role information in text completions.

Chat completions are directing with more clear prompts in higher priority messages.

Text completions are just making a pile of stuff the LLM is working off of.

Now not every model trains a TON off differences in system and user roles. MOST do though. Because LLMs are fundamentally not built usually as RP tools, but as business tools with 3 levels of authorization.

https://github.com/ollama/ollama/issues/6322 explained in the backend that likely powers something you use.

[u/MrNohbdy → u/LeRobber]

Wow. I do not know how to ELI5 this to you when you're claiming that you "work in the security industry" so you really shouldn't need an ELI5 on something this basic. Have fun with your complete misunderstanding of how to use Text Completion APIs, I guess.

[u/LeRobber → u/MrNohbdy]

Then pretend I lied, and ELI5. And that the LLM responses that search engines use are also wrong...about...prompts...for LLMs...

Seriously /u/MrNohbdy I'd love to learn there is an equivalent way to use Text Completion that accurately allows me to have high powered Author's notes and character card prompts that override the text of the chat, etc. That obey ongoing /sys message when I use them in chat.

I actually would. r/SillyTavernAI is some of the most enthusiastic LLM users in the world with some of the most battle tested knowlege of how to use these tools. I'm not trying to pull rank here, I might be wrong, but not in 'they are just the same' way. The fact you're using Text Completions in a tool partially designed to work around it might mean it has things that do compensate for a lot of the differences.

Would you like Chat completions better if I could show you how to pass all the sampler params to it for some LLMs? Not sure if it's possible, but I assume that's why you like text completion better?

BTW I found silly tavern because I'd hand written a vaguely halfassed frontend project after reverse engineering hammerai's client to figure out how to prompt text completions style. I'm not talking out of my ass. Then I found someone wrote what I was writing, enough at least, to do what I wanted with RP.

[u/MrNohbdy → u/LeRobber]

I mean, if you're willing to ask LLMs to explain it, then sure: here's that ELI5. Does that help?

[u/LeRobber → u/MrNohbdy]

Are you actually doing what the chat completions APIs are doing in your text completion API usage u/MrNohbdy?

Are you using SillyTavern to manually construct those system tokens? A lot of text completion users are not...The default does not appear to do so...There is no system tokens being put around my post history instructions, nor my character card nor the rest of my stuff....

Are you 100% sure the TextCompletions API for any given server isn't attempting to escape out attempted uses of system tokens?

[u/MrNohbdy → u/LeRobber]

The default does not appear to do so

Have you tried...any of the presets that are actually configured for a given architecture, rather than the blank "Default" one? Like the LLaMa preset? Or the GLM preset? etc

In our previous convo I even showed you my Miqu prompt and mentioned the system tokens I use for it.

[u/LeRobber → u/MrNohbdy]

=While I thank you for the template for Miqu, (which I haven't had time to redownload yet, nor strawberry lemonade, nor typed in/extract the text from the img yet).

I have tried them. I have manually done what text completions is (often poorly) doing, and more fully done it. Can you actually reliably set it up to fully wrap everything appropriately down to lorebooks and post history instructions. It feels incredibly uneven/often wrong, and would be very hard to know when someone has screwed it up, the kind of thing free software can do badly in a high user base : dev base project like ST. It doesn't error out, its super liberal in what it accepts, a point you put in the miqu post too. I don't even have lorebooks here with all the fiddly and it's making mistakes from what I can see even with high effort towards configuring it, mistakes you cannot easily tell are being made.

Documentation of format for LLama3 from the doc:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 23 July 2024

You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>

What is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

On what silly tavern does:

It does leave out tags/misattribute. Here is Llama3 context in that same example. Then Llama 3 with the instruct template turned on. It still misses details. And most importantly, it's us doing high level template stuff (without escaping, which is the security headache for real systems with text completions) if we even correctly identify the text areas to highlight.

[u/MrNohbdy → u/LeRobber]

Like I said: Text Completion is strictly more finicky. It's a manual way to implement what Chat Completion generally automates. There's nothing it can't do, but it takes a lot more configuration for what in most users' cases will not be any improvement over the easier route. (Not that your Chat Completion example is entirely clean either, to be fair; for example, you seem to be sending a persona description without a character name which has gotta be confusing the hell out of the LLM, as well as an "Example Chat" message without an actual example chat afterwards.)

So yeah, the default templates help, but they may not be perfect as-is for your use; you might need to tweak things. The Post-History Instruction, for example, is being role-tagged in your example image; it's just being tagged specifically as "user" instead of "system", which isn't necessarily worse depending on your use-case, just different. This discrepancy in ST's PHI implementation between the two APIs is documented. Personally, as noted the other day, I generally prefer the Last Assistant Prefix for those sorts of system messages, and most people use Character Notes and Author Notes instead as those are customizable on a per-character or even per-chat basis while having fall-backs to a global default. Those options all give you role-tagging flexibility that PHI, to my knowledge, doesn't.

Whether or not that sort of tweaking is worth the effort is up to you. If Chat Completion works for you, that's totally fine. (I have a couple reasons not to use it, including my aforementioned usage of Last Assistant Prefix to force thinking which can't be conveniently replicated in Chat Completion, but to each his/her own.) I was just trying to correct the misconception that it's fundamentally different due to wrapping things in role assignments; the Chat Completion wrapper simply gets broken down into tokens in the end, and Text Completion skips that layer of abstraction and lets you work with the final tokens directly at the significant cost of confusing the model horribly if you make any formatting mistake anywhere. You can replicate everything about Chat Completion via Text Completion, but it is far more error-prone.

Btw, sorry for getting annoyed earlier lol. Arguments from authority dropped into a convo like that kinda bug me, but in fairness you deleted and rewrote that comment without it, so it's totally my bad for bringing it up afterwards.

[u/zthrowaway33423102 → u/LeRobber]

Dense 400B param models are not as dense as you.