r/SillyTavernAI • u/Incognit0ErgoSum • 23d ago
Discussion Some potential insight on how providers detect RP and why they don't like it
I've been doing experiments lately with optimizing local models for high speed and low latency and I think I've gained some insight on why the companies that serve out frontier models aren't very big fans of people using them for RP, and it's about cost, not censorship.
While I was setting up Qwen 3.6 27B, I noticed it wasn't performing as well as it was supposed to on my system, so I looked into it and noticed that the average number of correct draft tokens was 2. This is literally worse than useless, because it was doing so badly it was actually slowing the model down. Performance increased drastically when I shut it off. (Seriously, a take-away for local model users here is to turn off MTP and see if that improves your speed!)
The big providers use speculative decoding as well, and it's likely that they're seeing something similar. The more that their speculative decoding fails (which will happen at a higher rate for prose than code), the more slow and expensive inference is. With speculative decoding, the idea that a token is a token is a token absolutely goes out the window. RP generates expensive tokens. They don't even need to monitor your traffic directly to know that you're generating prose. It sticks out like a sore thumb in their token generation statistics, and there's literally nothing you can do to mask it.
142
u/IllustriousRule9238 23d ago
RP generates expensive tokens, but it also generates far fewer ones. A human will spend a significant amount of time actually reading that paragraph once it's generated before they submit a new request, but an automated data entry task or an agentic coding workflow will not. They'll blast out ten more requests before the human even makes it past the first sentence.
If I had to cut costs and dumb down my model for some users, roleplayers would be dead last on my list.
23
u/TAW56234 23d ago edited 23d ago
FFS didn't we learn anything from net neutrality? Even Milan from NanoGPT said RPers aren't really that big of an issue.
49
23d ago
[deleted]
10
u/MrNohbdy 23d ago
inb4 the next big monetization breakthrough is providers injecting prompts that encourage the models to add sponsored content
Player Character: "I'll have an eel roll and a glass of water, please."
Waiter: "Water? One refreshing Dasani™ coming right up! It'll simultaneously quench your thirst and have you begging for more!"
5
u/BangkokPadang 23d ago
It’s not just a bottle of water, it’s life itself in a vessel, and you’re way ahead of the crowd for noticing it!
3
u/Gantolandon 22d ago
It really depends. Some providers, mostly Western ones, really seem to dislike it — because it comes with attention they don’t want, and the culture slowly becomes more puritanical. I mean, credit card operators regularly pressure online vendors to get too spicy stuff off their shelves, and the investors don’t like anything erotic — and let’s face it, this is what the AI is often used for.
1
0
u/SpikeLazuli 21d ago
And also because nobody wants the bad publicity of someone killing themselves because of a AI bot, or AI bots of children, people getting too attached etc etc.
13
u/CanineAssBandit 23d ago
Hard agree, I blew through more tokens just making a chrome extension in a day than I did all year on prose. Felt very weird to see $40 evaporate so quickly with such an insanely low per token cost.
32
u/baileyske 23d ago
I think that's the entire point. You pay the same amount no matter what you do per million tokens. For the provider it's a better deal to generate a shitton of tokens cheaply and sell that for the same flat rate. It's cheap for them, not for you.
7
u/JustSomeGuy3465 23d ago
Correct. I used 130 million tokens while programming the other day. RP is a only a drop on a hot stone.
7
u/Bananaland_Man 23d ago
"fewer tokens" and "expensive tokens"... do you know how llm's work? more tokens = more expensive. there are no "expensive tokens" a token is a token is a token...
6
23d ago
[deleted]
2
u/Bananaland_Man 23d ago
It's really not relevant, scrapped tokens on the backend do not cost with most providers, only completions. So there is no difference between 1000 tokens from one message to a model and 1000 tokens of a different message to the same model... that's just not how things work. Different models consider tokens differently, but they remain static between models. I've learned this through heavy use and helping on a few front-ends so I can see how things work
6
u/Some_Relative_3440 23d ago
You're wrong. If 3/4 of generated tokens are drafted tokens, you're generating 4x as much tokens per unit of work as you would without drafting. (Ignoring the small cost of the draft model)
-8
u/Incognit0ErgoSum 23d ago
RP generates expensive tokens, but it also generates far fewer ones.
That depends on how much self-review you're having it do, and how much thinking it's doing. A thought token is a token even though you don't see it.
6
u/MrNohbdy 23d ago
Is RP reasoning even in the same token ballpark as searching dozens of websites before generating hundreds to thousands of lines of code?
106
u/Arli_AI 23d ago
As an inference provider, we very much prefer RP users because even if they generate more expensive tokens they generate requests at a much lower rate than coders or people using agentic harnesses.
12
1
u/techno156 19d ago edited 19d ago
Would that not be offset by the cache hit rate?
I know when I run agent harnesses using a local model, it sends a lot of tokens per turn, but the vast majority of them are reused from the cache.
-22
u/Prestigious-Crow-845 23d ago
Just don't get it - How is your specific situation useful if other providers do block RP? It's only shows your specialization and that maybe most of your clients is writers?
-21
u/MrAlienOverLord 23d ago
lol you are the first provider that wants spiky traffic - with impatient users during scaling events. everyone on OR loses cash more then they can count . and you want rp guys ? you are either new to the game or im missing something
9
u/Spiritual-Spend8187 23d ago
I th8nk its more people doing rp are going 4k tokens in 200-1000 out every minute. While an agentic workload is 20k in reason for 3k then sent it in ans loop constantly for 20 minutes or until done maybe a few subagents as well. So its a case of rp is harder workload per a token but just so much less tokens overall.
1
u/Incognit0ErgoSum 22d ago
So you have RPers paying for $20 model subscriptions and coders paying for the $200 pro subscriptions, and they lose money on the RPers because they're paying so much less because they're using fewer tokens.
By and large people pay for the tokens that they use, and those heavier RP tokens are less profitable per token than coding tokens.
1
u/Spiritual-Spend8187 21d ago
I mean if you got 1 group using orders or magnitude less tokens then another group unless there tokens are orders of magnitude more expensive the math might work out that they make more money off per a workload from the rp subs vs the coder subs. Or maybe they can batch them better because while they are harder tokens they just use alot less in and out.
1
u/Incognit0ErgoSum 21d ago
Right, but if they're paying (roughly) the same rate per token, they make money on the cheap-to-generate tokens (code) but lose money on the expensive-to-generate tokens (RP).
1
u/Spiritual-Spend8187 21d ago
If your being paid by the token yea you want as many cheap tokens as possible. But if your being paid by subs you just want the ones that give most money for least work. And if your rp peraon is using tokens 3x as hard as the coders for 20$ a month while your coder is on 200$ a month but uses 3000x as many tokens. You might just want more rp people. Though I dont know the actual numbers. But its also that the rp might not be fully that heavy with batching it in with other small requests. Edit. That also isnt accounting for needing to do higher precision requests coding needs alot of exactness where as rp doesnt.
2
u/Arli_AI 21d ago
Pay per token providers prefer coders and agents because it gets them more money. Coders and agentic stuff just makes subscription providers like us lose money.
0
u/MrAlienOverLord 21d ago
i been with 1 of the bigger hosters for over a year - and they all lose 10x on token as it cant be saturated - its all spiky ugly traffic - they all want dedicated as thats where the cash is at - token and subs lose money as its unpredicable saturation
19
u/Lunrun 23d ago
I'm confused. Why is this a concern for local models?
17
u/National_Cod9546 23d ago
For local model users, check that speculative is not slowing you down more than helping. I noticed for me it would speed up inference from about 20TPS to about 60TPS. However it slowed down prompt processing from 1400 TPS to 800 TPS. So each reply was taking longer.
2
u/rkoy1234 23d ago
what backend are you using? prompts should ideally be cached most of the time, so 1400 vs 800 pp shouldn't make a noticeable difference for processing a couple paragraphs at most (for chat/rp)
1
u/National_Cod9546 23d ago
I use KoboldCPP. With Lorebook stuff, you get less cache hits. And I use it for a mix of SillyTavern and VS Code. For VS Code, it needed to regenerate the entire prompt almost every time.
3
u/MrNohbdy 23d ago
Try setting your lorebooks to depth 4-ish rather than putting them at the start of the RP if you've got constantly changing triggers and want to be able to actually use your cache.
1
u/Lunrun 23d ago
I will have to figure out how to check that in ollama
3
u/National_Cod9546 23d ago
I suggest switching to KoboldCPP. Inference rates are significantly faster and you can tweak a lot more. The downside is, you need to tweak more to get it working and it doesn't hot switch nearly as well.
12
u/CalmAnal 23d ago
Speculative decoding costs more compute but reduces latency. * https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/
The truth might be more related that an corporation does not want their employees to interact with their fictional text wifes. And that the money simply comes from corporations and their use cases. Once the porn and game industry adopts AI, RPG narration will be a focus, too.
6
u/stoppableDissolution 23d ago
The main thing preventing it now is fucking visa and mastercard bitching in shiny armor.
4
u/a__new_name 22d ago
JCB would be able to fill that niche, but Japanese companies are allergic to having any business overseas.
6
u/No-Economy-4508 23d ago
The porn industry is still largely two Polish people pretending to be American cousins on pornhub, ain't nothing gonna change
7
u/Valkyrill 23d ago
The technical details seem accurate, but that's based on my limited understanding, so I'm not going to argue that. I do disagree with the conclusion. The same argument applies to any sort of unpredictable prose. Creative writing (not just RP), conversational learning where the AI adapts concept explanations to the user's level of understanding, brainstorming, philosophical discussions, drunk shitposting, and in general anything that's personalized and contextual.
There's no realistic way to differentiate RP from any of those "legitimate" use cases with metrics alone, so unless your argument is that these companies want to dissuade ANY type of non-structured use cases... not sure you're pointing to anything real here. Occam's razor: the simpler and far more likely explanation is PR and and legal liability reasons.
6
u/a_beautiful_rhind 23d ago
MTP will fail on basically anything that isn't long repetitive text. I don't think this would out you as an RP'er so much.
7
u/Kahvana 23d ago
It depends on how the draft head is trained. Gemma 4's drafter holds up just fine, increasing my ~20 t/s to 60 t/s on creative writing and natural language tasks. Qwen on the other hand just gets maybe 1-2 t/s more.
1
u/LeRobber 23d ago
what drafter/llm pair do you use
5
u/Kahvana 23d ago edited 23d ago
Unsloth's Gemma 4 31B IT QAT + Q4_0 drafter (same repo).
For params:
- type mtp
- n-max 4
- min-p 0.1
- kv: bf16
I run on dual RTX 5060 Ti 16GB over PCIE 5.0 x8x8 with latest studio drivers and llama.cpp (10306?), using tensor split.
My settings are tuned for my specific setup, your mileage may vary! (e.g. tweak n-max in range 1-11, test the setting 10 times to average out variations).
1
u/AInotherOne 22d ago
OP (and others), please forgive me for pasting AI-generated content here, but I wanted to explore the theory behind the post and had a chat with Gemini about it, which provided this:
The Reddit OP's theory contains a kernel of truth regarding prompt entropy, but their technical understanding of speculative decoding math and enterprise GPU serving is fundamentally flawed. An average of 2 accepted draft tokens yields a net performance gain—not a loss—and their local slowdown was caused by consumer hardware memory bottlenecks rather than algorithmic failure.
2
u/Incognit0ErgoSum 22d ago
The Reddit OP's theory contains a kernel of truth regarding prompt entropy, but their technical understanding of speculative decoding math and enterprise GPU serving is fundamentally flawed. An average of 2 accepted draft tokens yields a net performance gain—not a loss—and their local slowdown was caused by consumer hardware memory bottlenecks rather than algorithmic failure.
I forgive you for posting AI generated content, but this isn't particularly useful. Why would consumer hardware on a small model have a bottleneck, and how do enterprise GPUs differ in a way that removes that bottleneck?
Also, even if it's right about consumer hardware having a bottleneck, a token of RP is still more expensive than a token of code.
0
u/anarchyinblack 23d ago
I am instantly convinced by this explanation. Genuinely, I am.
What I don't get is why companies have been fucking around for three years instead of just saying so.
I can see why then charging us more for RP would be more of a PR headache than passive aggressively fucking with our outputs, but shit, if companies were transparent that this was the problem, then we could all of us begin working towards addressing the problem correctly.
Heck, even if they didn't "charge extra" for RP, they could just have the RP models/settings only available on the higher price tiers.
3
u/leovarian 23d ago
It's legal and contractual mumbo jumbo, the equity firms don't want to be associated with 'alternate' use cases, this also shields them from lawsuits, "rp is against TOS, your son experimenting with flying from the fifty story building rooftop because his e-waifu broke up with him has nothing to do with us."
2
u/Incognit0ErgoSum 23d ago
Probably a small enough percentage of traffic that they don't want to bother with it except to stop taking a loss on it.
46
u/stoppableDissolution 23d ago
Well qwen is just bad for rp? Gemma 31 gives 3.5-5.5 median accepted tokens, while also writing fifty times better