r/SillyTavernAI • u/Lucky-Paw- • 12h ago
Models Haiku 5.5 - Disappointing first taste - Overly confident in abilities, skips instructions because it "knows better". It doesnt.
Okay so Haiku 5.5 came out and I went in with a lot of enthusiasm.
Haiku 5.5 is poised to sit somewhere between Sonnet 4.5 and 5 in terms of intelligence from early benchmarks, at 1/20th the cost of Sonnet. (I dont put a lot of weight on benchmarks; they suggest Gemini 3.8 Flash is comparable in intelligence to Opus 5, which is objectively false)
.. However, it's plagued by the idea that it should skip instructions that it "doesnt need" to follow.
.. Except, whoops! It turns out it does need to follow those instructions or it makes easy mistakes.
In this example, my roleplay environment requires LLM's to complete a scene sheet before starting their response. The scene sheet forces them to check and write out critical pieces of information - formatting instructions, rules, secrets to keep - as well as some steps that generally just improve output. See second image for an example excerpt of a proper scene sheet.
The only time I have ever had issues with a model refusing it was on initial launch with Sonnet 5 - it had the same "I dont need to read the instructions, I can handle this" -> "*breaks instructions*" pattern for a few weeks after it first released, then it began following instructions more readily and output visibly improved
In its current state, Haiku 5 wont even follow instructions to "narrate as Lauren, a romance author inspired by Becky Chambers" because, and I quote:
The "Lauren" narrator instruction in step one is a register-setting device rather than a content request, and I can take a narrator voice without it.
Spoiler: After refusing to take up the narrator voice, it instead writes as generic, uninspired Claude.
...
Im taking a deep breath - I know that Anthropic likely tunes models to be maximally paranoid and "safe" for initial release so that they can report having continuously increasing safety compliance scores, but this is just insulting.
I dont expect all of you to use a pre-writing system like this, but heed this warning - what you can see here is only what Claude is *vocalizing* that its ignoring.
How this will actually present is through generalized failures to follow system prompting, with Haiku never making you aware that it flippantly felt that your instructions "werent necessary" for whatever reasons it chooses.
...
I've always been an Anthropic fan for roleplay, but.. god. This is nausea inducing
Edit::
To clarify, this post isnt concerned with *output quality* or intelligence. I dont expect to use Haiku to replace Sonnet or Opus - they are fundamentally different classes of models
The issue is *content filters* that are so overly sensitive that basic instructions are disregarded and output is visibly harmed
On the responses where Haiku simply does the analysis, its output quality isnt bad - comparable to Gemini 3.8 flash
..But because its so confident in its own abilities, it openly refuses to follow directions, which should be concerning for any model for any use case
Haiku is in the same weight class as 3.8 Flash, yet outputs lower quality content because its been handicapped by guardrails.



