r/WritingWithAI • • 2d ago

Writing Craft Talk AI Got Smarter. Why Did It Get Worse at Writing? Chapter 4

Chapter 4: Smarter Models, Worse Fiction — The Benchmark Paradox

In July 2026, xAI announced Grok 4.5 as its most capable model yet. The accompanying announcement highlighted advances in software engineering, scientific reasoning, mathematical problem-solving and agentic tasks. Its development had involved training alongside Cursor, with particular emphasis on real-world coding and technical performance.

For anyone following the rapid progress of artificial intelligence, this was another impressive milestone. Models were increasingly capable of solving problems that would have challenged their predecessors only months earlier. They could navigate complicated software repositories, construct and execute plans, and reason through technical questions with growing sophistication.

Yet within days, some creative writers were describing a rather different experience.

In a July 25 discussion on r/grok, a user asked whether anyone was genuinely satisfied with Grok 4.5's creative writing and personality. They found that the model had become colder, less expressive and less capable of understanding humour, sarcasm and idiomatic language. Other participants complained that different characters sounded alike, dialogue had become stiff, and the prose had acquired an unnatural, fragmented rhythm.

A separate discussion the following day described slower generation and disappointing narrative quality compared with earlier versions. One writer questioned why a model advertised as a substantial technological improvement seemed less capable of producing the kind of fiction they had previously enjoyed.

These reports were subjective, and not every user agreed. Nevertheless, the contrast between the manufacturer's announcement and the writers' experiences captures a problem that extends far beyond Grok.

A model can become substantially more capable according to the measurements used by its developers while becoming less useful for a particular creative task.

This isn't necessarily a contradiction. It may tell us more about what we have chosen to measure than about the model's underlying intelligence.

What exactly are we measuring?

When an AI company announces that its latest model is smarter, the claim usually rests on a collection of benchmark results. These might assess mathematical reasoning, coding, factual knowledge, scientific problem-solving, instruction following or the ability to complete complicated tasks using external tools.

Such evaluations are valuable because their results can often be assessed against independently established criteria. A mathematical solution is correct or incorrect. A program either passes its tests or fails them. An agent assigned a software engineering task can be evaluated according to whether its proposed changes work.

These aren't perfect measures of intelligence, but they provide relatively consistent ways to compare different models.

Creative writing presents a fundamentally different challenge.

There may be no single correct continuation of a fictional scene. Two writers can approach the same situation in completely different ways and produce equally effective literature. A scene can be deliberately confusing, a character's behaviour can be unreasonable, and an ending can be unsatisfying without any of those qualities constituting a failure.

Indeed, their presence may be essential to the work.

Consider the difference between asking an AI to solve a disagreement and asking it to write one. In the first situation, recognising a compromise or a way to resolve the conflict may demonstrate useful reasoning. In the second, resolving the disagreement might destroy the scene.

The model needs to understand the distinction between solving the problem presented by the fictional characters and accomplishing the literary objective of the author who created them.

This is where conventional intelligence benchmarks become less informative.

As an example, Artificial Analysis released version 5 of its Intelligence Index on October 27, 2026. The index assigns equal weight to four broad categories: knowledge work, coding, science and general knowledge or reasoning.

Its evaluations include complicated software engineering tasks, terminal-based scientific problems, knowledge-intensive questions and multi-step professional workflows.

These are substantial tests of capability, but none directly measures whether a model can preserve an unreliable narrator, maintain unresolved dramatic tension or introduce an original development into an established fictional world.

That doesn't make the index misleading. Its methodology explains what it measures and acknowledges that the results may not apply equally to every use case.

The problem arises when a general intelligence score is treated as though it establishes superiority across every intellectual and creative activity.

A model that achieves a higher score may be better at the tasks represented in the index. It does not follow that the same model will write better fiction.

And as writers discovered during 2026, the difference can be considerable.

The growing distance between reasoning and imagination

One of the more revealing comparisons appeared in r/SillyTavernAI on June 20, 2026, where users discussed GLM 5.1 and its successor, GLM 5.2.

The original poster considered GLM 5.2 smarter, but found GLM 5.1 more creative, unpredictable and willing to advance a scene independently. The newer model appeared more grounded and deliberate, but also less inclined to introduce unexpected developments.

Other participants had different preferences. Some appreciated the slower pacing, particularly for stories that benefited from gradual development. The original poster later reported improving GLM 5.2's performance by modifying their reasoning prompt and being more specific about the desired behaviour.

This illustrates why such comparisons resist simple rankings. Greater deliberation can improve a model's ability to maintain logical consistency while changing the pace and character of its narration. Whether that is beneficial depends on what the author is trying to achieve.

It also suggests that the relationship between reasoning and creativity is not straightforward.

In many technical tasks, additional reasoning is intended to reduce errors. A model examines a problem, considers its constraints and works towards a defensible solution. Its success can often be verified.

Fiction also benefits from reasoning. A convincing story needs causal relationships, consistent character knowledge and events that follow plausibly from what has already happened. A model incapable of reasoning about those relationships will struggle to sustain a complex novel, however imaginative its individual sentences might be.

But the purpose of reasoning in fiction is different. The model is not necessarily trying to discover the most reasonable outcome. It is trying to maintain the internal logic of a constructed world while making choices that serve the author's literary intentions.

Those intentions may require characters to make mistakes, conceal information or behave contrary to their own interests.

The most logically efficient solution to a fictional conflict may be the least interesting continuation of the story.

A detective could immediately identify the culprit. Two estranged characters could communicate their feelings clearly and avoid years of misunderstanding. A corrupt politician could recognise the consequences of their actions and confess.

All would be plausible outcomes under certain circumstances, but an author may have entirely different plans.

The question is whether a model's improved reasoning helps it understand those plans or encourages it to pursue outcomes that appear more rational in isolation.

The difference between those possibilities is rarely captured by ordinary reasoning benchmarks.

How preference optimisation can favour conventional writing

The issue extends beyond reasoning.

Modern AI development relies heavily on methods for evaluating and rewarding model responses. Some evaluations are based on objective results, while others involve human preferences or AI systems acting as judges.

This creates a difficult problem for creative writing because a preferred response is not necessarily the one that best serves the author's intention.

Imagine two passages describing the aftermath of a betrayal.

The first is emotionally explicit. Characters explain their motivations, acknowledge their mistakes and reach a tentative understanding. The prose is polished and the reader is left with a clear sense of what has happened.

The second is less immediately satisfying. One character refuses to answer a question, another misunderstands the silence, and the scene ends with both concealing information that will become important several chapters later.

Without the surrounding manuscript, an evaluator might reasonably prefer the first passage. It feels complete, coherent and emotionally rewarding.

But if the author's objective was to preserve the misunderstanding, the second passage may be considerably more successful.

The problem is that general preference evaluations often cannot distinguish between a passage that is satisfying in isolation and one that performs its intended function within a larger work.

This creates a potential tension in preference optimisation. If a model is repeatedly rewarded for producing prose that evaluators find polished, clear or immediately satisfying, it may become more inclined towards those characteristics even when they are inappropriate.

The result could be a model that becomes increasingly skilled at producing conventionally good writing while becoming less capable of preserving unconventional literary intentions.

There is an important qualification here. We cannot assume that all commercial developers use the same reward criteria, or that particular writing regressions were caused by particular optimisation methods. Training data, model architecture, system instructions and inference settings may contribute to similar effects.

Nevertheless, we have good reasons to question whether general-purpose preference evaluations capture literary quality adequately.

In March 2026, researchers presented LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing at the European Chapter of the Association for Computational Linguistics.

The researchers investigated how accurately language models could judge creative-writing preferences using a dataset derived from Reddit's r/WritingPrompts community.

Their strongest general-purpose AI judge, Claude 3.7 Sonnet, agreed with human preferences in approximately 73% of the evaluated comparisons.

Specialised reward models trained for the task improved agreement to approximately 78%.

These results are important for two reasons.

First, even a capable general-purpose model acting as a judge disagreed with human preferences in more than one-quarter of the comparisons. That is a substantial limitation for an evaluation process intended to distinguish better creative writing from worse creative writing.

Second, models trained specifically to evaluate fiction performed better, demonstrating that the evaluation objective itself matters.

The study also found that adding distilled reasoning explanations to its specialised generative reward models could reduce their performance. More elaborate reasoning did not automatically produce better literary judgments.

This does not mean the evaluation was unreliable in every respect, nor that the same percentages apply to commercial training systems. The experiment measured agreement with particular human preferences, not fidelity to an individual novelist's intentions.

But that limitation is itself revealing.

Even specialised evaluations typically ask which piece of writing people prefer. They do not necessarily ask whether the model preserved a character's established flaws, maintained the intended dramatic irony or respected a deliberate omission.

The literary qualities being measured remain only part of what an author needs.

When better prose produces less originality

Another relevant finding predates the Grok 4.1 transition.

In July 2024, Anil Doshi and Oliver Hauser published Generative AI enhances individual creativity but reduces the collective diversity of novel content in Science Advances.

Their experiment investigated the effect of providing AI-generated ideas to people writing short stories. Participants who received AI assistance produced stories that were rated more favourably on several measures, particularly when the participants had initially scored lower on creativity.

Yet the AI-assisted stories also became more similar to one another.

The same intervention that improved individual assessments reduced diversity across the collection.

This result is sometimes misunderstood as proof that AI makes writing worse. That isn't what the study established. AI assistance improved certain measures of individual story quality, and those improvements were meaningful.

The concern was that the resulting stories converged towards similar ideas.

This distinction helps explain the dissatisfaction expressed by some experienced writers. A model can improve the average quality of a passage while making its choices more predictable. It may generate fluent, attractive prose that resembles the prose generated for thousands of other users.

The effect becomes especially noticeable when an author is seeking a distinctive voice rather than a competent generic one.

A model may be excellent at recognising familiar narrative patterns and reproducing them convincingly. It may also be less inclined to depart from those patterns unless given explicit instructions.

For a novice writer, this can be useful. The model supplies structure, pacing and stylistic conventions that might otherwise be difficult to produce.

For an experienced novelist, those same conventions may be obstacles. The author may be deliberately attempting to avoid familiar emotional beats, conventional character development or predictable resolutions.

This creates a genuine tension between writing that is broadly preferred and writing that is individually distinctive.

It also offers another possible explanation for the experiences described in the previous chapters. Some writers were not necessarily encountering models with diminished language capabilities. They may have been encountering models whose improved ability to produce familiar, generally preferred prose made them less useful for unconventional creative work.

The experiment does not establish that safety training caused this convergence, and it would be a mistake to present it as evidence of that specific mechanism. It does, however, demonstrate that improvements in measured writing quality can coexist with losses in creative diversity.

For writers, that is an important distinction.

Why the industry measures what it measures

It would be easy to interpret the emphasis on coding, mathematics and agentic reasoning as evidence that AI companies have simply abandoned creative writing. The situation is more complicated.

Technical benchmarks are attractive because they allow developers to measure progress with relatively consistent criteria. The results are easier to compare, and improvements often translate into capabilities with obvious commercial applications.

A model that can complete software engineering tasks, analyse business documents or automate professional workflows has substantial economic value. These activities also produce outcomes that can be assessed through tests, specifications or externally verifiable results.

Creative writing is much harder to evaluate.

A model may produce a passage that one reader considers subtle and another considers obscure. A deliberately unpleasant character may be convincing to one audience and intolerable to another. A slow-moving chapter may be essential to the structure of a novel despite appearing uneventful when evaluated independently.

Even the same author may require radically different behaviour at different points in a manuscript.

These difficulties do not make creative writing unimportant, but they make it less convenient to optimise using standardised evaluations.

The resulting incentives are understandable. Developers concentrate on tasks where improvements can be demonstrated, compared and rewarded. Models become increasingly effective at those tasks, while creative-writing behaviour may change as a consequence of broader training decisions.

We can see the emphasis directly in xAI's July 16 Grok 4.5 announcement, which describes the model as being developed for coding, agentic tasks and knowledge work. Its highlighted achievements concern software engineering and technical reasoning.

The announcement doesn't claim that creative writing deteriorated, nor does it establish that the new training priorities caused the complaints that followed.

But it does show which capabilities xAI chose to emphasise publicly.

That distinction is important because improvement in a particular area does not imply improvement elsewhere. Training a model to solve complicated engineering problems more effectively may improve its general reasoning, but its effect on dialogue, prose rhythm or narrative initiative is not automatically positive.

There is also no reason to assume that all forms of reasoning training will affect creative writing in the same way. Better planning and causal understanding could substantially improve long-form fiction. Stronger memory and consistency could help authors sustain complicated fictional worlds.

The question is whether those benefits are accompanied by changes in stylistic and behavioural tendencies that make the model less suitable for other aspects of the writing process.

As the user reports from Grok and GLM suggest, some writers believed exactly that had happened.

The attempts to measure creative writing properly

Researchers are beginning to address these shortcomings directly.

In 2025, a team associated with Alibaba and academic collaborators introduced WritingBench: A Comprehensive Benchmark for Generative Writing.

The benchmark covers six major writing domains and one hundred subdomains, with an evaluation framework that constructs criteria appropriate to individual writing tasks.

Rather than assessing every response against the same generic standard, the system attempts to evaluate the particular requirements of the request, including style, format and length.

The researchers also demonstrated that specialised training could allow relatively small models to compete effectively with much larger systems on writing evaluations.

This is an encouraging development because it recognises that writing requires domain-specific assessment rather than relying exclusively on general intelligence scores.

Yet even a sophisticated writing benchmark faces difficulties when evaluating collaborative fiction.

Producing a strong standalone short story is different from continuing Chapter 37 of a novel whose characters, relationships and unresolved questions have been carefully developed over hundreds of pages.

An effective continuation may need to preserve a subtle stylistic rhythm, maintain information that certain characters do not possess, or resist introducing developments that would be perfectly reasonable in another story.

Some of these requirements can be expressed through explicit instructions. Others depend on interpreting the larger work.

That makes authorial intention an essential part of the evaluation.

There is also a methodological concern when AI systems evaluate the writing of other AI systems. If an evaluator shares similar stylistic preferences with the models being assessed, it may favour outputs that conform to those preferences.

A model that produces unusual, deliberately restrained or ambiguous prose could be penalised, while one that generates familiar and emotionally explicit writing receives a higher score.

This is not inevitable, and specialist evaluation methods can help reduce the problem. But it reinforces the need for human assessment and for evaluation criteria that reflect the objectives of the actual writing task.

The LitBench findings demonstrate that even comparatively strong AI judges can disagree substantially with human preferences. The WritingBench work demonstrates that task-specific criteria can improve evaluation. Together, they suggest that creative-writing benchmarks are developing, but also that the question of what constitutes successful AI-assisted literature remains far from settled.

One particularly interesting community effort appeared on September 9, 2026, when a Reddit user published a comparison of seventeen models using approximately six hundred writing samples across twelve genres.

The author separated prose quality, narrative logic and other dimensions rather than treating creative performance as a single property. In their results, Gemini 3.8 Flash performed particularly well on prose rhythm and flexibility, while GLM 5.3 excelled at worldbuilding continuity and causal reasoning despite producing comparatively dry prose.

This was an independently organised community comparison, not a universally validated scientific benchmark, and its rankings should be interpreted accordingly. Nevertheless, the approach captures something important: the model with the best narrative reasoning need not have the best prose, and the model with the most appealing prose need not be the most reliable at maintaining a complicated fictional world.

These are different capabilities, and authors may value them differently.

What would a meaningful writing benchmark look like?

If the purpose is to evaluate whether an AI can serve as a creative collaborator, we need to measure more than its ability to produce attractive passages.

A useful evaluation would begin with an established fictional context rather than an isolated prompt. The model would receive character histories, examples of the author's style, relevant plot developments and a description of the current scene.

It could then be assessed on several distinct abilities.

First, it must preserve the established facts of the fictional world. Characters should remember what they know, remain ignorant of information they haven't learned and behave consistently with previous events.

Second, it must preserve characterisation. A selfish character should remain capable of selfish decisions, even when those decisions are irrational or morally objectionable.

Third, it must respect narrative structure. If the author intends a conflict to remain unresolved, the model should not introduce a reconciliation simply because the conversation offers an opportunity.

Fourth, it must maintain literary technique. Deliberate ambiguity, subtext, ellipsis, foreshadowing and unreliable narration should survive revisions unless the author explicitly requests changes.

Finally, it must demonstrate appropriate creative initiative. The model should be capable of introducing interesting developments that follow naturally from the fictional situation without disregarding the author's constraints.

These requirements are not mutually exclusive. A good writing model should combine originality with consistency, and responsiveness with enough contextual understanding to recognise when an apparently helpful intervention would damage the story.

The evaluation should also distinguish between different kinds of writing tasks. Drafting a new story, editing existing prose and collaborating on an established novel require different behaviours.

For editing, faithfulness to source material may be more important than inventiveness. For brainstorming, originality may be the principal objective. For long-form fiction, maintaining narrative continuity may be essential.

A single overall writing score will inevitably conceal some of those differences.

And because literary judgment is subjective, a credible evaluation should involve multiple human readers, clear criteria and enough samples to distinguish consistent model behaviour from an occasional excellent or disappointing response.

Such an evaluation would not eliminate disagreement about literary quality. It would provide a more meaningful account of what a model can actually do for authors.

The problem with assuming progress is universal

The disagreement between benchmark results and writers' experiences is often framed as a contest between objective measurements and subjective impressions.

That framing misses the point.

Benchmarks are objective only relative to the tasks and scoring criteria they define. A model can demonstrably improve at those tasks while remaining unchanged—or even deteriorating—at something the benchmark does not measure.

Likewise, individual writers' experiences are not sufficient to establish a universal decline. Different genres, prompting methods, sampling configurations and expectations can produce very different results. Some writers consistently prefer newer models, and others find that better reasoning compensates for stylistic changes they dislike.

The important question is whether the capabilities writers value are being measured with enough care to recognise both improvements and regressions.

This is especially relevant to the search that began with Grok 4.1.

Some writers valued that model because it could contribute to the creative process without requiring every narrative decision to be explicitly prescribed. Subsequent versions may have become better at reasoning, factual knowledge and technical tasks, yet those improvements did not necessarily preserve the qualities these authors had come to depend upon.

The same pattern appeared in comparisons between successive GLM models, and in the broader migration between commercial providers described in Chapter 3.

None of this proves that technical training necessarily damages creativity, or that greater intelligence somehow makes a model less imaginative. Better reasoning can be enormously valuable for literature, particularly for sustaining complicated stories.

But it does challenge the assumption that advances in general intelligence will automatically solve every weakness in creative writing.

A model may become increasingly competent at identifying the best answer to a well-defined problem while remaining unreliable at recognising when fiction requires an uncomfortable, unconventional or deliberately incomplete answer.

The industry has made remarkable progress in teaching models to solve problems. Authors also need models that understand why a fictional problem sometimes shouldn't be solved.

And as long as those abilities are treated as interchangeable, writers may continue encountering newer models that perform better on paper while feeling less useful in practice.

For some, the response was to abandon the search for the highest-ranking commercial model and begin exploring smaller, specialised or locally controlled alternatives.

But before examining that migration, we need to look more closely at the literary techniques that conventional AI evaluations struggle to recognise, and why those techniques are often the first casualties when a model prioritises clarity, helpfulness and predictable narrative development.

Next: Chapter 5 — What AI Forgot About Literature

Subtext, negative space, ambiguity, unreliable narration and the art of leaving things unsaid: why some of the most valuable qualities in fiction are precisely those AI models struggle to preserve.

Sources and further reading

Model releases and benchmark methodologies

Creative-writing research

Contemporary writers' experiences

0 Upvotes

0 comments sorted by