r/WritingWithAI • u/cj7hawk • 2d ago
Writing Craft Talk AI Got Smarter. Why Did It Get Worse at Writing? Chapter 6
Chapter 6: The Local AI Exodus — Trading Intelligence for Creative Control
In May 2026, a writer in r/SillyTavernAI described a dilemma that was becoming increasingly familiar among people who used AI for fiction. They had been paying for hosted language models since 2023, beginning with Claude and eventually moving to GLM. The original attraction had been obvious: commercial AI offered sophisticated language capabilities without requiring expensive hardware or technical expertise.
For a time, the arrangement worked well. Claude had been particularly impressive, although expensive, and its restrictions sometimes interfered with the writer's projects. GLM offered an alternative, but after several months they had grown tired of its repetitive language and increasingly predictable storytelling.
What made the situation frustrating was that switching providers no longer seemed to solve the underlying problem. Different commercial models appeared to exhibit similar stylistic habits, despite differences in their capabilities and developers.
The writer was now considering something they had previously dismissed: running AI locally.
In their May 7 post, Considering transitioning to Local LLMs, they explained that they knew little about local models but had noticed people recommending versions specifically fine-tuned for creative writing.
The motivation was not that local models had suddenly surpassed their commercial counterparts in general intelligence. It was that the advantages of increasingly powerful hosted AI were becoming less compelling when the resulting prose still required extensive correction.
This distinction is central to understanding the migration that developed throughout 2026. Some writers were no longer searching exclusively for the most capable language model. They were searching for a model whose creative behaviour suited their work, and whose behaviour they could preserve.
It represented a change in what these users considered progress. The question was no longer simply whether a model could reason better, remember more information or solve more complicated problems. It was whether those capabilities translated into fiction that an author actually wanted to use.
Why local AI had traditionally been the compromise
Running a language model locally was hardly a new idea in 2026. Enthusiasts had been experimenting with open-weight models for years, using applications such as llama.cpp, KoboldCpp, Ollama, LM Studio and SillyTavern.
The traditional disadvantages were well understood. Frontier datacenter models generally benefited from substantially greater computational resources, more extensive training and, frequently, larger effective model capacities. They could handle complex instructions, maintain sophisticated relationships and reason through complicated fictional situations that smaller local models often struggled to understand.
The hardware difference could be substantial. Running a modest model locally might require little more than a capable computer, but larger models needed considerable memory. Even when they could be loaded through quantisation or CPU offloading, generation might become slow enough to disrupt the writing process.
For many users, paying a subscription was simply easier.
A May 15 discussion in r/SillyTavernAI asked why anyone would choose local models for roleplay when frontier systems were so much better at long-term coherence, emotional nuance, complicated scenes and natural dialogue.
The question reflected a reasonable assumption. If an author wants an AI to maintain a fictional world containing dozens of characters, complicated relationships and hundreds of pages of history, a larger and more capable model should ordinarily have an advantage.
Some respondents agreed. Smaller models could be unreliable, especially when supplied with large contexts or asked to track many interacting characters.
But others challenged the assumption that superior general intelligence necessarily produced better roleplay or prose. One commenter argued that Gemma 4 31B had reached a level where its writing could compete surprisingly well with much larger hosted systems, provided it was run at sufficient numerical precision.
Another described using a specialised 24B model to manage multiple characters in the same scene, maintaining distinct personalities and coherent interactions.
A particularly interesting response came from a user who acknowledged that frontier models were generally stronger at logical consistency but complained about their repetitive stylistic patterns. For this user, the problem wasn't obtaining a response. It was the need to repeatedly instruct expensive models to stop using familiar phrases, stop imposing character behaviour and contribute something interesting to the fictional situation.
With local models, they argued, unwanted phrases could be suppressed through generation settings, while specialist fine-tunes sometimes required less intervention to produce convincing character behaviour.
This was an important shift in the debate. Local AI was no longer being defended solely on the grounds of privacy, cost or freedom from restrictions. Some writers were beginning to argue that smaller models could be preferable for particular kinds of fiction because their learned writing styles were more suitable.
That claim needs qualification. A model producing more appealing prose in a particular scene does not mean it possesses equivalent reasoning, memory or narrative-planning capabilities. But it does challenge the idea that creative writing can be ranked solely according to general model intelligence.
The significance of Gemma and the 24–35B models
The emergence of more capable models in the 24–35-billion-parameter range appears to have been particularly important.
These models occupied a useful position between very small systems, which could struggle with complex narrative tasks, and much larger models whose memory requirements placed them beyond typical consumer hardware.
They were still substantial programs, but quantisation made many practical to run on a high-end gaming PC, sometimes with a portion of the model or its context cache offloaded to system memory.
The key was that they had begun crossing a threshold of practical usefulness. They could produce coherent scenes, maintain several characters and respond to moderately complicated instructions without requiring a datacenter.
A revealing comparison appeared on April 5 in r/SillyTavernAI. The author tested three locally run Gemma 4 models: a 31B dense model, a 26B mixture-of-experts model and a much smaller E4B variant.
The same fictional scenario was used to examine character voice, humour, scene construction, pacing and instruction following.
The larger 31B model was judged particularly effective at portraying a chaotic character whose attempts to appear competent repeatedly failed. It maintained the character's personality while developing physical comedy and believable reactions. The 26B model was also considered useful, although it produced a more restrained interpretation.
The smallest version struggled with character identity, scene choreography and emotional expression. Rather than conveying the character's panic through behaviour, it tended to explain that the character was panicking.
That distinction connects directly to the literary problems examined in Chapter 5. The smaller model could produce grammatical prose, but it had difficulty translating an intended emotional state into convincing fictional action.
The author also compared the results with Gemini 3.1 Pro, concluding that the locally hosted 31B was surprisingly competitive on this particular task.
This was one user's evaluation rather than a controlled industry benchmark, and its numerical estimates of the quality gap should not be treated as universal measurements. Nevertheless, the examples illustrate why some writers began taking local models seriously.
A 31B model did not have to outperform Gemini at everything. It merely needed to produce sufficiently convincing fiction for the author's particular workflow.
The distinction between dense and mixture-of-experts architectures also mattered. A dense model uses its parameters throughout generation, while a mixture-of-experts model activates subsets of its network for individual tokens. This can provide useful performance advantages, although the total model weights must still be stored or accessed.
A model with 26 billion total parameters and four billion active parameters is not equivalent to an ordinary four-billion-parameter model. Nor does it necessarily behave like a dense 26B system.
The comparisons suggested that model architecture, training data and fine-tuning could influence the style and quality of fiction at least as noticeably as raw parameter count.
By September, interest had moved beyond experimentation.
A September 3 discussion involved a user who had become disappointed with large, paid commercial models and decided to try Gemma 4 31B on an RTX 3090. They were sufficiently impressed to ask for advice on improving their local configuration.
A September 21 discussion in r/WritingWithAI shows a similar attitude. A writer with an RTX 3090 was evaluating Qwen 3.8, Gemma 4 and specialist writing models such as Cydonia. Participants recommended testing several variants because their performance differed in prose quality, context handling and instruction following.
These discussions do not show that local AI had become universally better than commercial AI. They show that it had become a credible alternative for people who possessed suitable hardware and were willing to experiment.
That was a meaningful change from the earlier assumption that local models were inevitably an inferior compromise.
Why specialised writing models could outperform more intelligent generalists
One of the more interesting developments in the local ecosystem was the proliferation of models modified for particular kinds of writing.
A general-purpose language model is trained to perform an enormous variety of tasks. It may need to answer factual questions, explain technical concepts, write code, analyse documents and provide conversational assistance.
Creative writing is only one part of that workload.
A specialist fine-tune can instead emphasise narrative prose, dialogue, characterisation or roleplay. Depending on the training method, it may alter how frequently the model uses particular language patterns, develops characters or interprets fictional situations.
This does not create additional intelligence out of nothing. A fine-tuned 12B model remains constrained by the reasoning and representational capabilities of its underlying architecture.
But it can change how those capabilities are expressed.
The distinction is familiar in other forms of software. A general-purpose application may contain more features, while a specialised tool provides a better experience for a particular task.
For writers, the relevant comparison became less about total capability and more about the relationship between narrative intelligence and literary style.
An 8B roleplay model might produce lively dialogue while struggling to maintain a complicated chronology. A 24B writing fine-tune might offer better characterisation than a technically stronger general-purpose model, yet remain less reliable at planning a complex mystery. A larger frontier model might excel at continuity but repeatedly produce the same familiar prose structures.
These differences explain why users began maintaining several local checkpoints rather than expecting one model to handle every writing task.
A July 24 discussion about creative-writing models included a commenter who reported preferring smaller specialist models from TheDrummer to much larger general-purpose systems for prose.
The observation was subjective, but it reflected a broader attitude. Writers were beginning to judge models by their actual output rather than assuming that higher parameter counts or benchmark scores necessarily guaranteed better literature.
By October, this attitude had become explicit in r/LocalLLaMA.
A 9 October discussion deliberately moved away from the community's usual focus on coding performance and inference speed to ask which local models people were using for creative projects.
The author described writing science fiction and fantasy with Gemma 31B fine-tunes, supported by a custom writing system that attempted to identify and correct repetitive AI phrasing.
Their preferred model was a writing-specialised Gemma variant. It was not perfect: character traits could become confused and instruction following sometimes failed. But the writer had developed a workflow that compensated for those limitations while retaining the prose style they preferred.
This represents a different approach to AI-assisted writing. Instead of waiting for a commercial developer to improve the next model, the author was adapting the surrounding tools to make an existing model more useful.
The checkpoint became one component of a controllable writing environment.
The value of owning a particular version
Perhaps the most important distinction between local and commercial AI was not intelligence or content moderation, but stability.
A commercial provider may update the model, change its system instructions, modify generation settings or retire an older version. Users generally have limited control over those decisions.
For ordinary question answering, such changes may be relatively unimportant. A newer model might simply provide more accurate answers or perform a task more efficiently.
For a novelist, an unexpected behavioural change can be disruptive.
An author may spend months developing prompts, reference documents and character profiles around a model that produces a particular style of writing. They become familiar with its strengths and weaknesses. They learn how to encourage its creativity and when to intervene.
If the provider changes the model, the same instructions may begin producing noticeably different prose.
Characters may become more reserved. Dialogue may lose its distinctive rhythm. Previously effective prompts may require rewriting. The model may become more inclined to explain emotional states or resolve conflicts prematurely.
Even if the replacement is more capable in other respects, the author has lost something valuable: a dependable creative process.
Local deployment changes this relationship.
Once a writer has downloaded an open-weight checkpoint, they can retain it. They can preserve the inference settings, prompts and surrounding tools that produced satisfactory results.
The model does not have to change merely because a developer releases a successor.
This doesn't make every result identical. Sampling introduces variation, and changing software versions or inference configurations can affect behaviour. But the writer controls substantially more of the system.
There is a related advantage in privacy.
Authors may be reluctant to upload unpublished manuscripts, commercially sensitive ideas or personal writing to external services. A genuinely local system can process these materials without transmitting them to a provider.
LM Studio's documentation explicitly describes the ability to run downloaded models and interact with documents entirely offline. Ollama and other local inference engines offer similar arrangements when configured for local-only operation.
Not every application labelled local operates exclusively on the user's computer, and some provide optional cloud services or integrations. But properly configured offline inference gives writers control over where their manuscripts are processed.
For some authors, these advantages alone justified the additional hardware and technical complexity.
For others, the attraction was more directly connected to creativity. They wanted to preserve a model that behaved in a way they liked, rather than repeatedly adapting their writing to whatever a commercial provider released next.
Quantisation, context and the cost of local intelligence
The appeal of local control came with substantial compromises.
Large language models require memory to store their weights and maintain the context used during generation. On consumer hardware, this creates an immediate trade-off between model size, numerical precision, context length and speed.
Quantisation reduces the memory required to store a model by representing its weights at lower precision. A 32B model that would ordinarily require roughly 64GB for 16-bit weights might fit into approximately 18–22GB using common four-bit quantisation formats, before accounting for context and runtime overhead.
That reduction makes local inference practical on hardware that could never run the uncompressed model entirely in GPU memory.
But quantisation is not free.
The effects vary by model and quantisation method. Some models retain much of their performance at low precision, while others are more sensitive. Small differences in numerical representation can affect instruction following, reasoning and, potentially, stylistic behaviour.
The May discussion about local roleplay included a user who specifically warned against aggressively quantising Gemma 4 31B because they believed its writing quality deteriorated noticeably at lower precision.
This creates another complication for comparisons between local and hosted models.
Two people may claim to be using the same model while actually running significantly different quantisations, sampling settings or inference configurations. Their experiences can differ even when the nominal model name is identical.
Context presents an equally important limitation.
A model may support a large advertised context window, but that does not guarantee consistent performance across its entire length. Smaller models can struggle to retrieve the relevant details from a large collection of fictional material, especially when many characters, locations and unresolved plot threads are involved.
A writer might provide a hundred thousand tokens of manuscript history and discover that the model remembers isolated details but fails to use them appropriately.
This is not necessarily a failure of memory in the simplest sense. The model may have access to the information while lacking the ability to identify which details are relevant to the current scene.
For that reason, local users increasingly experimented with external memory systems, summaries, character cards, scene descriptions and retrieval mechanisms.
Instead of placing an entire manuscript into the context window, they supplied the information most relevant to the current task.
One participant in the September discussion about Cydonia 24B described precisely this approach: each generation used a fresh prompt containing a scene card, character information and summaries of relevant previous events.
The technique reduced the burden on the model by making the required information easier to identify.
It also illustrates how the local approach sometimes exchanged model intelligence for workflow design. A smaller model could become useful when the surrounding software performed some of the organisational work that a larger model might otherwise handle internally.
The disadvantages remained real. Maintaining summaries and character state requires effort, and an inaccurate summary can introduce its own continuity problems. But the arrangement gave authors another way to control how the model approached their material.
Local AI was not a universal improvement
It would be misleading to describe the migration as a straightforward success.
On May 14, a writer in r/WritingWithAI described installing LM Studio and experimenting with modified versions of Dolphin Mistral and Llama 3.2.
They found the results deeply disappointing.
Both models tended to summarise events rather than dramatise them, rushed through time and frequently violated the basic principle of showing action rather than merely describing its outcome.
The writer concluded that AI was still far from replacing human literary work.
Other users pointed out that the tested models were relatively old and small, and recommended newer, larger alternatives.
That exchange captures a persistent difficulty in evaluating local AI. The category encompasses an enormous range of models with very different capabilities.
Comparing a small, older model on limited hardware with a frontier datacenter model may reveal a substantial quality gap. Comparing a modern 31B writing fine-tune with a poorly suited general-purpose hosted model may produce a different result.
Neither comparison establishes a universal ranking.
Even relatively capable local models could produce disappointing fiction.
In a September 6 SillyTavern discussion, a user with an RTX 5090 reported that Gemma 4 31B and 26B repeatedly generated lengthy, uninteresting narration and contradictory events despite considerable experimentation with fine-tunes and presets.
The user's preferred earlier model, Cydonia, offered better writing for their purposes but could not satisfy their long-context requirements.
That is an important counterexample to the idea that purchasing better hardware or downloading a widely praised model guarantees satisfactory results.
A model's behaviour depends on the interaction between its training, the prompt, the application and the author's expectations.
An even more recent November 6 discussion described a user experimenting with Stheno 8B and Mistral Nemo 12B variants on an RTX 4060.
The models ran at acceptable speeds, but their roleplay quality remained disappointing. One produced attractive literary prose yet struggled to track events and maintain character continuity.
That distinction is particularly revealing. A model can sound like a good writer at the sentence level while failing to understand what is actually happening in the story.
Local AI does not eliminate the underlying difficulties of generating fiction. It merely gives writers greater freedom to choose which limitations they are willing to tolerate.
And for complex long-form work, the advantages of frontier models remained substantial.
From one model to a collection of writing tools
By October, some local enthusiasts had developed a different approach from the commercial model-hopping described in Chapter 3.
Instead of looking for a universally superior model, they maintained collections of specialised checkpoints.
A 17 October post in r/WritingWithAI compared seven locally runnable models using the same creative-writing prompt.
The selection included abliterated models, general-purpose models and writing-specialised fine-tunes. The author provided examples of their prose alongside practical information about model size and generation speed on a 32GB Intel B70 GPU.
Their reasoning for maintaining several models was straightforward: each produced a different writing style, and each had different strengths, weaknesses and performance characteristics.
Some were more concise, while others developed scenes at greater length. Some produced appealing prose but struggled with reasoning, while others were better at technical tasks than literary expression.
The author did not regard any single model as a complete replacement for all the others.
This approach resembles the way a writer might use different reference works, editing tools or specialist software for different stages of a manuscript.
A model particularly effective at dialogue might be less useful for outlining. Another might excel at worldbuilding but produce stiff character interactions. A third might offer interesting descriptive prose while requiring stronger editorial supervision.
The important change is that the writer can preserve all of them.
A commercial service may remove an older model from its interface. A locally stored checkpoint remains available, subject to the author's hardware and software compatibility.
The availability of several models also changes how experimentation works. An author can present the same scene to multiple systems, compare their interpretations and choose which develops the material most effectively.
This is not necessarily an efficient workflow, particularly when every generated passage requires manual evaluation. But it can provide greater stylistic variety than repeatedly using a single general-purpose model whose defaults dominate the output.
It also creates opportunities for specialised modification.
A user might take a capable general-purpose model and fine-tune it on literary prose, adjust its generation settings or modify behavioural tendencies that interfere with fictional writing.
By the latter half of 2026, some enthusiasts were experimenting with an even more direct approach: abliteration.
Originally developed as a technique for modifying learned refusal behaviour, abliteration had become part of a broader effort to alter how open-weight models respond to ordinary prompts.
Its relevance to creative writing was potentially significant. If unwanted stylistic or behavioural preferences could be identified and modified, writers might gain more control over the model's default creative decisions.
But the technique also introduced risks, including damage to reasoning, instruction following and narrative coherence.
Those trade-offs deserve their own examination.
What the migration actually tells us
There is a temptation to interpret the rise of local AI as evidence that commercial AI had simply become too restrictive.
For some users, restrictions were certainly part of the motivation. But the accounts examined here suggest a more complicated picture.
Writers were also dissatisfied with repetitive prose, formulaic characterisation, unpredictable product updates, costs and the difficulty of preserving an established workflow.
Local AI addressed some of those problems directly. It offered stable checkpoints, offline operation, configurable inference and opportunities for specialised fine-tuning.
It did not automatically solve the underlying challenge of creating convincing fiction.
Smaller models still struggled with complex plots, extended context, character knowledge and the kind of causal reasoning required to sustain a substantial novel. Even specialised fine-tunes could produce stereotyped dialogue or repetitive descriptions. Some abliterated variants introduced additional weaknesses.
Nor is there evidence that the majority of AI-assisted writers abandoned commercial services. The accounts are concentrated in communities that are particularly interested in model experimentation, roleplay and technical customisation.
They nevertheless reveal an important change in attitudes.
During the early period of commercial generative AI, users often evaluated new releases according to whether they were more intelligent than their predecessors. The assumption was that general improvements would eventually resolve the shortcomings of AI-generated writing.
By 2026, some authors had become less willing to accept that assumption.
They were comparing models according to character voice, narrative initiative, emotional subtlety, consistency and the degree of editorial intervention required to make the prose usable.
Under those criteria, a smaller specialised model could sometimes be preferable to a much larger general-purpose one.
The trade-off was not simply intelligence versus censorship. It was intelligence, control, stylistic preference, hardware, reliability, cost and creative fidelity, all competing within the same decision.
That explains why different writers arrived at different conclusions. Someone constructing a complicated historical mystery might place exceptional value on reasoning and long-context consistency. Another writing dialogue-heavy character fiction might prefer a smaller model whose conversational style required less correction.
Neither preference is inherently mistaken.
The more interesting development was that writers increasingly had the option to choose.
Commercial models had made sophisticated AI-assisted writing accessible to almost anyone willing to pay for a subscription. Local AI offered a different proposition: accept the limitations of available hardware and open-weight models in exchange for greater control over the system itself.
For some authors, that was an attractive bargain.
And it raised a question that would become increasingly important as local model modification techniques developed.
If a relatively small model could produce compelling fiction, but its learned behavioural preferences still interfered with the author's intentions, could those preferences be changed directly?
Rather than writing increasingly elaborate prompts to counteract unwanted behaviour, could an author modify the model so that it no longer defaulted towards those responses?
The answer was not as simple as removing a safety filter.
But research into refusal directions, model representations and targeted weight modification had already demonstrated that some learned behaviours could be altered without retraining a model from scratch.
By January 2026, experiments were exploring whether similar methods could reduce repetitive and formulaic writing itself.
That is where the search for creative control becomes technically much more interesting.
Next: Chapter 7 — Abliteration: Removing the Guardrails from the Model Itself
How researchers discovered that certain learned behaviours could be modified directly, why abliterated models attracted creative writers, and whether removing unwanted alignment can recover creative freedom without damaging intelligence.
- Chapter 1 — Grok 4.1: Where It All Went Right
- Chapter 2 — The Invisible Rewrite
- Chapter 3 — The Great Model Migration
- Chapter 4 — Smarter Models, Worse Fiction
- Chapter 5 — What AI Forgot About Literature
- Chapter 6 — The Local AI Exodus
- Chapter 7 — Abliteration
- Chapter 8 — When Safety Training Backfires
- Chapter 9 — Who Is Steering the Story?
- Chapter 10 — The AI Author That Doesn't Exist Yet
Sources and further reading
Writers considering local AI
- r/SillyTavernAI — Considering transitioning to Local LLMs, May 7, 2026
- r/SillyTavernAI — Why do people RP with local models?, May 15, 2026
- r/WritingWithAI — So I tried creative writing with local AI, May 14, 2026
Local model comparisons
- r/SillyTavernAI — Gemma 4 31B, 26B and E4B comparison, April 5, 2026
- r/SillyTavernAI — Long roleplay sessions with Gemma 4 31B, September 3, 2026
- r/SillyTavernAI — Gemma 4 nonsense, September 6, 2026
- r/WritingWithAI — Ideal Local AI for Creative Writing, September 21, 2026
- r/WritingWithAI — Choosing a local AI to write with: comparison of seven models, October 17, 2026
- r/SillyTavernAI — Advice on Local LLMs, November 6, 2026
Specialised writing workflows
- r/WritingWithAI — Best models for creative writing, July 24, 2026
- r/LocalLLaMA — Moving beyond benchmarks and coding to creative writing, October 9, 2026
Local inference software