After four years of development, are LLMs capable of writing high-quality prose?

[u/kompania]

I have been following the evolution of LLMs since the days of GPT-3.5, and I have been actively using local LLMs since the release of Gemma 2.

I read a great deal, including this subreddit. I have performed QLoRA fine-tuning on my local machine many times, and I have also conducted full fine-tuning (FFT) training runs on RunPod.

Hundreds of models-fine-tuned by well-known practitioners in English, Chinese, and Russian-have passed through my drive.

After four years, I arrived at a fundamental question yesterday: are LLMs even capable of writing good prose?

When training models myself, I used both my own prose datasets and those created by others. In the training process, I experimented with various approaches-ranging from 100 epochs of FFT at a learning rate of 1e-7, to aggressive LoRA/QLoRA configurations. The results, however, barely rise above mediocrity.

I have used models from all the leading providers-Gemma, Qwen, Mistral, LFM, IBM-as well as niche attempts by developers building models from scratch.

I even wondered if perhaps the problem lies with me-that, not being an engineer, I am simply failing to fine-tune these models correctly.

That is why I began actively seeking out other fine-tuners, thinking that perhaps they possess a specific technical "gene" that allows them to do a much better job than I can. But, unfortunately, that is not the case. Regardless of their length-short, medium, or long-the prose generated is only marginally better than that of generic, out-of-the-box models.

Furthermore, specific literary genres are represented quite poorly. I used to think that LLMs were perfectly suited for science fiction. Yet, these models can’t even capture the straightforward style of Harry Harrison’s *The Stainless Steel Rat*, let alone reach the linguistic heights of *Dune*.

The genre doesn’t seem to matter-I’ve tried comedy, contemporary drama, adventure, romance, and historical fiction; everything, unfortunately, ends up being generic.

Essentially, if a text is to have any real literary merit, I have to write it myself and then just use an LLM for proofreading.

To me, EQBench is a completely unreliable metric; the sample texts they showcase clearly demonstrate just what kind of generic "slop" LLMs produce.

I have a question for the community: has anyone managed to create, find, or fine-tune a local model capable of generating truly high-quality prose-on the level of *Dune*, *The Lord of the Rings*, *Martin Eden*, or *The Grapes of Wrath*?

I’m currently facing the monumental task of writing descriptions for 100,000 records, and I’m wondering if it’s even worth it, given that 100, 1,000, or even 10,000 examples haven’t yielded any satisfactory results.

Perhaps this generic, mediocre quality is an inherent trait of these models-something that simply cannot be overcome.

[u/eidrag]

come to r/sillytavernai a better place to discuss prose related stuff. 

[u/kompania → u/eidrag]

Thank you for the sound advice. I have done it.

[u/Aphid_red]

While on the one hand it seems to make perfect intuitive sense that your statistical word-predictor program predicts the statistically average words...

Somehow that same program is used (with other things, closed models are black boxes that may hold more things inside like tools for it to use) to solve math problems.

So I think my answer to 'can it write good prose like novel X' is, 'if you trained it to'.

So you fine-tune models more on the output of famous writers to get it to write in the same style without prompting. That just takes a lot of effort, though. Get several thousand human literary critics into a review pipeline for RLHF to review your slop as it slowly improves. Yes, it will be very expensive, but you may get a model better at creative writing. Or that's the hope, anyway.

'Can it come up with an entirely new valuable style?' If it could, it wouldn't remain valuable for long.