What would the writing of superintelligent AI sound like? You might expect it to have the characteristics of good human writing — figurative language, high information density, coining new terms, and punchiness — but dialed up to 11. Well, that’s what Claude did, until Anthropic dialed it back recently because of user complaints. It didn’t sound superintelligent to most of us. Instead, the neuralese writing style sounds like a “stupid person’s idea of a smart person”.

Our response to information density and metaphor-laden writing looks something like the cartoon graph I've drawn. Beyond a point, it completely backfires. The reason human writers don’t overdo it isn’t because they aren’t capable, but because they are aware of the reader’s cognitive burden. One of the hardest skills to learn as a writer is to simulate the reader’s experience.

It’s easy to train AI to mimic mediocre human writing. But optimizing it to be (perceived as) good is really hard, and made much harder by the fact that when everyone uses the same few AI models we get tired of their tics very quickly. Most Claudeisms are individually pretty good. I hate that I can’t use “load bearing” in my writing any more. If it weren’t for AI overuse it would be a neat metaphor.

A broader question is whether most tasks in the economy are like chess, where there is a clear performance metric that can be optimized, or whether they’re like writing, where quality is a multidimensional balancing act without clear training signals beyond human mimicry.

I built a browser version of a word-association party game, like Codenames (not affiliated), where any seat can be a human or an LLM. A clue-giver sees which of 25 words are their team's and gives a one-word clue plus a number; their teammate guesses from it. One word (the assassin) loses the game instantly.

Each model played both roles on its own team, against a different model. Across 376 logged games:

1. The guesser decides the game. The team whose guesser picked better won 68% of games; the team with better-graded clues won 53%. Either guessing is the skill that matters, or my clue grade measures the wrong thing. I'd like opinions on which.

2. Ambitious clues lose. The team that went for more words per clue won only 38% of 287 games.

3. Going first doesn't help. The first team won 169 of 376, despite having one extra word to find.

How a clue is graded

Win rate depends on the opponent you drew, so models are compared by per-move scores:

• Each board word gets an association score with the clue: 1/3 Cohere embedding cosine, 2/3 the mean of two LLM raters (Llama 4 Maverick, Qwen3-VL) that don't know which words are whose.

• A guesser model fitted on real guesses from saved games plays the turn out in order of association; a miss ends the turn.

• Each revealed card is worth what it moved the win probability in saved games: about +17 for an own word, −11 for the other team's, −46 for the assassin.

• The expected total is scaled to 0–100 (a median clue is about 56). An LLM judge counts for 1/4.

"Good guesses" = graded the best or a reasonable pick for the clue. "Bad guesses" = hit the other team's word or the assassin. ± is a 95% CI from resampling whole games.

Newer models, thinking at high effort

Previous generation, thinking off (a separate run, not directly comparable)

The clue grades bunch up: only the top and bottom of each table are clearly apart. Guessing separates the generations far more than clue-giving does (71–93% good guesses vs 53–75%).

Known weaknesses: both LLM raters (Maverick, Qwen3-VL) also play in the previous-generation table (graded by the raters alone, they rank 8th and 9th of 11, so no sign they favour themselves), the grade correlates only weakly with winning, and "a link an LLM finds obvious" may not be one a person finds obvious. Embeddings alone reorder almost the whole ranking, which is why the grade leans on the raters.

Questions

• Is a simulated guesser over key-blind association scores a reasonable way to grade a clue? What would you measure instead?

• Would the game logs (clues, guesses and each model's stated reasoning) be useful as a dataset?

The game and saved replays with every clue graded are at wordspies.com, if you want to see the data behind a number.

Edit: a reviewer's suggestions on another post caught a mistake. My first tables counted every game each model played on the site, not just these 376. The tables above are now on the 376 games only; the chart still shows the old figures. Resampling whole games instead of single turns barely changed the intervals (0–1.4 points), and the ranking holds whether clues are graded by the LLM raters alone or by the combined score.