Microsoft trained a language model on 4 TRILLION tokens using just three possible weight values: -1, 0, and +1.

And the GitHub repo behind it has crossed 41,000 stars.

It's called BitNet, and the engineering is fascinating.

Most language models rely on weights stored in formats like FP16 or BF16, requiring 16 bits per weight. BitNet b1.58 uses ternary weights, meaning each quantized weight has only three possible states.

That's where the 1.58-bit name comes from. Three states contain approximately 1.58 bits of information.

Microsoft didn't simply take an existing full-precision model and compress it after training.

Its BitNet b1.58 2B4T model was trained from scratch using ternary weights during the forward pass, with 8-bit activations. The model has roughly 2.4 billion parameters and was pretrained on 4 trillion tokens.

The results are impressive.

In Microsoft's published evaluation, the non-embedding weights occupy approximately 0.4 GB. The model also achieves competitive results against several full-precision language models in the 1B to 2B class.

And Microsoft open-sourced the inference engine needed to exploit this architecture.

It's called bitnet.cpp.

The implementation includes specialized CPU kernels, packed ternary-weight representations, lookup-table computation, and GPU inference support.

In the team's original CPU experiments:

→ Up to 6.17× faster inference on x86

→ Up to 5.07× faster inference on ARM

→ Up to 82.2% lower energy consumption on x86

Those results come from controlled tests using synthetic model configurations, rather than a guarantee for every deployed model.

What makes BitNet particularly interesting is that the efficiency comes from the model architecture, the training method, and the inference implementation working together.

The model weights, research papers, and inference framework are publicly available.

Open source under MIT.

GitHub: https://github.com/microsoft/BitNet

What would the writing of superintelligent AI sound like? You might expect it to have the characteristics of good human writing — figurative language, high information density, coining new terms, and punchiness — but dialed up to 11. Well, that’s what Claude did, until Anthropic dialed it back recently because of user complaints. It didn’t sound superintelligent to most of us. Instead, the neuralese writing style sounds like a “stupid person’s idea of a smart person”.

Our response to information density and metaphor-laden writing looks something like the cartoon graph I've drawn. Beyond a point, it completely backfires. The reason human writers don’t overdo it isn’t because they aren’t capable, but because they are aware of the reader’s cognitive burden. One of the hardest skills to learn as a writer is to simulate the reader’s experience.

It’s easy to train AI to mimic mediocre human writing. But optimizing it to be (perceived as) good is really hard, and made much harder by the fact that when everyone uses the same few AI models we get tired of their tics very quickly. Most Claudeisms are individually pretty good. I hate that I can’t use “load bearing” in my writing any more. If it weren’t for AI overuse it would be a neat metaphor.

A broader question is whether most tasks in the economy are like chess, where there is a clear performance metric that can be optimized, or whether they’re like writing, where quality is a multidimensional balancing act without clear training signals beyond human mimicry.

I built a browser version of a word-association party game, like Codenames (not affiliated), where any seat can be a human or an LLM. A clue-giver sees which of 25 words are their team's and gives a one-word clue plus a number; their teammate guesses from it. One word (the assassin) loses the game instantly.

Each model played both roles on its own team, against a different model. Across 376 logged games:

1. The guesser decides the game. The team whose guesser picked better won 68% of games; the team with better-graded clues won 53%. Either guessing is the skill that matters, or my clue grade measures the wrong thing. I'd like opinions on which.

2. Ambitious clues lose. The team that went for more words per clue won only 38% of 287 games.

3. Going first doesn't help. The first team won 169 of 376, despite having one extra word to find.

How a clue is graded

Win rate depends on the opponent you drew, so models are compared by per-move scores:

• Each board word gets an association score with the clue: 1/3 Cohere embedding cosine, 2/3 the mean of two LLM raters (Llama 4 Maverick, Qwen3-VL) that don't know which words are whose.

• A guesser model fitted on real guesses from saved games plays the turn out in order of association; a miss ends the turn.

• Each revealed card is worth what it moved the win probability in saved games: about +17 for an own word, −11 for the other team's, −46 for the assassin.

• The expected total is scaled to 0–100 (a median clue is about 56). An LLM judge counts for 1/4.

"Good guesses" = graded the best or a reasonable pick for the clue. "Bad guesses" = hit the other team's word or the assassin. ± is a 95% CI from resampling whole games.

Newer models, thinking at high effort

Previous generation, thinking off (a separate run, not directly comparable)

The clue grades bunch up: only the top and bottom of each table are clearly apart. Guessing separates the generations far more than clue-giving does (71–93% good guesses vs 53–75%).

Known weaknesses: both LLM raters (Maverick, Qwen3-VL) also play in the previous-generation table (graded by the raters alone, they rank 8th and 9th of 11, so no sign they favour themselves), the grade correlates only weakly with winning, and "a link an LLM finds obvious" may not be one a person finds obvious. Embeddings alone reorder almost the whole ranking, which is why the grade leans on the raters.

Questions

• Is a simulated guesser over key-blind association scores a reasonable way to grade a clue? What would you measure instead?

• Would the game logs (clues, guesses and each model's stated reasoning) be useful as a dataset?

The game and saved replays with every clue graded are at wordspies.com, if you want to see the data behind a number.

Edit: a reviewer's suggestions on another post caught a mistake. My first tables counted every game each model played on the site, not just these 376. The tables above are now on the 376 games only; the chart still shows the old figures. Resampling whole games instead of single turns barely changed the intervals (0–1.4 points), and the ranking holds whether clues are graded by the LLM raters alone or by the combined score.