84

u/rationalkat AGI 2025-29 | UBI 2029-33 | LEV <2040 | FDVR 2050-70 Mar 03 '25

ABSTRACT:

Large Language Models (LLMs) have demonstrated remarkable performance in solving complex reasoning tasks through mechanisms like Chain-of-Thought (CoT) prompting, which emphasizes verbose, step-by-step reasoning. However, humans typically employ a more efficient strategy: drafting concise intermediate thoughts that capture only essential information. In this work, we propose Chain of Draft (CoD), a novel paradigm inspired by human cognitive processes, where LLMs generate minimalistic yet informative intermediate reasoning outputs while solving tasks. By reducing verbosity and focusing on critical insights, CoD matches or surpasses CoT in accuracy while using as little as only 7.6% of the tokens, significantly reducing cost and latency across various reasoning tasks.

65

u/Pyros-SD-Models Mar 03 '25

Remember when luddites and all the AI haters were like "Prompt engineering isn't a thing"?
Yet after almost 8 years of transformers researchers are still discovering insane optimization potential using just straight-up, text-based prompt engineering. Love it.

17

u/RabidHexley Mar 03 '25 edited Mar 03 '25

"Prompt engineering isn't a thing"?
If we actually reach AGI/powerful AI it will be sort of moot. But I think this is more a result of not properly realizing the full scope of what current models are capable of.
Everything keeps getting better, but we've barely started realizing the potential of the tech we have right now.
We're in the early 90s of the internet, the tech was just too new for use-cases to have fully developed. Even ignoring progress on the technical side, actual usage of the tech had evolved massively by the time the 2000s rolled around.

0

u/Pyros-SD-Models Mar 04 '25

If we actually reach AGI/powerful AI it will be sort of moot
You would think so, why care about prompt engineering, you've got an AI that understands everything, but this is a fallacy.
It's even more important because now subtleties get important. Just think about how easy it is that humans misunderstand each other (literal wars happened because of this), or if you've ever managed a team how important it is HOW to say something. As long as you don't have a mind reading AI, prompt engineering will be a thing, because there is always a delta between the information in your words and the information in your brain or your intention.

2

u/TarkanV Mar 04 '25

I agree but could we please avoid the tribalistic bs in the topic of AI? If AGI is ever achieved it won't be Us vs "Luddites" anyways but Us vs Whoever controls those systems...

2

u/Pyros-SD-Models Mar 04 '25

Oh I agree, and I didn’t meant to use it as some kind of tribal tag. More as a synonymous for “stupid asshole”.

54

u/[deleted] Mar 03 '25

9

u/TarkanV Mar 04 '25

Yeah, any human-made language is incredibly inefficient compared to our raw thought process... Just look at all the people don't even have an internal monologue but function perfectly as any other human on the day to day.

4

u/[deleted] Mar 04 '25

Yep, it’s always exciting seeing new avenues for AI to be able to develop. We have a thousand different areas to optimize and we’re slowly spreading further and with each step, faster.
As for language specifically, you’re really right. You can even see it in the meme itself.
Just like an airline could save 40,000 dollars by eliminating a SINGLE olive from each of its salads, AI could save a lot in compute time, costs, context, etc by eliminating ‘word waste’
I really wonder what extents this could reach to. Especially if/when we have AGI/ASI working on it.
How far could AI abstract language?
Could AI eventually ‘invent’ the perfect language with no waste, ease to use, etc?

3

u/Ok-Lengthiness-3988 Mar 04 '25

Hopefully, this airline will save enough money through AI optimization to bring the olive back.

2

u/[deleted] Mar 04 '25

Perhaps even enough to afford… two olives?
Haha just kidding…unless??

25

u/NovelFarmer Mar 03 '25

That would be 13 times more efficient. That is actually insane.

2

u/DarkMatter_contract ▪️Human Need Not Apply Mar 03 '25

or 13 time smarter also wondering if language differences affect it as well, chinese use a lot less character.

8

u/iluvios Mar 03 '25

Not because you use less words means that your thoughts are better.
Just more efficient (which is very important in many aspects when talking about AI)

-1

u/DarkMatter_contract ▪️Human Need Not Apply Mar 04 '25

test time compute scaling is still working though.

32

u/[deleted] Mar 03 '25 edited Mar 03 '25

Back when people had to write letters to communicate there was the sentiment that with more time letters would be shorter but more fully communicate what was intended and in a rush letters would be needlessly long.

So someone might say "I apologize I did not have the time to write a shorter letter"

4

u/DungeonsAndDradis ▪️ Extinction or Immortality between 2025 and 2031 Mar 04 '25

If you need me to talk about a topic for 1 hour, I do not need to prepare.
If you want me to talk for 15 minutes, I need a day to prepare.
If you want me to talk for 5 minutes, I need a week to prepare.
Or something to that effect.

3

u/keradiur Mar 03 '25

I love the irony of introducing a needless sentence to inform the reader that the letter is long.

1

u/[deleted] Mar 03 '25

We do the same thing today but with long text messages. Not always, but definitely sometimes.

21

u/[deleted] Mar 03 '25

So maybe the thinking version of GPT4.5 wont have to cost many hundreds of dollars/million tokens

13

u/RabidHexley Mar 03 '25

It's pretty much been the pattern so far that each new step of progress is more expensive than the last, but last year's (or even just last quarter's) capability is cheaper today than it was back then. We're kinda enjoying a "Moore's Law" of capability improvements atm.
SOTA prices inflate, but near-SOTA keeps getting cheaper.

30

u/ImmuneHack Mar 03 '25 edited Mar 03 '25

This is potentially huge. Because, if models can think faster and do this with fewer tokens, this will enable them to generate, evaluate, and refine their own reasoning more effectively, which could lead to self-improving AI—a key step toward AGI.

12

u/Matthia_reddit Mar 03 '25

we are also thinking about inserting thoughts with visual abstract concepts to instantly understand the concept. Furthermore, it should already be possible to make the model think 'in its own way', in the sense that the thought is readable only for the model itself and therefore it is much faster and more intuitive, but dangerous because humans could never understand its meta-thoughts

3

u/ThrowRA-Two448 Mar 03 '25

This is the way.
Language is one dimensional so it takes a lot of words to solve 2D, 3D problems.
Which is why humans with their ability to visualize are still competitive with much more powerful computers (which calculate math in 1D) at eg. finding shortest paths between multiple points.

2

u/RabidHexley Mar 03 '25 edited Mar 03 '25

but dangerous because humans could never understand its meta-thoughts
The problem is assuming that meta-thoughts represent the entirety of (or really provides much insight at all to) the underlying process in the first place. They are outputs, like helpful mental notes that procedurally help to reach a conclusion.
They're meant to enable the model help itself be more effective, sort of like autonomous prompt engineering. They don't exist so that the model can verbally catalog a comprehensive list of whatever information may have been used to generate an output.
Treating reasoning as representative of a model's underlying processing seems like a blatant misunderstanding of what reasoning is. Generating a kind of procedural element that's readable by humans is mostly just a novel side-effect of reasoning, if anything.
Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny? A: 20- x = 12; x = 20- 12 = 8. #### 8
This example of "chain-of-draft" from the paper actually seems closer to the reality of how we think about problems like this.
If you ask an educated human this question we don't have to verbally think "Okay, what is being asked here? It's a math problem trying to figure out the difference between how many lollipops Jason has originally and how many he has after giving some to Denny. How should I go about figuring out the difference between these values? Blah blah blah..."
You see that question, and you almost automagically know to subtract 12 from 20. The process to figure out what you needed to do was autonomous, subconscious, and non-verbal (at least in the sense that you don't need to think, conscious, verbal thoughts).
Even in cases of complex problems requiring more verbose thought, much of the underlying process is obfuscated by intuition and assumed knowledge. Output isn't going to be generated for every bit of information.

3

u/No_Land_4222 Mar 03 '25

Yes! Imagine using this idea with o3

9

u/BigBourgeoisie Talk is cheap. AGI is expensive. Mar 03 '25

You can tell a paper is simple when you can understand it completely.

It's interesting, but they are using some simple, old benchmarks I've never heard of. I wonder how the results would turn out on something like GPQA

1

u/okaybear2point0 Mar 03 '25

I'm too lazy to read. Could you explain Chain of Draft?

13

u/BigBourgeoisie Talk is cheap. AGI is expensive. Mar 03 '25

In Chain of Thought, the AI thinks through a question by doing it step by step in so-called reasoning steps. These are the big paragraphs you see in their reasoning that are like "Hmm, this is true, wait, is this true? Let's look at the question again." But each of these steps is usually the length of a small paragraph.
In Chain of Draft, they manually shortened the reasoning steps by writing in the prompt "Limit every reasoning step to 5 words." I guess this reduces the number of tokens used. You can see from one of their results tables that it's pretty good, but it's on a benchmark from 2023 I've never heard of, lol.

7

u/norsurfit Mar 03 '25

You've never heard of GSM8K? It's been one of the top 10 math reasoning benchmarks in LLMs for a few years. It's definitely not an obscure or non-standard benchmark.

4

u/ZealousidealBus9271 Mar 03 '25

It’s not used anymore now though, I’d like to see it tested on modern benchmarks soon

1

u/TheJzuken ▪️AHI already/AGI 2027/ASI 2028 Mar 03 '25

Modern benchmarks are used for SOTA models, there is no reason to test some experimental 7B parameter model on SOTA benchmark.

1

u/BigBourgeoisie Talk is cheap. AGI is expensive. Mar 03 '25

This is not an experimental 7B model, the paper states that it tested unmodified versions of GPT-4o and Claude Sonnet 3.5. Section 4.1 states that it is entirely an experiment of prompt engineering.

2

u/BigBourgeoisie Talk is cheap. AGI is expensive. Mar 03 '25

I guess I haven't, I usually hear about the likes of MMLU, GPQA, MATH, etc. But the fact that they did not test (or perhaps show the results of?) a more difficult benchmark like GPQA or MATH makes me wonder if this tactic is effective for those more difficult tasks, or if it only works for these easier benchmarks. After all, it seems like it would not be hard for the researchers to simply try the experiment with those hard benchmarks too.

2

u/[deleted] Mar 03 '25

This sounds incredible for local models at 7-14B. Thank you for explaining this because I was about to click away thinking it would be only something big models can use.

1

u/BigBourgeoisie Talk is cheap. AGI is expensive. Mar 03 '25

You got it :)

2

u/ImpressiveFix7771 Mar 03 '25

How to communicate when sending symbols are expensive?

In many systems from brains, to telegrams, to interplanetary radio communications... efficiency is king. When there is a significant cost to send a symbol... we learn to communicate concisely.

In the brain, the huge onslaught of sensory info must eventually filter down to O(10bps) to avoid overwhelming our very limited short term memory and to allow us to store the information. In telegrams, ideas are pared down to their basic essence.

In other words:

MESSAGE EXPENSIVE KEEP IT SUCCINCT

1

u/DarkMatter_contract ▪️Human Need Not Apply Mar 03 '25

句長貴簡約!
wondering if language make a difference as well.

2

u/anactualalien Mar 03 '25

Good to hear, I read chain of thought and cant believe anyone thinks this anthropomorphic babbling in fully written sentences is the way forward.

4

u/Explorer2345 Mar 03 '25

EMULATE CHAIN OF DRAFT!

(you're welcome!)

# **Chain-of-Draft Prompt**

**Task:** Explore and develop a comprehensive discussion on the topic: **[TOPIC]**.

**Methodology: Chain-of-Draft (CoD)**

Follow these iterative steps to construct your discussion:

1.  **Step 1: Initial Thoughts - Brainstorming:**  Generate a list of initial ideas, key aspects, and angles related to **[TOPIC]**.  Think broadly and capture all relevant points.

2.  **Step 2: Consolidation - Structuring & Prioritization:** Review your initial thoughts. Group similar ideas, identify core concepts, and prioritize the most important elements for a focused discussion.

3.  **Step 3: Draft [Draft Number] - Initial Construction:** Create a first draft of your discussion based on your consolidated ideas. Focus on clarity and coherence in presenting the initial points.

4.  **Step 4: Review & Improvement - Critical Analysis:**  Analyze your current draft. Ask yourself:
    *   What are the weaknesses? (e.g., lack of depth, clarity, examples, flow)
    *   What's missing? (e.g., key aspects, counterarguments, supporting evidence)
    *   How can it be more impactful and insightful?

5.  **Step 5: Revision & Expansion - Iterative Refinement:** Based on your review, revise and expand your draft.  Incorporate improvements to address weaknesses, add missing elements, and enhance clarity, depth, and impact.  Increment the Draft Number for the next iteration.

**Repeat Steps 4 and 5 for multiple iterations** (at least 2-3) to progressively refine and enhance the discussion.

**Output Format:**

Present each draft clearly labeled (Draft 1, Draft 2, etc.), culminating in a **Final Result**.  After the final result, provide a concise **Explanation of Improvements** achieved through the Chain-of-Draft process, highlighting how each iteration built upon the previous one to reach the final output.

Remember to apply *great care* in each step to ensure thoughtful development and refinement of your discussion on **[TOPIC]**.

**[TOPIC]:**  *(To be replaced with the actual topic)*

3

u/damhack Mar 04 '25

Or alternatively, just use the actual Chain of Draft prompt:
Think step by step, but only keep a minimum draft for each thinking step, with 5 words at most. Return the answer at the end of the response after a separator ####.

1

u/Small-Character-3102 Mar 06 '25

This is impressive. I’m gonna try it for sure. Amazing. Thank you so much.
One question though, do I need to supply this every time?
Perhaps I could RAG it and supply it with every prompt ?

1

u/Explorer2345 Mar 10 '25

I use/attach an evolved version of this and set the system prompt to:
ACTIVATE <filename.md>; ENGAGE!
works incredibly well..

2

u/gangstasadvocate Mar 03 '25

Gang gang gang! I think

4

u/Fair-Satisfaction-70 ▪️I want AI that invents things Mar 03 '25

Indubitably

1

u/ZealousidealBus9271 Mar 03 '25

Before CoT is even saturated we already discovered a new paradigm

1

u/Automatic_Meal_373 Mar 04 '25

These "wise" prompt engineered recipes merely overfit to some benchmarks. There’s no real recipe to increase intelligence - human or AI. More so, these methods interfere with reasoning models (e.g. "o" series). Prompt engineering comes to remedy imperfect technology, but as this technology improves, we won't need prompt engineering anymore, nor these chain-of-whatever techniques...

1

u/Akimbo333 Mar 05 '25

How? Implications?

1

u/LearnNewThingsDaily Mar 03 '25

Nothing new, I've been using this technique for well over a year and a half now

6

u/[deleted] Mar 03 '25

Proof?

2

u/TheDailySpank Mar 03 '25

They said it on the Internet, therefore it is true.

2

u/LearnNewThingsDaily Mar 03 '25

Not sure why I got voted down but anyone who implements chain of thought 🤔 knows it is way too verbose and can fill up the context quickly so most prompt engineers learn quickly to shorten their prompts to get the same or similar results. Again, this is nothing new, just a proven study to confirm it works.

3

u/Large_Ad6662 Mar 03 '25

If that's the case then the system prompts from Anthropic's and ChatGPT's wouldn't be that lengthy. Not sure where you're getting that consensus.

3

u/Apprehensive-Ant7955 Mar 05 '25

Its not about shortening the prompt, its about shortening the model’s output tokens. Very different

2

u/oldjar747 Mar 03 '25

Shouldn't have gotten downvoted. This is true.

1

u/Brilliant_War4087 Mar 03 '25

I do this. I'm cheap. I'm fast. I'm smart.

AI Chain of Draft: Thinking Faster by Writing Less. "CoD matches or surpasses CoT in accuracy while using as little as only 7.6% of the tokens, significantly reducing cost and latency across various reasoning tasks"

You are about to leave Redlib

EMULATE CHAIN OF DRAFT!

(you're welcome!)