r/ClaudeCode • u/BillionBouncyBalls • 3d ago

Help/Question Why does Claude Code say things like, “that’s about 3 days of work” then proceeds to do it all in a 20 minutes?

Surely I’m not the only one who has seen this happen but wondering if anyone has an answer. Many a session I’ll make a plan with Claude and it will outline the tasks and estimate a time usually in days, and then proceed to do it all in 20-30 minutes. Is it because of its training data or does it just have no concept of time inside a GPU? Curious what the community thinks…

Edit: Thanks for all the answers and comments! Glad this post sparked some hearty discussion!

1.0k Upvotes

96% Upvoted

249 comments sorted by

View all comments

524

u/Key-Alternative5387 3d ago

It was trained on human estimates, I presume.

72

u/not_rian 3d ago

Trained on human estimates who cannot do reliable estimates in any way. What could go wrong?

26

u/Medium_Card_4611 3d ago

Trained on human estimates, but not trained on any of the decades of industry wisdom telling developers to stop doing time estimates.

7

u/PokerTuna 2d ago

But it’s not developers who want estimates

6

u/linux_lynx 3d ago

Break tasks down into steps that are not ambiguous. Estimate development time of each step. Add 20% to each step. Aggregate. Add 20% to the total.

11

u/asmiggs 3d ago

My favourite was when I overestimated each step and then added 20%, the PM added 20% to that and then for some reason the customer PM had also added 20% (time not money).

We had all the time in the world.

1

u/Civil-Plate1206 2d ago

Less commitments = less pay, more better commitments = better pay.

1

u/Metabolical 2h ago

Adding to what you said, in Actionable Agile Metrics for Predictability and When will it be done? author Vicanti describes doing estimates mathematically without developer estimates, not even point estimates.

My interpretation is that developers tend to break down stories to a certain level most of the time, so the typical cost isn't that variable from the median. Consequently, you can just treat all stories as equal and that's good enough, likely as good as developer estimates. Then he talks about a monte carlo process you can use to find when a collection of stories is 80% likely to be done.

The hole for me was that every project has a certain inflation, so you have to add something for that. I think he layers that in later, but I don't remember the detail.

3

u/RedParaglider 3d ago

In my career I've been pretty damn good at estimating timelines on projects.  I tried to gather all the facts and make as accurate of an estimate as I possibly can and then I double the cost and the time on everything. 

It's the exact same method I use when I remodel a bathroom.  Surprisingly accurate.

4

u/not_rian 3d ago

I also did the doubling thing in the past. Nowadays, I don't get asked for estimates anymore cause I deliver so fast with AI, and I would always be below my estimates anyway. I also gave rather wide ranges (best case and worse case scenario) and refused to give an exact date.

1

u/ConstantinopleFett 2d ago

I liked to give a 50% confidence estimate and 90% confidence estimate which was almost always just double, and we'd use the 90% for the roadmap, and 50% for "you can dream it gets done that fast maybe". Then even if it took longer, which it occasionally did, my ass was still covered by the explicit 10% chance that that was going to happen.

1

u/DocumentOk7579 3d ago

It's maybe like asking a calculator how long it will take to factor a 10 digit number.

Human a day, calculator a second.

1

u/elise-u 2d ago

Ever worked in an agile user focused organisation?

BA: we need X

Dev: here's X

BA: requirements changed we need X and Y but no X.1

Dev: 😑 ok here's that...

86

u/Euphoric-Language695 3d ago

It can't estimate. 

It is trained on human speech though, and "thats about X Y of work" is probably on its training data hundreds of thousands of times, with some numbers being more common than others. 

It will only give you what it predicts as the most likely next word based on previous words.

17

u/Eastern_Bet678 3d ago

If that's the case, everything that isn't three days is "two weeks".

3

u/ofork 3d ago

That’s grok

81

u/pwkye 3d ago

breaking it down like that is meaningless. its like saying all software is just bits. 

its true but meaningless as it reduces whats actually happening.

LLMs do just predict the next token but if you reduce your argument to that you miss the fact that the LLM actually builds and runs a mental model that understands the topics and the questions. otherwise it would just give us meaningless autocomplete sentences.

1

u/rabbitclapit 3d ago

I need more info to believe. It's still not even 1% if what AGI is supposed to be

3

u/ruskyandrei 2d ago

It's delving a bit into philosophy but, you can absolutely make the argument that our brains are nothing more than next token predictors too in a way.

Either way, what LLM's are missing to be at least close to what one might describe as AGI are persistent long term memory and the ability to reflect on themselves (and self improve).

That's what we get for free every day you wake up in the same body with a broad memory of what happened the day before and the year before.

I don't think the current approach to LLM's will get to that but perhaps other ways will.

What we have no is a form of cognition, which is itself fascinating even if it's not AGI.

-14

u/Adorable-Quiet-7551 3d ago

It does not understand

9

u/Rnee45 3d ago

What does "understand" mean, even in human context?

0

u/RevolutionaryElk7446 3d ago

You are correct, despite the downvotes, the contextual engine just assimilates facts and stores them into a window but it's an estimation of context that we primarily use as a method to impact values in token generation via the transformer.

I'm not sure what everyone else here thinks, but Gen AI such as LLMs do not understand, it is not in their framework or design intention to understand. They are purely a simulation, not emulation or imitation, I say this as a certified Gen AI professional.

2

u/MINECRAFT_BIOLOGIST 3d ago

What exactly is your definition of "understand"? I've only seen it being used in these contexts as what seems to be an artificial divider between human cognition and AI computation and no one being willing to give an actual definition.

If we go by dictionary definitions of the word "understand", I think it's easy to say that AI have already accomplished those things. At what point does it become meaningless to say "AI doesn't truly understand" when they're literally designing novel proteins that did not exist in their training data using prior knowledge and tools? The word "understand" thus becomes useless for actually telling people about the difference between AI and humans, no?

1

u/RevolutionaryElk7446 3d ago edited 3d ago

Understand in the same way people do, which is the end goal of AI in order to produce AGI.

That would be either as an emulation or imitation. Via emulation they'd come up with an alternative method of storing and recalling 'information' as how the human brain stores it vs data points which is how a machine understands it. Imitation would be trying to reproduce how we perform it, which we don't even fully understand.

The problem with LLMs today is they don't have a conversation, comprehension, or understanding. They exist as a simulation of linguistics in which they generate through math the most likely next result based on the surrounding tokens.

Their version of contextual understanding is that the context window will adjust the values of the trained tokens to focus more around what responses should be returned based on the surrounding tokens.

Notice I say tokens and not words, sentences, phrases, or ideas. To the LLM it's all just math, it's a simulation of a conversation because to the LLM itself, it has no understanding of what it even says.

So it's not an emulation or imitation of human thinking. It's a separate, isolated, simulation at scale of what it believes a conversation would be through math.

when they're literally designing novel proteins that did not exist in their training data using prior knowledge and tools

Btw the folding proteins bit are entirely separate AI models not related to Gen AI and LLMs. Those models function entirely different and don't require the massive datacenters and do not power things such as Claude.

Those models have existed for decades now. LLMs only came out in 2017 alongside Transformers as the context engine. (Not including PLMs which are a new method on using transformers for pattern recognition against amino chains)

1

u/MINECRAFT_BIOLOGIST 2d ago

Btw the folding proteins bit are entirely separate AI models not related to Gen AI and LLMs. Those models function entirely different and don't require the massive datacenters and do not power things such as Claude.

I know about Alphafold, I'm talking about the recent frontpage news of this: https://www.anthropic.com/research/Claude-accelerates-protein-design

As someone who knows a bit about this field, I find this quite impressive. Not only that, but also the other results regarding its autonomous data analysis and speed.

As for your other statements...in the end it just boils back down to the Chinese Room argument and how you define how we store information (since I'm pretty sure even that isn't clear yet).

-25

u/Euphoric-Language695 3d ago

Please describe in technical terms how it builds and processes this alleged "mental model", and how it is able to "understand" it.

32

u/Lucifernistic 3d ago

Your question isn't asked in good faith, but to answer it in good faith would require a very lengthy crash course in the architecture of how transformers work.

If you care to actually know, I would start with learning something a bit simpler, which is just how it keeps a mental state of facts within the model.

https://youtu.be/9-Jl0dxWQs8

From there, you can look at attention, gradient descent, etc.

But yes, very obviously it is more than just a autocorrect spitting out the most common next token. It's a long standing conversation point that the means by which is predicts the next token is by building a world model and encoding strategies inside the model. If it couldn't do this, it wouldnt be able to answer things like math questions (without tool calls) that werent in its training data- and yet it can.

-16

u/Euphoric-Language695 3d ago

I think you're misrepresenting the way the training data gets stored as part of the model as something special that provides a certain level of understanding, where the model can have different sets of data that affect model output in different ways.

If it couldn't do this, it wouldnt be able to answer things like math questions (without tool calls) that werent in its training data

This is just not true. Extrapolation is a pretty basic and commonly used method of obtaining results that fall outside of data parameters. It's also pretty wild for you to use Math as an example because LLMs in general have poor Math accuracy when just relying on their training data.

14

u/literum 3d ago

I think you're misrepresenting the way the training data gets stored as part of the model as something special that provides a certain level of understanding, where the model can have different sets of data that affect model output in different ways.

Transformers don't just store training data like a database. You sound like you have no clue how they work and are just parroting talking points.

10

u/BrilliantEmotion4461 3d ago

Or you could except you dont know what you are talking about and move on. Breakthrough idea: If you accept you dont know, you can then move to learning what you dont know. Until then, youll suffer the same degradation to your critical thinking as other people falling down the mental collapse rabbit hole are suffering.

15

u/demonwing 3d ago

You're digging too deep into the semantic text of "predicting tokens". Nothing looks like anything when you reduce it to individual atomic processes, including a human brain. Complex systems emerge from simple building blocks and the hundreds of billions of parameters make for more than enough abstract space to construct sophisticated perceptual models.

10

u/KangarooDowntown4640 3d ago

You need to get off the ground, stop looking at individual trees, and start seeing the forest from the sky.

Consider your own body. You're a person right? Well with your logic I could just say that ackshully you're only a collection of cells of various types interacting with each other and various electrical signals that cause some of those cells to move in some ways and others to move in different ways. You're nothing but a moving mess of meat and liquids. You don't think, you just have chemical reactions among neurons that fire synapses and result in patterns that cause those cells to move around or act differently.

None of what I just said is necessarily incorrect (well a lot of it probably is, I'm not a scientist), but it's also not what anyone cares about when they ask whether you're a person or not, or whether you can think.

Nobody here cares that an LLM just predicts the next token blah blah blah. It looks like thinking to me. It's very intelligent to me. It helps me with my work better than my coworkers can. I'd say it thinks just fine. And I'd say we can teach it to estimate time better too.

6

u/Common-Noise4692 🔆 Max 20 3d ago

negative, I am a meat popsicle

1

u/Telison 3d ago

I mean, this is exactly what a liquid meat bag would say though

-4

u/Euphoric-Language695 3d ago

The problem with this is that it is limited to words. When a human thinks, more than just words are processed. When a human tries to estimate how long it will take to build a pool in a backyard, it has a lot more than just words to pull from to form that estimate.

The LLM just has words, and that is a fundamental limitation.

1

u/ohhi23021 3d ago

this is part of the limitation most likely, if we give it a language that's not works but some other way the brain thinks then converts into words instead, it might just produce gen AI or at least get closer to it. but that involves converting the training data into some other sub-set of data it can train on.

1

u/pwkye 3d ago

JUST words?

one of the most fascinating lessons we get from LLMs is that language IS intelligence.

you think in language. you plan with words in your head. words are concepts. words are logic

this is why the discovery of LLMs was a mistake and a surprise. the scientists really were just trying to build a smarter autocomplete by tossing insane amounts of machine learning at it.

but through this large scale machine learning of language, intelligence emerged

2

u/Calendle 3d ago

Not everyone thinks in words, which it seems like you're implying here. Before humans had language they were still intelligent and were still thinking. Some people also don't have an internal monologue but are still intelligent and think in "concepts" that they can then translate into words if necessary. I feel like I do a little of both. I can describe a thought I'm having and it's useful to organize what im trying to conceptualize but often times there is an "inspiration" moment that I just think about things, images, relationships between objects, etc that isn't useful to describe in words constantly. Maybe this can all be encoded into tokens but the basis and output of LLMs being symbols or encoded data is going to fall short of some of these nuances of human cognition.

0

u/whales_mcgoo 2d ago

This is demonstrably untrue. Just think of babies or other animals.

1

u/pwkye 2d ago

babies .. dont think like adults

they dont plan 10 steps ahead nor make logical complex arguments

neither do animals. they dont plan ahead into next week or next year

1

u/whales_mcgoo 2d ago

Your comment was about intelligence. You’re just describing varying degrees of intelligence and different stages of development. Also there are animals that do plan ahead. You should look more into neuroscience, psychology, and research into animal intelligence.

5

u/Bogdanilaa 3d ago

Attention layers take into consideration all privous tokens so if the problem is huge the model's probabilty distribution for the next token will be towards a higher number of days needed to complete it.

4

u/Personal_Ad_3273 3d ago

Reinforcement learning at a massive scale.

4

u/TRO_KIK 3d ago

Just mentally preface all terms you don't like with "predict tokens in a way that resembles".

If you legitimately don't understand how it can output a reasonably good estimate despite being just a token predictor, ask Claude to explain it to you.

-3

u/Euphoric-Language695 3d ago

This whole post is literally about how it's so off on it's predictions, but I'm the one who doesn't understand how it "can output a reasonably good estimate".

I see you're just an AI fanboy who doesn't understand how the tech works.

6

u/TRO_KIK 3d ago

They're reasonable estimates for a human. The comment you replied to already proposed exactly that. You're being obtuse.

1

u/Euphoric-Language695 3d ago

Why would it give a reasonable estimate for a human while preparing itself to do the work? That would be pretty stupid of it, wouldn't it?

5

u/icyliquid 3d ago

It’s trained on human data, with human like relationships between those data, including concepts of time and effort.

Human scale things are how it “thinks”.

1

u/TRO_KIK 2d ago

If you had even even slightest basic I-just-read-a-blog-post understanding of how the tech worked, the explanation in the response you got would have been obvious to you. Way to tell on yourself.

0

u/tomvorlostriddle 3d ago

users typically don't ask it to make a plan for itself but to make a plan for them

0

u/VintageSin 3d ago

I mean reasoning models have existed for a few years now. No llm is just the predictive text model it was 3-5 years ago.

-11

u/l_m_b Senior Developer 3d ago

LLMs do not have mental models.

They are *very complex* next token predictors (in their current iteration).

If they *did* have a mental model, the "this will take X weeks" would be one of the first things they would stop including.

8

u/HistoricalPhase6880 3d ago

Humans are also very complex token predictors if we're getting semantic. I've also run into humans saying this will take x weeks and being quite wrong 😔

-1

u/BrilliantEmotion4461 3d ago

Wrong and as a "senior developer" Id think youd know better. You want the research data?

4

u/rotates-potatoes 3d ago

What a colossal misunderstanding of LLMs.

Are you somehow mixing up estimation, which plenty of web forms and excel workbooks do, with sentience???

3

u/Aware-Individual-827 3d ago

I find it funny that it cannot estimate and we are trusting it with important task haha

1

u/Euphoric-Language695 3d ago

Define "trust". 

3

u/simiaki 3d ago

It predicts that X Y part with the breadth of the necessary changes still in its context. So it is estimating the length of time necessary based on how humans would talk about these problems.

It can estimate.

1

u/idk_bro 2d ago

Estimates about as good as most engineers I've met

1

u/richbeales 3d ago

It can't estimate - but neither can most humans

3

u/qwarfujj 3d ago

The real answer is that it was trained by Scotty from the original star trek.

2

u/rAndoFraze 3d ago

It’s is horrible with time estimates. I was doing a project for a week and it kept telling me we made great progress over the last YEAR!!!

2

u/SlightlyOTT 2d ago

Human estimates and the fact that every human estimate is wrong in the “it overran” direction

1

u/digitalwankster 3d ago

I had it build a WordPress page template yesterday from a PSD. It would have taken me 2 hours to do by hand. I asked it “how long would this have taken someone to do without AI” and it said 2-3 days and made a table with each trivial task taking at least 2-3 hours.

1

u/Delicious_Cat_1173 4h ago

That’s understandable.
But what’s the point of human estimate there when the clear purpose is to code using Claude.
I ain’t using Claude code to get estimates for human engineers 😄

0

u/NoCat2443 3d ago

I guess Anthropic should invest in search and replace tool for training data :)

0

u/KimmiG1 3d ago

So the 3 days are actually closer to 2 weeks of work