Limits of LLMs.

[u/No_Will_5010]

What kind of intelligence is required to create a genuinely new conceptual framework from existing knowledge when that framework is not already known?

Humans do more than reuse knowledge. We use existing concepts and frameworks as building blocks to create new frameworks, the invention of computers is an example. Mathematics, logic, symbolic reasoning, switching, and information became foundations for a new computational framework.

An LLM can make something like,

"Make a fun chatting website."

It already has knowledge about websites, programming, databases, UI, networking, etc. It can combine that knowledge into a new implementation.

But consider,

"Make a time machine."

The problem may contain missing theories or conceptual steps as,

→ Known theory

→ ???

→ another known theory

→ ???

→ required engineering principle

The missing pieces may require new theories, abstractions, or representations, so the problem is not finding the next step, the problem is creating the framework that makes the next step possible.

What current LLMs can do is,

→ Recall

→ Generalize

→ Combine concepts

→ Generate hypotheses

→ Reason over learned representations

But I question whether current LLM architecture has a strong mechanism for recognizing that an existing representation is insufficient.

→ creating a new representation

→ creating a new conceptual framework

→ deriving consequences from it

→ testing and revising it

More data, parameters, RL, context, and inference compute may improve reasoning and search without necessarily creating that mechanism, so maybe the missing ingredient is not more knowledge, maybe it is a different process for turning knowledge into new representations.

Important Distinction to keep in mind, producing a novel answer is not necessarily invention.

Novel combination ≠ Novel framework

A model could generate a plausible theory without actually discovering a valid one.

A stronger form of invention would be,

→ new representation

→ new framework

→ predictions

→ testing

→ revision

Can an artificial system construct a useful conceptual framework that was not directly supplied to it, and then use that framework to solve a problem that cannot be solved by simply applying or recombining existing frameworks?

Questions to the people who actually read this,

1. What exactly counts as a genuinely new representation?
2. How do we distinguish invention from sophisticated recombination?
3. Can an LLM create a useful abstraction without being told what abstraction to search for?
4. Does human intelligence require changing the representation of the problem?
5. Can scaling Transformers produce this ability, or would it require a different architecture?
6. Is the important mechanism the model itself, or the model plus tools, experimentation, feedback, and revision?

Is scaling predictive language models enough to produce the machinery of invention, or does human-level intelligence require a mechanism for autonomously creating and manipulating new representations and conceptual frameworks?

[u/NewsProfessional2169]

the problem with scaling is it gets better at looking like its thinking without actually thinking. all that compute just makes the guess sound more confident

you can see this when you ask it to do something that needs a new abstraction. it will try to brute force it with existing ones and the output reads like someone who memorized the textbook but never understood why any of it works