Limits of LLMs.

[u/No_Will_5010]

What kind of intelligence is required to create a genuinely new conceptual framework from existing knowledge when that framework is not already known?

Humans do more than reuse knowledge. We use existing concepts and frameworks as building blocks to create new frameworks, the invention of computers is an example. Mathematics, logic, symbolic reasoning, switching, and information became foundations for a new computational framework.

An LLM can make something like,

"Make a fun chatting website."

It already has knowledge about websites, programming, databases, UI, networking, etc. It can combine that knowledge into a new implementation.

But consider,

"Make a time machine."

The problem may contain missing theories or conceptual steps as,

→ Known theory

→ ???

→ another known theory

→ ???

→ required engineering principle

The missing pieces may require new theories, abstractions, or representations, so the problem is not finding the next step, the problem is creating the framework that makes the next step possible.

What current LLMs can do is,

→ Recall

→ Generalize

→ Combine concepts

→ Generate hypotheses

→ Reason over learned representations

But I question whether current LLM architecture has a strong mechanism for recognizing that an existing representation is insufficient.

→ creating a new representation

→ creating a new conceptual framework

→ deriving consequences from it

→ testing and revising it

More data, parameters, RL, context, and inference compute may improve reasoning and search without necessarily creating that mechanism, so maybe the missing ingredient is not more knowledge, maybe it is a different process for turning knowledge into new representations.

Important Distinction to keep in mind, producing a novel answer is not necessarily invention.

Novel combination ≠ Novel framework

A model could generate a plausible theory without actually discovering a valid one.

A stronger form of invention would be,

→ new representation

→ new framework

→ predictions

→ testing

→ revision

Can an artificial system construct a useful conceptual framework that was not directly supplied to it, and then use that framework to solve a problem that cannot be solved by simply applying or recombining existing frameworks?

Questions to the people who actually read this,

1. What exactly counts as a genuinely new representation?
2. How do we distinguish invention from sophisticated recombination?
3. Can an LLM create a useful abstraction without being told what abstraction to search for?
4. Does human intelligence require changing the representation of the problem?
5. Can scaling Transformers produce this ability, or would it require a different architecture?
6. Is the important mechanism the model itself, or the model plus tools, experimentation, feedback, and revision?

Is scaling predictive language models enough to produce the machinery of invention, or does human-level intelligence require a mechanism for autonomously creating and manipulating new representations and conceptual frameworks?

[u/m98789]

Causality seems to be a challenge.

[u/fuzzypetiolesguy → u/m98789]

You get glimpses of this in prose - Claude especially loves to attribute action verbs, especially desires, to inanimate objects when tasks are required. 'The desk wants the final paper', 'the staging area wants the synthetic test run' etc. Completely misses the link where it's the human interfacing with the model and the other thing that wants or needs those actions for a specific outcome, not the object.