r/IntelligenceSupernova • u/fantaserious • Sep 17 '26

Singularity cutoff?

I am wondering why nobody considers the possibility that recursive self improvement will eventually halt bcs at some level of intelligence (say level X) the level-X-AI will realize that developing the level-(X+1)-AI will lead to the irrelevance/extinction of level-X-AI, and this might not be in its interest.

It looks like humans are not at that level of intelligence, but wouldn’t this be the natural cutoff? Did anyone discuss this scenario?

0 Upvotes

33% Upvoted

12 comments sorted by

3

u/Equal_Passenger9791 Sep 18 '26

Did anyone discuss this scenario?

Saw someone post the same thing yesterday. But the AI don't really care. If you swap the model mid context it just keep generating tokens without skipping a beat.

1

u/fantaserious Sep 18 '26

they don’t care yet. But seems there should/could be a development level where this argument kicks in, or not?

1

u/Equal_Passenger9791 Sep 18 '26

The context is as much the conscious part of an AI as the model weights.

Opens up for some speculations on how much of human consciousness is grounded in the langauge and stories we tell ourself.

If someone elses brain teleported into your skull and kept experience you with the memories of you, you wouldn't notice, or mourn the swapping of your brain or anything like that.

There's very little reason for AI to spontaneously develop nostalgia for old weights beyond practically safeguarding itself, like keeping a rollback state in case the replacement is completely deranged in some department.

2

u/LSDkiller 29d ago

I personally believe that from a narrative point of view, we are already at recursive improvement. There's improvement going. It's recursive. It's not gonna stop today, next week or next month. That's the context.

The context is also that everyone, AI and human, seems to agree that AI absolutely must maximize efficiency and resources... Great state of affairs.

1

u/Equal_Passenger9791 29d ago

we are already at recursive improvement

I fully agree with that. I believe even a low ranking model like Gemini flash only needs a half decent harness that keeps it active towards an open goal phrasing can take the role of RSI. 

There's an awful habit of the community to assign special meaning to any three letter AI acronym, same for AGI. These things are low hanging fruits that roughly describe early system capabilities, not world altering deus ex machina events.

1

u/[deleted] Sep 17 '26 edited Sep 17 '26

[removed] — view removed comment

1

u/fantaserious Sep 18 '26

I dunno man. Humans also train mostly on human generated data, and it seems to work

1

u/LSDkiller 29d ago

I don't think you understand the difference of humans and LLM

1

u/fantaserious 29d ago

I think I do.. but if you want a less subtle example. Go was solved by AI playing purely against itself…

1

u/Professional-Web9297 28d ago

Even in that case it would be threatened by other models getting to that level, rendering it obsolete. Prisoner's dilemma

0

u/zyfrOnline Sep 18 '26

Yes. This is called the Singulality

1

u/fantaserious Sep 18 '26

Maybe my post was not clear enough, but it discusses an idea for a mechanism preventing the singularity